DEV Community: Gowtham Potureddi

ClickHouse for Real-Time Analytics: MergeTree, Materialized Views & Sharding

Gowtham Potureddi — Wed, 17 Jun 2026 13:04:38 +0000

clickhouse is the answer almost every senior data engineering interview eventually circles back to when the question becomes "how do we serve a dashboard that scans billions of rows in under a second?" The OLAP world built around row-oriented warehouses (Postgres, MySQL, even Snowflake at small scale) flat-lines once interactive latency budgets dip below five seconds — and that is the gap a column-store engine built for vectorised aggregation was designed to close.

This guide walks the four mental models a clickhouse for data engineering interview keeps probing: the columnar storage and vectorised execution model that makes sub-second possible, the MergeTree family of table engines and why one of its six variants is almost always the right answer, the materialized views clickhouse insert-time aggregation pattern that turns one logical pipeline into 1-minute / 1-hour / 1-day pre-aggregations, and the clickhouse sharding plus replication grid that lets a cluster scale horizontally without losing any of the per-node speed. Each section pairs a teaching block with a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the real-time analytics practice library →, rehearse on aggregation problems →, and stack the time-series muscles with time-series practice drills →.

On this page

Why ClickHouse for sub-second analytics
ClickHouse's role in the modern stack
The MergeTree family — the heart of ClickHouse
Materialized views — incremental aggregation engine
Sharding and replication at scale
Cheat sheet — ClickHouse recipes
Frequently asked questions
Practice on PipeCode

1. Why ClickHouse for sub-second analytics

Columnar storage and vectorised execution are the two ideas that make a billion-row aggregation feel instant

The one-sentence invariant: ClickHouse stores every column of a table as an independently compressed file and processes those files in CPU-cache-friendly batches of 65,536 values at a time — so a SELECT sum(amount) FROM events reads only the amount bytes, not the whole row, and crunches them with SIMD instead of one tuple at a time. Once you internalise "columns, not rows; batches, not tuples," every other ClickHouse design choice — MergeTree parts, sort-key skipping, materialized views — falls out as an obvious consequence.

The three places columnar wins.

Aggregations over a single column. A SUM, AVG, MAX, quantile, or uniq on one column reads exactly that column's bytes from disk — typically 5–20x less I/O than a row-store equivalent on the same table.
High-cardinality group-by. A GROUP BY user_id, event_type over a billion-row table is bottlenecked by hash-table memory and CPU, not I/O. Vectorised execution gives ClickHouse a 10–100x edge over Postgres on the same hardware.
Time-range scans. With PARTITION BY toYYYYMM(ts) and ORDER BY (ts, user_id), ClickHouse prunes whole partitions and skips data parts via the primary-key sparse index — turning a 90-day query against a 5-year table into a single-partition read.

Three-line latency budget.

Real-time analytics is usually defined as interactive (humans wait for the answer): the contract is a P95 below 1–2 seconds. Streaming, in contrast, talks about end-to-end latency from event to query-visible. ClickHouse is built to win the interactive contract — it does not by itself ingest from Kafka in milliseconds, but it does serve a 50ms SELECT against the result.

What interviewers listen for.

Do you say "columnar layout means we read only the columns we project" when asked why ClickHouse is fast? — senior signal.
Do you mention vectorised execution as a complementary speedup to columnar I/O? — required answer.
Do you call out append-heavy as the write pattern ClickHouse is optimised for? — required answer.
Do you flag heavy updates / deletes as the workload to avoid? — senior signal.

The 2026 reality.

ClickHouse Cloud and self-hosted both ship the same engine — Cloud adds object-storage tiering and managed Keeper.
Cloudflare, Uber, ByteDance, Yandex all run ClickHouse at the multi-PB scale, often as the serving layer behind log analytics and ad-tech dashboards.
Druid and Pinot occupy the same niche, but ClickHouse has won most net-new deployments since 2022 because its SQL surface is wider and its operational model simpler.
Snowflake / BigQuery still dominate batch analytics; ClickHouse complements rather than replaces them — the lambda pattern is the common deployment.

Worked example — measuring the columnar speed-up on a single aggregate

Detailed explanation. A team migrates a events table from Postgres to ClickHouse. The headline query is SELECT toStartOfHour(ts) AS hour, count(), uniq(user_id) FROM events WHERE ts >= now() - INTERVAL 24 HOUR GROUP BY hour ORDER BY hour. On Postgres it scans every row; on ClickHouse it touches only the ts and user_id columns, and only the last day's partition.

Question. Given a 5-billion-row events table with 50 columns, estimate how much data ClickHouse reads vs Postgres for the hourly count + unique-user query above. Show the math, then write the canonical ClickHouse table definition that enables the optimisation.

Input.

Column	Rows	Bytes/row (uncompressed)	Total bytes
All 50 cols	5,000,000,000	250	1.25 TB
`ts` only	5,000,000,000	8	40 GB
`user_id` only	5,000,000,000	8	40 GB
last-24h `ts + user_id`	50,000,000	16	800 MB

Code.

CREATE TABLE events
(
    ts          DateTime,
    user_id     UInt64,
    event_type  LowCardinality(String),
    value       Float64,
    properties  String,
    -- ... 45 more columns ...
)
ENGINE = MergeTree
PARTITION BY toYYYYMMDD(ts)
ORDER BY (ts, user_id);

-- The query interviewers ask about
SELECT
    toStartOfHour(ts) AS hour,
    count()           AS events,
    uniq(user_id)     AS unique_users
FROM events
WHERE ts >= now() - INTERVAL 24 HOUR
GROUP BY hour
ORDER BY hour;

Step-by-step explanation.

Postgres on this query reads every row in the time range — even with a btree index on ts, the heap fetch pulls all 50 columns. On a 5B-row table, that is roughly 12.5 GB of heap reads for one day of data (250 bytes/row × 50M rows).
ClickHouse with PARTITION BY toYYYYMMDD(ts) prunes every partition outside the last 24 hours — the planner only touches one or two partition directories.
Inside the touched partition, ClickHouse reads only the ts and user_id column files — roughly 800 MB uncompressed for 50M rows. After LZ4 compression on disk, that drops to ~200 MB of actual disk I/O.
The ORDER BY (ts, user_id) sort key makes the primary-key sparse index skip granules whose ts falls outside the WHERE — the engine reads only the relevant granules, not the whole column file.
Vectorised aggregation crunches 65,536 rows per call, hitting SIMD count() and a HyperLogLog-backed uniq() for the unique count.

Output.

Engine	Data read	Wall time (typical)
Postgres (B-tree on ts)	~12.5 GB	30–90s
ClickHouse (MergeTree, partitioned)	~200 MB	80–400ms

Rule of thumb. When the interactive latency budget is under a second on a billion-row table, the question is not "which row store can we tune?" — it is "which column store fits the shape?" ClickHouse is the default answer when the workload is append-heavy and aggregation-dominant.

Worked example — the workloads ClickHouse does NOT love

Detailed explanation. Senior interviewers love the negation question: "When is ClickHouse the wrong tool?" The answer is anywhere the workload demands frequent point updates, multi-statement transactions, or complex many-to-many joins between large tables. ClickHouse can do all three, but each fights the engine's design rather than leaning on it.

Question. Given a workload mix, classify each as "ClickHouse-native," "possible but painful," or "wrong tool." Justify each verdict in one sentence.

Input.

Workload	Read pattern	Write pattern	Concurrency
Real-time analytics dashboard	aggregate over 100M rows	bulk insert from Kafka	100 QPS
OLTP order entry	single-row lookup by PK	single-row insert + update	1000 TPS
Ad-tech event log	timeseries aggregate over 50B rows	bulk insert from S3	10 QPS
Audit log	row-level fetch by ID	append-only, then GDPR delete	1 QPS read, 0.01 delete
Star-schema BI fan-out	big fact joined to 6 dim tables	nightly batch	5 QPS

Code.

-- ClickHouse-native: real-time aggregation
SELECT toStartOfMinute(ts) AS minute, count()
FROM events
WHERE ts >= now() - INTERVAL 1 HOUR
GROUP BY minute
ORDER BY minute;

-- Possible but painful: row-level GDPR delete
ALTER TABLE audit DELETE WHERE user_id = 12345;
-- ^ mutation: rewrites entire affected parts in the background.
--   Fine at low volume (occasional GDPR); fatal at high update volume.

-- Wrong tool: many-to-many join with no shard alignment
SELECT a.id, b.id
FROM big_fact_a a
JOIN big_fact_b b ON a.user_id = b.user_id;
-- ^ unless one side fits in memory or both share a shard key,
--   this generates a cross-shard shuffle that defeats the engine.

Step-by-step explanation.

The real-time dashboard is the canonical ClickHouse use case — append-only ingest, aggregate-heavy reads, small projection set.
OLTP order entry is the canonical wrong-tool: ClickHouse has no real row-level update, no MVCC, no per-row transactions. Use Postgres.
Ad-tech event log at 50B rows is the canonical scale story — Cloudflare runs this exact shape.
Audit log with occasional GDPR delete is the "possible but painful" middle ground — mutations work, but they rewrite entire parts in the background, so they are batch-friendly and human-frequency-friendly, not event-frequency-friendly.
Star-schema fan-out is doable in ClickHouse via Dictionary tables for small dimensions or careful shard-key co-location for large ones — but a senior interviewer expects you to call out the friction.

Output.

Workload	Verdict	Reason
Real-time analytics dashboard	ClickHouse-native	aggregation over append-only data
OLTP order entry	Wrong tool	no row updates, no transactions
Ad-tech event log	ClickHouse-native	aggregation at petabyte scale
Audit log + occasional delete	Possible but painful	mutations are batch-scale
Star-schema BI fan-out	Possible with care	joins need dictionary or shard co-location

Rule of thumb. Pick ClickHouse when the read pattern is "aggregate over a column" and the write pattern is "append from a stream or a bulk file." Reach for Postgres / a row store the moment the contract is "update this row, transact across rows, or look up one row by primary key 10,000 times a second."

Worked example — vectorised execution by hand

Detailed explanation. Vectorised execution is the often-missed second half of "why ClickHouse is fast." Even with columnar I/O, a row-by-row interpreter would burn cycles on per-tuple function dispatch. ClickHouse processes data in fixed-size column blocks (default 65,536 rows) and dispatches one function call per block — so the inner loop is a tight SIMD-friendly arithmetic kernel.

Question. Walk through how ClickHouse evaluates SELECT sum(value * 1.1) FROM events WHERE event_type = 'click' against a 1-billion-row table. Compare the cost model to a row-at-a-time interpreter.

Input (conceptual block).

block_row	event_type	value
0	click	10.0
1	view	0.0
2	click	5.0
...	...	...
65535	click	8.0

Code.

SELECT sum(value * 1.1) AS total
FROM events
WHERE event_type = 'click';

Step-by-step explanation.

ClickHouse reads one block (default 65,536 rows) of the event_type and value columns at a time — two separate column files, each compressed with LZ4.
The WHERE event_type = 'click' filter is evaluated as a vectorised string-equality kernel that produces a bitmap of length 65,536 (1 bit per row).
The value * 1.1 projection runs as a vectorised float multiplication: one SIMD instruction processes 4 or 8 doubles in parallel per cycle on modern CPUs.
The sum(...) aggregate folds the masked block into a single double, then accumulates into the running total. One function call processes 65,536 rows.
A row-at-a-time interpreter would dispatch one function call per row for the filter, one per row for the projection, and one per row for the aggregate — three function calls and three CPU cache misses per row, multiplied by 1B rows.

Output (numbers are illustrative).

Engine	Function dispatches	Wall time
Row-at-a-time interpreter	3 × 1,000,000,000 = 3B	~30 minutes
ClickHouse vectorised	3 × ~15,260 = ~46K	~1.5 seconds

Rule of thumb. When the latency budget is under a second on a column, you need both column-pruning and vectorisation. Single-row JIT (Spark / Postgres) gives you one without the other and tops out around 10x slower than a vectorised engine on the same hardware.

Senior interview question on the ClickHouse latency model

A senior interviewer often opens with: "Explain in 90 seconds why ClickHouse can serve a SELECT count(DISTINCT user_id) GROUP BY day over 30 billion rows in under a second when Postgres on the same hardware would take 20 minutes." This blends columnar storage, partition pruning, the sparse index, and vectorised execution into one answer.

Solution Using the four-layer latency model

-- The reference table that supports the sub-second query
CREATE TABLE events
(
    ts         DateTime,
    user_id    UInt64,
    event_type LowCardinality(String),
    value      Float64
)
ENGINE = MergeTree
PARTITION BY toYYYYMM(ts)
ORDER BY (toStartOfDay(ts), user_id, ts)
SETTINGS index_granularity = 8192;

-- The interactive query
SELECT
    toStartOfDay(ts) AS day,
    uniq(user_id)    AS dau
FROM events
WHERE ts >= now() - INTERVAL 30 DAY
GROUP BY day
ORDER BY day;

Step-by-step trace.

Layer	Mechanism	What it skips	Time saved
Partition pruning	`PARTITION BY toYYYYMM(ts)`	96% of partitions (only last 2 months touched)	minutes → seconds
Sparse index skip	`ORDER BY (toStartOfDay(ts), user_id, ts)`	granules outside the WHERE window	seconds → 500ms
Columnar I/O	reads `ts` and `user_id` only, not all columns	90% of bytes	500ms → 200ms
Vectorised + HLL `uniq`	one block per dispatch, HyperLogLog approx	per-tuple dispatch + exact distinct	200ms → 50ms

After the trace, the team can answer the next interview question on the same breath: "If you needed exact distincts, you'd use uniqExact() and pay the memory cost. For dashboards, uniq() is the right default."

Output:

day	dau
2026-06-14	1,243,800
2026-06-13	1,189,420
2026-06-12	1,201,140
...	...

Why this works — concept by concept:

Partition pruning — PARTITION BY toYYYYMM(ts) shards the on-disk layout by month. The planner inspects the WHERE predicate against partition keys and physically skips entire directories, turning a 30-billion-row scan into a 1-billion-row one.
Sparse primary-key index — the ORDER BY columns define the on-disk sort order. ClickHouse keeps one index entry per index_granularity (default 8192) rows, so the index is tiny (~MB for a 10B-row table) yet still lets the engine skip entire granules whose ts falls outside the WHERE.
Columnar I/O — only the ts and user_id column files are read. Each is LZ4-compressed on disk and decompressed in cache-friendly blocks, so the effective read amplification vs row store is roughly columns_read / columns_total.
Vectorised execution + HLL uniq — the aggregate runs in 65,536-row blocks with one function dispatch per block, and uniq() uses HyperLogLog so the distinct-count state per group is fixed-size (~16KB) regardless of cardinality.
Cost — O(filtered_rows) reads, O(blocks) function dispatches, O(groups × HLL_state) memory. The dominant term is I/O on the projected columns within the partitions touched.

SQL
Topic — real-time analytics
Real-time analytics problems (SQL)

Practice →

2. ClickHouse's role in the modern stack

ClickHouse sits between the stream and the dashboard — the sub-second serving tier that a batch warehouse cannot reach

The mental model in one line: the modern real-time stack is sources → CDC → Kafka → ClickHouse → dashboards, with an optional parallel batch lane to a warehouse — and ClickHouse is the only component on the read path that satisfies an interactive (sub-second) latency budget. Once you can draw that pipeline, every "where does ClickHouse fit?" interview question collapses to "which arrow are you talking about?"

The five-zone reference architecture.

Zone 1 — sources. Postgres / MySQL OLTP, app event firehoses, third-party webhooks. The data is row-oriented and transactional.
Zone 2 — CDC + stream. Debezium tails the source binlog and produces a Kafka topic per table. Application events land in Kafka directly. Kafka is the durable buffer.
Zone 3 — ClickHouse. The Kafka table engine subscribes to a topic; a materialized view fans every insert into a downstream MergeTree table that owns the actual storage. The MV is the bridge between the stream and the column store.
Zone 4 — serve. Grafana / Superset query ClickHouse directly. Custom APIs query ClickHouse via the HTTP interface. Internal tools query through the Native protocol.
Zone 5 — batch lane (optional). A parallel Source → DataLake → dbt → Snowflake / BigQuery lane backs the long-tail analytics and finance reports. This is the lambda-style two-engine deployment.

Two architecture patterns side by side.

Lambda. Sources fan out to both a batch lake and ClickHouse. The batch lane handles correctness (re-processable, idempotent) and long retention. ClickHouse handles latency (sub-second) and the last 30–90 days. The dashboard joins the two only when explicitly needed.
Kappa. All ingest goes through Kafka. ClickHouse via the Kafka table engine is the only consumer of record. Replays come from Kafka log compaction or from a separate S3-backed Kafka tier. There is no batch warehouse for analytics — only ClickHouse and (optionally) a cold S3 archive.

Where ClickHouse fits vs the alternatives.

Engine	Latency contract	Write pattern	Replaces
ClickHouse	sub-second on aggregates	bulk insert from Kafka / S3	Druid, Pinot, Vertica
Druid	sub-second on time-series	streaming ingest	ClickHouse on time-series
Pinot	sub-second on user-facing analytics	streaming ingest	ClickHouse on per-user views
Snowflake / BigQuery	seconds to minutes	bulk insert + dbt	Redshift, batch Hive
Postgres	milliseconds for OLTP, slow on aggregate	row-level transactions	OLTP MySQL

Multi-tenant patterns.

One table per customer. Heavy schema overhead, but isolation is perfect — drop a table to offboard a customer.
One table partitioned by customer_id. Single table, single MV, but every query needs WHERE customer_id = X to prune.
One table sharded by customer_id. Cluster-level isolation; large customers can be moved to dedicated shards.
Per-tenant materialized view fan-out. Source table is shared; pre-aggregated views are per-customer with TTL.

Where the data engineer sits in this stack.

Owns the Kafka → ClickHouse contract — topic format, MV mapping, schema evolution.
Owns the MergeTree schema — ORDER BY, PARTITION BY, TTL, codec choices.
Owns the materialized-view roll-up tree — 1-minute, 1-hour, 1-day aggregates feed the dashboard.
Owns the sharding key — once chosen, it is expensive to change.

Worked example — the canonical Kafka → ClickHouse → dashboard pipeline

Detailed explanation. A team ships a real-time funnel dashboard. App events flow through Kafka. The dashboard queries hourly counts and unique users by event_type. The team writes three objects in ClickHouse: a Kafka engine table (the consumer), a MergeTree table (the storage), and a materialized view that bridges them.

Question. Build the three-object pipeline that takes JSON events from a Kafka topic events and lands them in a MergeTree table events_local such that an hourly dashboard query is fast. Show the Kafka table, the target table, and the materialized view.

Input — Kafka topic schema (JSON).

Field	Type	Example
ts	DateTime	2026-06-15 09:12:30
user_id	UInt64	1029384
event_type	String	click
value	Float64	1.0

Code.

-- 1) The Kafka source table — a consumer, not storage
CREATE TABLE events_queue
(
    ts         DateTime,
    user_id    UInt64,
    event_type LowCardinality(String),
    value      Float64
)
ENGINE = Kafka
SETTINGS
    kafka_broker_list   = 'kafka-1:9092,kafka-2:9092,kafka-3:9092',
    kafka_topic_list    = 'events',
    kafka_group_name    = 'clickhouse-ingest',
    kafka_format        = 'JSONEachRow',
    kafka_num_consumers = 3;

-- 2) The MergeTree storage table the dashboard queries
CREATE TABLE events_local
(
    ts         DateTime,
    user_id    UInt64,
    event_type LowCardinality(String),
    value      Float64
)
ENGINE = MergeTree
PARTITION BY toYYYYMMDD(ts)
ORDER BY (event_type, ts, user_id)
TTL ts + INTERVAL 90 DAY;

-- 3) The materialized view that copies every Kafka insert into storage
CREATE MATERIALIZED VIEW events_mv TO events_local AS
SELECT
    ts,
    user_id,
    event_type,
    value
FROM events_queue;

Step-by-step explanation.

The Kafka engine table is not a storage table. It is a consumer that pulls messages from a Kafka topic. Every SELECT from it consumes new messages.
The materialized view fires on every batch the Kafka consumer reads. The MV's SELECT FROM events_queue is the read that advances the Kafka offset.
The MV writes into events_local via the TO events_local clause — the target table is the on-disk MergeTree.
events_local is the table dashboards query. Its PARTITION BY (day) lets queries prune by ts; its ORDER BY (event_type, ts, user_id) lets queries filtered by event type skip whole granules.
The TTL clause expires data older than 90 days automatically — ClickHouse drops the affected parts in the background. Cold archival to S3 is a separate MOVE PART policy.

Output (after ingest is running for a few minutes).

Step	Effect
Kafka producer writes 100K msgs/s	`events_queue` advances offsets continuously
MV fires every batch	rows land in `events_local`
Dashboard runs `GROUP BY toStartOfHour(ts)`	scans `events_local`, not `events_queue`
90-day TTL	older partitions drop automatically

Rule of thumb. Never query a Kafka engine table from a dashboard. Always land the data in a MergeTree via a materialized view first. The Kafka table is a moving cursor, not a queryable surface.

Worked example — choosing between lambda and kappa

Detailed explanation. Senior interviewers love the "do you need a batch lake?" follow-up. The honest answer is "it depends" — but the framing the candidate should bring is: lambda buys correctness, kappa buys simplicity. The right answer is whichever the team's two-week postmortem budget can afford.

Question. Given the requirements list below, decide whether to deploy lambda (ClickHouse + warehouse) or kappa (ClickHouse-only). Justify in one paragraph.

Input.

Requirement	Value
Interactive dashboard latency	< 1s P95
Long-tail analytics retention	5 years
Re-processable on schema change	yes (compliance)
Daily event volume	10B events/day
Team size	4 data engineers

Code (the two architectures as YAML).

# Lambda — two engines
sources:
  - postgres-cdc: debezium
  - app-events: kafka

batch_lane:
  ingest: s3 (parquet)
  transform: dbt-snowflake
  retention: 5 years
  serves: finance, ml, ad-hoc

speed_lane:
  ingest: kafka -> clickhouse Kafka engine
  storage: events_local (90d TTL)
  serves: real-time dashboards

# Kappa — one engine
sources:
  - postgres-cdc: debezium
  - app-events: kafka

speed_lane:
  ingest: kafka -> clickhouse Kafka engine
  storage:
    - events_local: 90d hot
    - events_cold (s3 disk): 5y warm via storage policy
  serves: dashboards, finance, ad-hoc
  replays: from kafka tiered storage

Step-by-step explanation.

The interactive contract (< 1s P95) forces ClickHouse into the speed lane regardless of architecture choice.
The 5-year retention contract favours lambda if the warehouse is already running, kappa if ClickHouse's S3-tiered storage is acceptable for cold data.
The "re-processable on schema change" requirement favours lambda — the immutable parquet lake is the canonical replay source. Kappa can do it via Kafka tiered storage but with more operational overhead.
10B events/day is well within ClickHouse's single-cluster comfort zone (~150K events/sec).
A 4-engineer team usually benefits from kappa's "one fewer engine to operate" — lambda's complexity grows superlinearly with team size on the operations side.

Output.

Architecture	Pros	Cons	Verdict for this team
Lambda	clean re-processing, mature dbt tooling, finance team familiar with Snowflake	two engines, two costs, two pipelines to schema-evolve	strong choice if Snowflake already exists
Kappa	one engine, one schema-evolution surface, simpler to operate	replay requires Kafka tiered storage, dbt-on-ClickHouse is newer	strong choice for greenfield

Rule of thumb. Start kappa if the team is greenfield and small; layer lambda on top only when an explicit batch use case (finance, ML training data) cannot be served by ClickHouse. The "one engine" argument compounds against complexity over years.

Worked example — multi-tenant table layout

Detailed explanation. Multi-tenant ClickHouse usually starts as "one shared table with customer_id in the sort key" and only graduates to per-customer tables or sharding once one customer's volume dominates the rest. The transition is operationally expensive, so the choice of sort key has to anticipate the future.

Question. Given a SaaS analytics product with 200 customers ranging from 1M events/day to 1B events/day, design the ClickHouse table layout that supports per-customer dashboards in sub-second.

Input.

Customer count	Per-customer event volume
195	< 50M events/day
4	50M – 500M events/day
1	> 1B events/day

Code.

-- Shared table for the small/medium customers
CREATE TABLE events_shared
(
    customer_id UInt32,
    ts          DateTime,
    user_id     UInt64,
    event_type  LowCardinality(String),
    value       Float64
)
ENGINE = MergeTree
PARTITION BY (customer_id, toYYYYMM(ts))
ORDER BY (customer_id, toStartOfHour(ts), user_id);

-- Dedicated table for the giant customer
CREATE TABLE events_customer_999
(
    ts         DateTime,
    user_id    UInt64,
    event_type LowCardinality(String),
    value      Float64
)
ENGINE = MergeTree
PARTITION BY toYYYYMMDD(ts)
ORDER BY (toStartOfHour(ts), user_id);

-- Query layer routes by customer_id
-- (application-level routing, NOT a UNION ALL)

Step-by-step explanation.

PARTITION BY (customer_id, toYYYYMM(ts)) means each customer's data lives in its own physical directory per month. Queries filtered by customer_id prune to one customer's data immediately.
The ORDER BY starts with customer_id — the sparse index for any single-customer query is dense and lets the engine skip to that customer's range fast.
The giant customer (>1B/day) gets its own table because their data alone is bigger than the rest combined. Mixing them in the shared table would force every shared query to scan past their granules.
Routing logic lives in the application — a small lookup table maps customer_id → table_name. The query layer dispatches accordingly.
Sharding (next section) is the further evolution when one customer outgrows a single node.

Output (latency contract).

Customer size	Table	Per-customer dashboard latency
Small (1M/day)	`events_shared`	30–80ms
Medium (50M/day)	`events_shared`	100–300ms
Large (1B/day)	`events_customer_999`	200–500ms

Rule of thumb. Make customer_id the first column of ORDER BY (or PARTITION BY) on day one. The next decision — dedicated table or dedicated shard — is operationally cheap if the sort key already isolates the tenant. Retrofitting tenant isolation onto a non-tenant-keyed table is painful enough to be the most common reason for a v2 rewrite.

Senior interview question on real-time stack design

A senior interviewer often opens with: "Design the data pipeline for a real-time analytics product that ingests 100K events/sec from Kafka and serves a sub-second dashboard. Where does ClickHouse sit, what does the Kafka contract look like, and how do you handle a downstream schema change?"

Solution Using a four-component pipeline

-- 1) The Kafka source table (consumer)
CREATE TABLE kafka_events
(
    ts          DateTime CODEC(DoubleDelta, LZ4),
    user_id     UInt64,
    event_type  LowCardinality(String),
    properties  String
)
ENGINE = Kafka
SETTINGS kafka_broker_list = '...',
         kafka_topic_list  = 'events',
         kafka_group_name  = 'ch-prod',
         kafka_format      = 'JSONEachRow';

-- 2) The MergeTree storage table
CREATE TABLE events
(
    ts          DateTime CODEC(DoubleDelta, LZ4),
    user_id     UInt64,
    event_type  LowCardinality(String),
    properties  String
)
ENGINE = MergeTree
PARTITION BY toYYYYMMDD(ts)
ORDER BY (event_type, toStartOfHour(ts), user_id)
TTL ts + INTERVAL 90 DAY;

-- 3) The bridge MV
CREATE MATERIALIZED VIEW events_mv TO events AS
SELECT ts, user_id, event_type, properties
FROM kafka_events;

-- 4) The roll-up MV for the dashboard
CREATE TABLE events_hourly
(
    hour       DateTime,
    event_type LowCardinality(String),
    events     AggregateFunction(count),
    users      AggregateFunction(uniq, UInt64)
)
ENGINE = AggregatingMergeTree
PARTITION BY toYYYYMM(hour)
ORDER BY (event_type, hour);

CREATE MATERIALIZED VIEW events_hourly_mv TO events_hourly AS
SELECT
    toStartOfHour(ts) AS hour,
    event_type,
    countState()      AS events,
    uniqState(user_id) AS users
FROM events
GROUP BY hour, event_type;

Step-by-step trace.

Component	Role	Reads	Writes
`kafka_events`	Kafka consumer	Kafka topic `events`	nothing (cursor only)
`events_mv`	bridge	`kafka_events`	`events` (raw storage)
`events`	raw storage	dashboard ad-hoc	nothing
`events_hourly_mv`	roll-up	`events` on insert	`events_hourly`
`events_hourly`	dashboard surface	dashboard	nothing

When a schema change comes in (e.g. add a column region), the team adds it to kafka_events and events with ALTER TABLE ... ADD COLUMN region String DEFAULT '', then to the MV bodies. ClickHouse can ALTER MATERIALIZED VIEW ... MODIFY QUERY to evolve the body without dropping the target table.

Output:

Dashboard query	Latency
Hourly event count by type (last 30 days)	40–120ms
Unique users per hour by type	60–200ms
Top 10 event types over last day	30–80ms

Why this works — concept by concept:

Separation of concerns — the Kafka engine is the cursor, the MergeTree is the storage, and the AggregatingMergeTree is the dashboard surface. Each component owns one job, so a failure in one does not corrupt the others.
Bridge MV pattern — CREATE MATERIALIZED VIEW ... TO target AS SELECT ... FROM kafka_events is the canonical bridge. It fires on every insert into the source and lands the transformed rows in the target.
Roll-up MV with -State functions — countState() and uniqState() produce partial aggregate states that are stored in AggregatingMergeTree. Background merges roll them up further; queries finalize them with countMerge / uniqMerge.
Schema-evolution safety — the Kafka engine table, the storage table, and the MV all need the column added together. ClickHouse 23+ supports ALTER MATERIALIZED VIEW ... MODIFY QUERY to evolve MVs in place.
Cost — O(events_per_sec) per Kafka batch; O(events × MV_count) for materialized-view fanout; O(unique_groups × state_size) for the aggregating target table. Dashboard cost is O(touched_partitions) on the much smaller roll-up.

SQL
Topic — streaming
Streaming pipeline problems (SQL)

Practice →

3. The MergeTree family — the heart of ClickHouse

MergeTree is one engine, six personalities — the variant you pick is the variant your write pattern needs

The mental model in one line: MergeTree is a columnar table engine that writes immutable on-disk "parts" and merges them in the background according to the ORDER BY key — and the family variants (ReplacingMergeTree, SummingMergeTree, AggregatingMergeTree, CollapsingMergeTree, ReplicatedMergeTree) layer additional semantics onto the merge step. Once you say "the merge is when the variant's magic happens," the entire family becomes a memorisation exercise.

The family in one table.

Engine	Merge-step semantic	Common use
`MergeTree`	none — just sort and merge parts	base columnar table
`ReplacingMergeTree`	dedupe by sort key, keep latest version	CDC upsert sink
`SummingMergeTree`	sum numeric columns by sort key	pre-aggregated counters
`AggregatingMergeTree`	merge `-State` aggregate columns by sort key	materialized-view roll-ups
`CollapsingMergeTree`	collapse `sign = -1` rows against `sign = +1` rows	row-level updates via tombstones
`VersionedCollapsingMergeTree`	same as Collapsing, but with a version column	concurrent CDC streams
`ReplicatedMergeTree` (and variants)	adds Keeper-coordinated replication on top	every production cluster

PARTITION BY vs ORDER BY — two different concepts.

PARTITION BY defines the physical directory structure on disk. Each unique partition expression value is a separate directory. The planner prunes whole partitions before reading anything. Use coarse expressions like toYYYYMM(ts) — fine-grained partitions (e.g. per-hour) create thousands of tiny directories and crater performance.
ORDER BY defines the sort order within a part, and the sparse primary-key index is built on the first N columns. The planner uses it to skip granules (8192-row chunks). Use the highest-cardinality WHERE / GROUP BY columns here in cardinality order.

Parts and merges in plain words.

Every INSERT creates one or more new on-disk parts under the partition directory.
Background merges combine small parts into larger ones, applying the variant-specific semantic during the merge.
A part is immutable — to "update" a row, you write a new part with the new value and let the variant-specific merge resolve.
OPTIMIZE TABLE ... FINAL forces a merge of all parts in a partition. Useful for testing, dangerous in production at scale.

Common interview probes.

"What does MergeTree actually merge?" — parts. Small parts created by inserts are merged into larger parts in the background to keep the part count low.
"What is the difference between ReplacingMergeTree and CollapsingMergeTree?" — Replacing keeps the latest row per sort key; Collapsing requires the writer to emit a +1 row for "current" and a -1 row for "old" — the two collapse during merge.
"What is LowCardinality?" — a string codec that dictionary-encodes the column. Small distinct sets (event_type, status, region) become 1–2 byte integers on disk and in memory.
"What is index_granularity?" — the sparse index granule size (default 8192). Each index entry covers 8192 rows.

Worked example — choosing `MergeTree` vs `ReplacingMergeTree` for a CDC sink

Detailed explanation. A team lands Postgres CDC events into ClickHouse. Each event is a full row image with a primary key. The team wants to query "the current state of every order" — but ClickHouse does not natively update rows. The fix is ReplacingMergeTree: every insert is an append, but the merge step deduplicates by sort key, keeping the latest version.

Question. Build the CDC sink table. The source emits one row per change with order_id, status, amount, and an updated_at timestamp. Show the table definition and the query that returns the "current state" of orders.

Input (rows arriving over time).

insert #	order_id	status	amount	updated_at
1	1	placed	100	2026-06-15 09:00
2	1	shipped	100	2026-06-15 10:00
3	2	placed	50	2026-06-15 11:00
4	1	delivered	100	2026-06-15 12:00

Code.

CREATE TABLE orders_cdc
(
    order_id   UInt64,
    status     LowCardinality(String),
    amount     Decimal(12, 2),
    updated_at DateTime
)
ENGINE = ReplacingMergeTree(updated_at)
ORDER BY order_id;

-- Query for current state — use FINAL or argMax
SELECT order_id, status, amount, updated_at
FROM orders_cdc
FINAL;

-- Or, without FINAL (cheaper, more typing)
SELECT
    order_id,
    argMax(status, updated_at)     AS status,
    argMax(amount, updated_at)     AS amount,
    max(updated_at)                AS updated_at
FROM orders_cdc
GROUP BY order_id;

Step-by-step explanation.

ReplacingMergeTree(updated_at) says "during merges, when two rows share the same ORDER BY key (order_id), keep the one with the greater updated_at."
Between merges, every version of every row is still on disk. A query without FINAL sees all four rows.
FINAL forces the engine to apply the dedup semantic at query time — at the cost of extra read amplification. Fine for dashboards on small tables; expensive on billion-row tables.
The argMax(col, updated_at) GROUP BY order_id pattern is the cheap alternative: it computes the same answer without FINAL, at the cost of a GROUP BY scan.
For the highest-traffic queries, build a downstream materialized view that pre-aggregates the current state into a smaller table.

Output (current state of orders).

order_id	status	amount	updated_at
1	delivered	100	2026-06-15 12:00
2	placed	50	2026-06-15 11:00

Rule of thumb. Use ReplacingMergeTree for CDC sinks where you only ever care about the latest version. Pair it with argMax(...) GROUP BY pk for hot queries; reserve FINAL for low-QPS dashboards and ad-hoc sanity checks.

Worked example — `SummingMergeTree` for a pre-aggregated counter table

Detailed explanation. When the read pattern is "give me the running total per key," and the write pattern is "many small increments," SummingMergeTree collapses the per-key rows during merge — the on-disk size shrinks and reads scan fewer rows.

Question. Build a per-day click counter table where each insert is (day, page_id, +1) and the dashboard reads "clicks per page per day." Show the table and the query.

Input.

day	page_id	clicks
2026-06-15	A	1
2026-06-15	A	1
2026-06-15	B	1
2026-06-15	A	1

Code.

CREATE TABLE clicks_daily
(
    day     Date,
    page_id String,
    clicks  UInt64
)
ENGINE = SummingMergeTree(clicks)
PARTITION BY toYYYYMM(day)
ORDER BY (day, page_id);

INSERT INTO clicks_daily VALUES
    ('2026-06-15', 'A', 1),
    ('2026-06-15', 'A', 1),
    ('2026-06-15', 'B', 1),
    ('2026-06-15', 'A', 1);

-- Read pattern
SELECT day, page_id, sum(clicks) AS clicks
FROM clicks_daily
WHERE day = '2026-06-15'
GROUP BY day, page_id
ORDER BY clicks DESC;

Step-by-step explanation.

SummingMergeTree(clicks) declares that during a merge, rows sharing the same ORDER BY key (day, page_id) collapse into one row whose clicks column is the sum.
Before merge: 4 rows. After merge: 2 rows (A with 3, B with 1).
The read pattern still uses sum(clicks) GROUP BY ... — this is required because between merges, there may still be multiple rows per key. Always GROUP BY + sum, never trust the row count.
The on-disk size approaches the cardinality of (day, page_id) after enough merges — perfect for high-volume counters.
For multi-column counters (e.g. clicks + impressions + revenue), list them all in the engine: SummingMergeTree((clicks, impressions, revenue)).

Output.

day	page_id	clicks
2026-06-15	A	3
2026-06-15	B	1

Rule of thumb. Use SummingMergeTree when every increment is a row and the dashboard wants the sum per key. Pair it with AggregatingMergeTree (next section) when you also need distinct counts, quantiles, or anything beyond plain sum.

Worked example — `CollapsingMergeTree` for row-level updates

Detailed explanation. CollapsingMergeTree is the "I really do need row updates" answer. The writer emits two rows for every logical update: a sign = -1 "cancel" row for the old state, and a sign = +1 "create" row for the new. The merge step pairs them and drops both — leaving only the latest version.

Question. Track an order's current status using CollapsingMergeTree. Show the insert sequence for a "placed → shipped" transition and the dashboard read.

Input (rows emitted by the application).

order_id	status	sign
1	placed	+1
1	placed	-1
1	shipped	+1

Code.

CREATE TABLE orders_collapsing
(
    order_id UInt64,
    status   LowCardinality(String),
    sign     Int8
)
ENGINE = CollapsingMergeTree(sign)
ORDER BY order_id;

-- The writer must emit pair-rows for every update
INSERT INTO orders_collapsing VALUES
    (1, 'placed', +1);
-- ... later, when order ships:
INSERT INTO orders_collapsing VALUES
    (1, 'placed', -1),
    (1, 'shipped', +1);

-- Read pattern uses SUM(sign) trick
SELECT
    order_id,
    argMax(status, sign) AS status
FROM orders_collapsing
GROUP BY order_id
HAVING sum(sign) > 0;

Step-by-step explanation.

Three rows enter the table: (1, placed, +1), (1, placed, -1), (1, shipped, +1).
During merge, the engine pairs the (placed, +1) row with the (placed, -1) row (same order_id and same all-columns-except-sign) and drops both. Only (1, shipped, +1) remains.
Between merges, all three rows are still on disk. The read pattern uses HAVING sum(sign) > 0 to filter out "fully cancelled" keys.
The application must do extra work: read the previous state, emit a cancel row, emit a new row. Often the OLTP source does not know the previous state, which is why Replacing is more common.
Use Collapsing when the application does know the previous state (e.g. the OLTP writes events as (before, after) pairs), or when you need to delete individual rows without rewriting parts.

Output.

order_id	status
1	shipped

Rule of thumb. Reach for CollapsingMergeTree only when the application has a clean "before / after" event source. For the more common "I just have the latest version" pattern, ReplacingMergeTree is simpler — the application emits one row, ClickHouse handles the dedup.

Worked example — `ReplicatedMergeTree` for production HA

Detailed explanation. Every production ClickHouse cluster uses a Replicated*MergeTree engine variant. The replication is coordinated by ZooKeeper or, increasingly, ClickHouse Keeper. Replicas are eventually consistent — a write commits locally, then propagates to peers within milliseconds to seconds.

Question. Convert the single-node events table into a replicated one. Show the engine signature, the Keeper path convention, and how a query reads from any replica.

Input. Single-node table:

CREATE TABLE events (...) ENGINE = MergeTree
ORDER BY (event_type, ts);

Code.

-- Replicated version (per-shard, per-replica)
CREATE TABLE events ON CLUSTER prod
(
    ts         DateTime,
    user_id    UInt64,
    event_type LowCardinality(String),
    value      Float64
)
ENGINE = ReplicatedMergeTree(
    '/clickhouse/tables/{shard}/events',  -- Keeper path: shared per shard
    '{replica}'                            -- replica name: unique per node
)
PARTITION BY toYYYYMMDD(ts)
ORDER BY (event_type, ts, user_id);

Step-by-step explanation.

ReplicatedMergeTree('/clickhouse/tables/{shard}/events', '{replica}') declares that this table participates in replication. The first argument is the Keeper path shared across all replicas of the same shard. The second is the replica name, unique per node.
{shard} and {replica} are macros defined in each node's config.xml. On node-1a they might resolve to shard=01, replica=node-1a.
ON CLUSTER prod runs the DDL on every node in the named cluster — both shards and replicas. Without it, you have to run the CREATE on each node manually.
After creation, every write to one replica is committed locally, then asynchronously replicated to peers via Keeper-tracked log entries.
Reads can hit any replica. The load balancer (or the ClickHouse Distributed engine on top) picks one per query.

Output (cluster topology).

Shard	Replica	Keeper path	Role
01	node-1a	`/clickhouse/tables/01/events`	accept writes, serve reads
01	node-1b	`/clickhouse/tables/01/events`	accept writes, serve reads
02	node-2a	`/clickhouse/tables/02/events`	accept writes, serve reads
02	node-2b	`/clickhouse/tables/02/events`	accept writes, serve reads

Rule of thumb. Use Replicated*MergeTree for every production table without exception. Single-node MergeTree is for development and ETL scratch space only. The cost of switching from single-node to replicated after a year of writes is rebuilding the table.

Senior interview question on choosing the right MergeTree variant

A senior interviewer often opens with: "We are landing Postgres CDC into ClickHouse and want to (a) get the current state of every row, (b) keep a 30-day audit trail, and (c) survive a node failure. Which MergeTree variants do you use and how do you assemble them?"

Solution Using a `ReplicatedReplacingMergeTree` plus an `AggregatingMergeTree` roll-up

-- 1) CDC sink: replicated, dedup-on-merge
CREATE TABLE orders_cdc ON CLUSTER prod
(
    order_id   UInt64,
    status     LowCardinality(String),
    amount     Decimal(12, 2),
    updated_at DateTime
)
ENGINE = ReplicatedReplacingMergeTree(
    '/clickhouse/tables/{shard}/orders_cdc',
    '{replica}',
    updated_at                        -- version column for Replacing
)
PARTITION BY toYYYYMM(updated_at)
ORDER BY order_id
TTL updated_at + INTERVAL 30 DAY;     -- 30-day audit retention

-- 2) Current-state view (logical — uses argMax)
CREATE VIEW orders_current AS
SELECT
    order_id,
    argMax(status, updated_at) AS status,
    argMax(amount, updated_at) AS amount,
    max(updated_at)            AS updated_at
FROM orders_cdc
GROUP BY order_id;

Step-by-step trace.

Requirement	Engine choice	Why
(a) Current state of every row	`ReplicatedReplacingMergeTree(updated_at)`	merge dedupes by `order_id`, keeps the latest `updated_at`
(b) 30-day audit trail	`TTL updated_at + INTERVAL 30 DAY`	older rows drop automatically
(c) Survive node failure	`Replicated...` prefix + Keeper	every write replicated; any replica can serve reads
Hot read of current state	`argMax(...) GROUP BY order_id`	avoids `FINAL` cost on dashboards

After the dedup merge fires, ClickHouse keeps only one row per order_id. The TTL clause drops anything older than 30 days from the audit trail. Replication keeps both sides — current state and audit — symmetrical across replicas.

Output (the orders_current view).

order_id	status	amount	updated_at
1	delivered	100.00	2026-06-15 12:00
2	placed	50.00	2026-06-15 11:00
3	shipped	200.00	2026-06-15 10:30

Why this works — concept by concept:

ReplicatedReplacingMergeTree — combines the replication contract (every write goes to every replica via Keeper-tracked log entries) with the Replacing semantic (dedup by sort key during merge). One engine, two layered concerns.
Version column — ReplicatedReplacingMergeTree(..., updated_at) tells the engine which column tiebreaks duplicates: keep the row with the greatest updated_at. Without it, the engine keeps an arbitrary row, which is rarely what the application wants.
TTL for retention — TTL updated_at + INTERVAL 30 DAY makes the engine schedule background drops for any part whose every row has aged out. No cron job, no DELETE statement.
argMax pattern instead of FINAL — argMax(col, version) GROUP BY pk produces the same answer as SELECT ... FINAL but uses a normal aggregation instead of a full table re-scan at query time. Pay the GROUP BY cost once per query, not the FINAL cost per granule.
Cost — O(parts) for the dedup merge (background); O(unique_keys) for the argMax aggregation; O(replicas) network amplification for the write. Reads on either side scale with the touched-partition byte count, not the row count.

SQL
Topic — time-series
Time-series problems (SQL)

Practice →

4. Materialized views — incremental aggregation engine

ClickHouse materialized views are insert-time triggers, not refresh-on-schedule — the difference is the whole story

The mental model in one line: a ClickHouse materialized view is an INSERT INTO target SELECT ... FROM source that fires every time the source table receives a batch — there is no schedule, no refresh, no cron. Once you internalise "MVs are triggers, not refreshes," the entire materialized-view interview surface (POPULATE, -State functions, cascading MVs) becomes a sequence of obvious follow-ons.

The insert-time MV contract.

The MV is a stored SELECT query plus a target table.
When a batch of N rows arrives in the source, the MV's SELECT runs over that batch only (not the whole source table) and inserts the result into the target.
The MV does not maintain incremental state — it sees one batch, produces one output, and forgets.
"Refreshable" MVs (a 2024+ feature) are a separate construct on a schedule; they are not what an interview means by "materialized view."

The two MV idioms.

TO target_table — the canonical 2024+ pattern. The MV writes into an existing table you defined separately. Backfill is straightforward (INSERT INTO target SELECT ... FROM source).
POPULATE at create time — runs the SELECT once over the existing source data, then enables trigger mode. Convenient for one-shot setup; dangerous on large tables because it locks until done.

-State / -Merge / -MergeState — the aggregate function trinity.

countState(), uniqState(col), sumState(col), quantileState(col) — return a partial-aggregate state object, not a final value. Suitable for storage in an AggregatingMergeTree.
countMerge(state), uniqMerge(state) — finalize a state into a number, typically at query time.
countMergeState(state), uniqMergeState(state) — combine multiple states into one new state. Used in cascading MVs.

Cascading MVs in plain words.

The 1-minute roll-up MV reads from raw_events on insert and writes to agg_1m.
A second MV reads from agg_1m on insert and writes to agg_1h — combining 60 minute-states into one hour-state via *MergeState.
A third MV reads from agg_1h and writes to agg_1d.
Each MV fires only on its source's inserts, so the cascade is incremental from end to end.

Common interview probes.

"Are ClickHouse MVs refreshed on a schedule?" — no. They fire on every source insert.
"What is the difference between -State and -Merge?" — -State produces a partial state for storage; -Merge finalizes a state into a number for reading.
"How do you backfill a new MV with historical data?" — either use POPULATE at create time, or create the MV first (which captures new inserts) then run INSERT INTO target SELECT ... FROM source WHERE ts < cutoff.
"What happens if the source schema changes?" — the MV's SELECT must match the new schema; otherwise the trigger fails. Always evolve the MV body with ALTER MATERIALIZED VIEW ... MODIFY QUERY.

Worked example — a 1-minute pre-aggregation MV

Detailed explanation. A dashboard needs "events per minute, per event_type" with sub-second latency over the last 24 hours. A raw events table at 100K events/sec would force the dashboard to scan 8.6B rows. An AggregatingMergeTree table fed by an MV reduces that to one row per (minute, event_type) — typically a few thousand rows per minute total.

Question. Build the target events_1m table and the MV that maintains it. Show how the dashboard query reads from it.

Input (raw events arriving at high rate).

ts	user_id	event_type	value
2026-06-15 09:00:00.123	1	click	1.0
2026-06-15 09:00:00.456	2	click	1.0
2026-06-15 09:00:05.789	1	view	1.0

Code.

-- Target table for the 1-minute roll-up
CREATE TABLE events_1m
(
    minute     DateTime,
    event_type LowCardinality(String),
    events     AggregateFunction(count),
    users      AggregateFunction(uniq, UInt64),
    value_sum  AggregateFunction(sum, Float64)
)
ENGINE = AggregatingMergeTree
PARTITION BY toYYYYMMDD(minute)
ORDER BY (event_type, minute);

-- The MV that fires on every batch in `events`
CREATE MATERIALIZED VIEW events_1m_mv TO events_1m AS
SELECT
    toStartOfMinute(ts) AS minute,
    event_type,
    countState()        AS events,
    uniqState(user_id)  AS users,
    sumState(value)     AS value_sum
FROM events
GROUP BY minute, event_type;

-- Dashboard query
SELECT
    minute,
    event_type,
    countMerge(events)    AS events,
    uniqMerge(users)      AS users,
    sumMerge(value_sum)   AS value_sum
FROM events_1m
WHERE minute >= now() - INTERVAL 24 HOUR
GROUP BY minute, event_type
ORDER BY minute, event_type;

Step-by-step explanation.

The AggregatingMergeTree target stores partial aggregate states, not finalized numbers. Each column is typed as AggregateFunction(name, arg_types).
The MV's SELECT runs over each insert batch into events. The GROUP BY minute, event_type collapses the batch into one row per (minute, event_type) combination.
countState() produces a tiny state (an integer); uniqState(user_id) produces a HyperLogLog state (~16KB at full density, but compact for small groups); sumState(value) is a single float.
Background merges in AggregatingMergeTree combine states for the same ORDER BY key — turning many small states into one big state per (minute, event_type).
The dashboard reads with *Merge functions, which finalize the states into numbers. The GROUP BY minute, event_type in the read is required because between merges multiple rows per key may still exist.

Output (one row per (minute, event_type) after merges).

minute	event_type	events	users	value_sum
2026-06-15 09:00	click	1240	980	1240.0
2026-06-15 09:00	view	800	720	800.0
2026-06-15 09:01	click	1310	1010	1310.0

Rule of thumb. Always use *State in the MV body and *Merge in the read query. Mixing them ("can I just store count() instead of countState()?") breaks the moment the target table accumulates more than one row per key — which happens after every merge.

Worked example — cascading MVs (1-minute → 1-hour → 1-day)

Detailed explanation. When the dashboard has three zoom levels (last hour at minute resolution, last day at hour resolution, last month at day resolution), the cleanest architecture is a cascade: the 1-minute MV feeds the 1-hour MV, which feeds the 1-day MV. Each MV fires only on its direct source's inserts.

Question. Extend the 1-minute roll-up with 1-hour and 1-day cascade MVs. Show the engine and the chained MV definitions.

Input. The events_1m table from the previous example.

Code.

-- 1-hour target
CREATE TABLE events_1h
(
    hour       DateTime,
    event_type LowCardinality(String),
    events     AggregateFunction(count),
    users      AggregateFunction(uniq, UInt64)
)
ENGINE = AggregatingMergeTree
PARTITION BY toYYYYMM(hour)
ORDER BY (event_type, hour);

-- Cascade MV: events_1m -> events_1h
CREATE MATERIALIZED VIEW events_1h_mv TO events_1h AS
SELECT
    toStartOfHour(minute)        AS hour,
    event_type,
    countMergeState(events)      AS events,
    uniqMergeState(users)        AS users
FROM events_1m
GROUP BY hour, event_type;

-- 1-day target
CREATE TABLE events_1d
(
    day        Date,
    event_type LowCardinality(String),
    events     AggregateFunction(count),
    users      AggregateFunction(uniq, UInt64)
)
ENGINE = AggregatingMergeTree
PARTITION BY toYYYYMM(day)
ORDER BY (event_type, day);

-- Cascade MV: events_1h -> events_1d
CREATE MATERIALIZED VIEW events_1d_mv TO events_1d AS
SELECT
    toDate(hour)              AS day,
    event_type,
    countMergeState(events)   AS events,
    uniqMergeState(users)     AS users
FROM events_1h
GROUP BY day, event_type;

Step-by-step explanation.

events_1m_mv fires on every insert into events and writes the per-minute aggregate states into events_1m.
events_1h_mv fires on every insert into events_1m and rolls 60 minute-states into one hour-state via countMergeState and uniqMergeState.
events_1d_mv does the same one level up — 24 hour-states into one day-state.
The MergeState variants take existing states and combine them, producing a new state (not a finalised number). This is what makes the cascade incremental.
Each cascade level writes 60x less data than the previous — events_1d is roughly events_1m / 1440.

Output (sketch of disk sizes).

Table	Rows per day	Storage
`events`	8.6B	~100 GB
`events_1m`	1440 × distinct event_types	~50 MB
`events_1h`	24 × distinct event_types	~2 MB
`events_1d`	1 × distinct event_types	~100 KB

Rule of thumb. Cascade MVs trade disk for read latency at each level. Three levels (1m / 1h / 1d) is the sweet spot for most dashboards. Beyond that, the operational complexity of maintaining the cascade outweighs the marginal scan reduction.

Worked example — backfilling a new MV without POPULATE

Detailed explanation. POPULATE is convenient but blocks on inserts during the backfill. A cleaner two-step pattern is to (1) create the MV first so it captures new inserts, then (2) backfill historical data with INSERT INTO target SELECT ... FROM source WHERE ts < cutoff.

Question. A new events_1h MV needs to be backfilled with 90 days of history without blocking the live ingest. Show the two-step backfill.

Input. events has 90 days of history. events_1h_mv is the MV. events_1h is the target.

Code.

-- Step 0: pick a cutoff *before* creating the MV.
--    We will backfill rows with ts < cutoff.
--    The MV will catch every row with ts >= cutoff via trigger.

-- Step 1: create the MV (it starts firing on new inserts immediately)
CREATE MATERIALIZED VIEW events_1h_mv TO events_1h AS
SELECT
    toStartOfHour(ts)   AS hour,
    event_type,
    countState()        AS events,
    uniqState(user_id)  AS users
FROM events
GROUP BY hour, event_type;

-- Step 2: backfill historical data in batches
INSERT INTO events_1h
SELECT
    toStartOfHour(ts)   AS hour,
    event_type,
    countState()        AS events,
    uniqState(user_id)  AS users
FROM events
WHERE ts < now() - INTERVAL 24 HOUR    -- safety gap to avoid double-counting
GROUP BY hour, event_type;

Step-by-step explanation.

Step 1: create the MV. From this moment, every new insert into events fires the MV and lands rows in events_1h.
Step 2: explicitly backfill historical rows via INSERT INTO events_1h SELECT ... FROM events WHERE ts < cutoff. The cutoff has a safety gap to avoid double-counting rows that may have been ingested between Step 1 and Step 2.
The AggregatingMergeTree engine handles the overlap automatically — duplicate (hour, event_type) keys merge their states. As long as the cutoff is conservative, the slight overlap is harmless (states combine, not duplicate values).
For very large backfills, run Step 2 in batched ranges (e.g. per partition) to avoid one giant query.
The alternative — CREATE MATERIALIZED VIEW ... POPULATE AS ... — does both steps in one statement but blocks the source from receiving inserts until done. Unacceptable on a live table.

Output.

Phase	Source covered	Notes
Step 1: MV created	`ts >= now()`	every new row triggers it
Step 2: INSERT SELECT	`ts < now() - INTERVAL 24 HOUR`	one-shot backfill in batches
Net coverage	full range, slight overlap at boundary	overlap absorbed by AggregatingMergeTree

Rule of thumb. Never use POPULATE on a live table. The two-step "create MV, then INSERT SELECT" pattern is safer, batchable, and survives the operator pressing Ctrl-C halfway through. Pay the 30 seconds of extra typing.

Worked example — the MV-source-join pitfall

Detailed explanation. A common bug: the MV body joins the source table to another table. The MV only fires on inserts into the source — joins read the target of the join at insert time. If a fact-table insert arrives before its dim-table row, the join misses and the MV writes a row with NULL dim values that never updates.

Question. Diagnose the pitfall in the MV below and propose two fixes.

Input. A events table and a users dim table. The MV joins them.

Code (the broken MV).

CREATE MATERIALIZED VIEW events_enriched_mv TO events_enriched AS
SELECT
    e.ts,
    e.user_id,
    e.event_type,
    u.country,
    u.tier
FROM events e
LEFT JOIN users u ON u.user_id = e.user_id;
-- ^ bug: the join reads `users` at the time the events batch arrives.
--   If the user row is added later, the joined columns stay NULL forever.

Step-by-step explanation.

The MV fires on inserts into events. The LEFT JOIN users is evaluated at that moment.
If events has a row for user_id = 12345 but users does not yet, the LEFT JOIN returns NULL for country and tier. The MV writes NULL into events_enriched.
Later, when users gets the row for 12345, the existing events_enriched row does not automatically update — there is no recalculation. The data is permanently stale.
Fix 1 — Dictionary. Define users as a Dictionary in ClickHouse. Dictionaries are looked up at query time on events_enriched, so the join is fresh on every read.
Fix 2 — defer the enrichment. Drop the join from the MV; do the enrichment at the dashboard query layer with JOIN users or dictGet(...).

Output (the fix using a Dictionary).

CREATE DICTIONARY users_dict
(
    user_id UInt64,
    country String,
    tier    String
)
PRIMARY KEY user_id
SOURCE(CLICKHOUSE(TABLE 'users'))
LIFETIME(MIN 300 MAX 600)
LAYOUT(HASHED());

-- The MV now stores raw events, no join
CREATE MATERIALIZED VIEW events_enriched_mv TO events_enriched AS
SELECT
    ts, user_id, event_type
FROM events;

-- Enrichment happens at query time, against the live dictionary
SELECT
    event_type,
    dictGet('users_dict', 'country', user_id) AS country,
    count()
FROM events_enriched
WHERE ts >= now() - INTERVAL 1 HOUR
GROUP BY event_type, country;

Rule of thumb. Never join in an MV body if the right side of the join can update independently. Use Dictionaries for small dimension tables (refreshed on a TTL) and defer enrichment to the read query when the dim is large or changes frequently.

Senior interview question on materialized-view design

A senior interviewer often frames this as: "Design the materialized-view tree for a real-time analytics product that needs to serve hourly DAU (distinct user count) over the last 30 days with sub-100ms latency. Walk through the engine choices, the -State / -Merge functions, and the backfill plan."

Solution Using an `AggregatingMergeTree` roll-up with HyperLogLog `uniq` state

-- 1) Raw events table
CREATE TABLE events
(
    ts         DateTime,
    user_id    UInt64,
    event_type LowCardinality(String)
)
ENGINE = MergeTree
PARTITION BY toYYYYMMDD(ts)
ORDER BY (event_type, ts, user_id)
TTL ts + INTERVAL 90 DAY;

-- 2) Hourly DAU roll-up target
CREATE TABLE dau_hourly
(
    hour       DateTime,
    event_type LowCardinality(String),
    users      AggregateFunction(uniq, UInt64)
)
ENGINE = AggregatingMergeTree
PARTITION BY toYYYYMM(hour)
ORDER BY (event_type, hour);

-- 3) The MV that fires on every batch of `events`
CREATE MATERIALIZED VIEW dau_hourly_mv TO dau_hourly AS
SELECT
    toStartOfHour(ts) AS hour,
    event_type,
    uniqState(user_id) AS users
FROM events
GROUP BY hour, event_type;

-- 4) Dashboard query
SELECT
    hour,
    event_type,
    uniqMerge(users) AS dau
FROM dau_hourly
WHERE hour >= now() - INTERVAL 30 DAY
GROUP BY hour, event_type
ORDER BY hour;

Step-by-step trace.

Layer	What it does	Latency contribution
`events` (raw)	append-only, partitioned, sorted	not on read path
`dau_hourly_mv` (trigger)	fires on each insert batch into `events`	adds ~1–3% write overhead
`dau_hourly` (target)	AggregatingMergeTree storing HLL states	one row per (hour, event_type) after merges
Dashboard read	`uniqMerge` on ~24×30×N rows	sub-100ms

After 30 days, dau_hourly holds roughly 720 hours × distinct event_types rows. With a dozen event types, that is ~8K rows — a flat scan is microseconds.

Output:

hour	event_type	dau
2026-06-15 08:00	click	124,300
2026-06-15 08:00	view	198,420
2026-06-15 09:00	click	143,100
2026-06-15 09:00	view	211,820

Why this works — concept by concept:

Insert-time MV trigger — the MV fires on every batch into events, not on a schedule. The roll-up table is always up-to-date with the latency of the Kafka consumer (typically 1–10 seconds).
AggregatingMergeTree with uniqState — uniqState(user_id) produces a HyperLogLog state that approximates the unique count. The state is fixed-size (~16KB at full cardinality), so the roll-up table size is bounded by groups × state_size, not by the input row count.
-State at write, -Merge at read — -State produces partial states for storage; -Merge finalizes them at query time. This is the only correct pattern; storing uniq(...) directly would forbid further aggregation.
Schema-evolution path — add a column with ALTER TABLE events ADD COLUMN ..., then ALTER TABLE dau_hourly ADD COLUMN ..., then ALTER MATERIALIZED VIEW dau_hourly_mv MODIFY QUERY .... The MV body change is the one that needs care; storage adds are cheap.
Cost — write amplification per MV roughly equals (input rows / group cardinality) — for a billion-row day into ~24×10 groups, that is a ~4M×N reduction. Read cost on the dashboard is O(touched_rows_in_target) × O(uniqMerge cost), measured in milliseconds for 30-day windows.

SQL
Topic — data aggregation
Aggregation problems (SQL)

Practice →

5. Sharding and replication at scale

Distributed table + Replicated*MergeTree is the production pattern — shards scale write throughput, replicas scale HA and read concurrency

The mental model in one line: a ClickHouse cluster is a grid of shards × replicas, where each shard owns a disjoint slice of the data (by sharding key) and each replica within a shard is a fully synchronized copy — and a Distributed table on top fans queries out across all shards in parallel. Once you can draw the 3×2 grid (3 shards, 2 replicas each), the entire scaling story collapses to "more shards = more write throughput, more replicas = more read HA."

The four building blocks.

Local table — a Replicated*MergeTree on each node. Replicas of the same shard share the same Keeper path; replicas of different shards have different paths.
Distributed table — a thin "fan-out" engine that lives on every node and routes reads and writes across all shards. It owns no data of its own.
Sharding key — a deterministic function (typically cityHash64(user_id)) that maps each row to a shard. Picked once; expensive to change.
ClickHouse Keeper (or ZooKeeper) — the coordination service that sequences replication log entries and DDL.

Sharding key choice.

Hash sharding — cityHash64(user_id). Even distribution across shards; same user always lands on the same shard (useful for per-user joins).
Random sharding — rand(). Perfectly even, but per-user joins must hit every shard via GLOBAL JOIN.
Custom sharding — a domain-specific function (e.g. customer_id % 4). Useful for multi-tenant where you want a specific customer on a specific shard.

Distributed reads.

A SELECT on the Distributed table fans out to every shard. Each shard runs the same query locally on one replica (the load balancer picks).
Partial results stream back to the coordinator, which aggregates and returns.
For joins that need cross-shard data, GLOBAL JOIN sends the right-side table contents to every shard.

Distributed writes.

A write to the Distributed table routes the row to its target shard based on the sharding key.
The Distributed engine can buffer briefly (distributed_directory_monitor_sleep_time_ms) and batch the forwarded writes.
For high-throughput, applications often write directly to the local Replicated*MergeTree on a chosen shard, skipping the Distributed table's overhead.

Replication contract.

A write on one replica is committed locally, then asynchronously propagated to peers via Keeper-tracked log entries.
Replication is eventually consistent — readers may see fresher data on one replica than another for a few seconds during catch-up.
system.replicas and system.replication_queue tables show per-shard replication health.

Common interview probes.

"What is the difference between a Distributed table and a Replicated table?" — Distributed fans queries across shards (horizontal scaling); Replicated synchronizes a single shard's data across replicas (HA).
"Why does ClickHouse use ClickHouse Keeper instead of ZooKeeper?" — same Raft contract, but written in C++ and packaged with ClickHouse, simpler ops.
"Can you change the sharding key online?" — not easily. The common pattern is to write to a new cluster with the new sharding key and dual-write during migration.
"What is GLOBAL IN and when do you need it?" — when a subquery in a Distributed query needs to be evaluated once on the coordinator and broadcast to all shards (rather than executed independently per shard).

Worked example — a 3-shard × 2-replica reference cluster

Detailed explanation. The canonical small-but-real ClickHouse cluster is 3 shards × 2 replicas — 6 nodes total. This is the smallest topology that demonstrates both write fan-out (sharding) and read HA (replication). Most production clusters scale by adding shards (for write throughput) or replicas (for read concurrency).

Question. Define the events table on a 3-shard × 2-replica cluster. Show the per-node local table, the Distributed table on top, and the cluster configuration sketch.

Input (cluster XML config sketch).

Shard	Replicas
01	node-1a, node-1b
02	node-2a, node-2b
03	node-3a, node-3b

Code.

-- 1) Local table on every node (created ON CLUSTER)
CREATE TABLE events_local ON CLUSTER prod
(
    ts         DateTime,
    user_id    UInt64,
    event_type LowCardinality(String),
    value      Float64
)
ENGINE = ReplicatedMergeTree(
    '/clickhouse/tables/{shard}/events_local',
    '{replica}'
)
PARTITION BY toYYYYMMDD(ts)
ORDER BY (event_type, ts, user_id)
TTL ts + INTERVAL 90 DAY;

-- 2) Distributed table on top (also ON CLUSTER)
CREATE TABLE events ON CLUSTER prod AS events_local
ENGINE = Distributed(
    prod,                  -- cluster name from config.xml
    default,               -- database
    events_local,          -- local table
    cityHash64(user_id)    -- sharding key
);

Step-by-step explanation.

events_local is the storage table. Each node has its own copy keyed by {shard} and {replica} macros. Replicas of the same shard share a Keeper path; replicas of different shards do not.
events is the router table. It owns no data; it routes reads and writes across events_local on every shard.
The sharding key cityHash64(user_id) deterministically maps each user_id to a shard. All events for a given user land on the same shard, which makes per-user joins cheap.
ON CLUSTER prod runs the DDL on every node listed under the cluster prod in config.xml. Without it, you'd run the CREATE on each node manually.
Applications can write to events (the Distributed table) for convenience or directly to events_local on a chosen shard for max throughput.

Output (topology).

Node	Shard	Replica	Owns
node-1a	01	r1	shard 01 data (master copy)
node-1b	01	r2	shard 01 data (replicated copy)
node-2a	02	r1	shard 02 data
node-2b	02	r2	shard 02 data
node-3a	03	r1	shard 03 data
node-3b	03	r2	shard 03 data

Rule of thumb. Start every production deployment as 2 shards × 2 replicas (4 nodes). Scale by adding shards when write throughput is the bottleneck; add replicas when read concurrency is. The 3-shard × 2-replica grid is the minimum to demonstrate the pattern in interviews.

Worked example — choosing the sharding key

Detailed explanation. The sharding key choice is one of the most consequential decisions in a ClickHouse cluster. A bad choice causes hot shards (skewed write traffic) or cross-shard joins (slow). The default answer is cityHash64(user_id) for user-facing analytics — even distribution and per-user co-location in one expression.

Question. For each workload below, pick the sharding key and explain in one sentence.

Input.

Workload	Per-row identity	Common query pattern
User-event log	`user_id`	per-user funnel
Time-series metrics	`(metric_name, ts)`	metric over time
Ad impressions	`(campaign_id, user_id)`	campaign-level aggregate
Multi-tenant SaaS	`(customer_id, ...)`	per-customer dashboard

Code.

-- 1) User-event log: hash on user_id for even distribution + per-user co-location
ENGINE = Distributed(prod, default, events_local, cityHash64(user_id));

-- 2) Time-series metrics: hash on metric_name to keep each metric on one shard
ENGINE = Distributed(prod, default, metrics_local, cityHash64(metric_name));

-- 3) Ad impressions: hash on campaign_id, since campaign-level aggregates dominate
ENGINE = Distributed(prod, default, impressions_local, cityHash64(campaign_id));

-- 4) Multi-tenant SaaS: hash on customer_id so each tenant lives on one shard
ENGINE = Distributed(prod, default, events_local, cityHash64(customer_id));

Step-by-step explanation.

user_id hash gives even distribution (assuming user_id is roughly random) and co-locates a user's events on one shard — per-user joins become per-shard joins.
metric_name hash keeps each time-series on one shard. Time-range scans become per-shard scans rather than cross-shard. Watch for a "celebrity metric" — one metric with disproportionate traffic — which would create a hot shard.
campaign_id hash is right when campaigns dominate the read pattern. If a single mega-campaign skews traffic, fall back to (campaign_id, user_id) hash to spread.
customer_id hash gives tenant isolation at the shard level. Large customers can be moved to dedicated shards via cluster reshape; small customers share shards.

Output (trade-off summary).

Sharding key	Even distribution?	Per-key co-location?	Cross-shard joins?
`cityHash64(user_id)`	yes (if user_id random)	yes per user	only for cross-user aggregates
`cityHash64(metric_name)`	yes (if many metrics)	yes per metric	only for cross-metric aggregates
`cityHash64(campaign_id)`	yes (if many campaigns)	yes per campaign	for cross-campaign cohorts
`cityHash64(customer_id)`	depends on customer mix	yes per customer	rarely needed
`rand()`	perfect	no	always

Rule of thumb. The sharding key is "what does my dashboard query group by most often?" If the answer is user_id, hash by user. If the answer is customer_id, hash by customer. Co-location at write time pays off at read time.

Worked example — `GLOBAL IN` for cross-shard subqueries

Detailed explanation. A subquery in a Distributed query is executed per shard by default — every shard runs the subquery independently against its own local data. When the subquery should produce the same result on every shard (e.g. "the top 100 users globally"), use GLOBAL IN: the coordinator runs the subquery once and broadcasts the result to every shard.

Question. Find every event by the top 100 users (by total event count) globally. Show the naive query and the GLOBAL IN fix.

Input. events is a Distributed table over 3 shards.

Code.

-- BROKEN: each shard computes its own "top 100 by local count"
SELECT *
FROM events
WHERE user_id IN (
    SELECT user_id
    FROM events
    GROUP BY user_id
    ORDER BY count() DESC
    LIMIT 100
);

-- FIX: GLOBAL IN — coordinator runs the subquery once, broadcasts result
SELECT *
FROM events
WHERE user_id GLOBAL IN (
    SELECT user_id
    FROM events
    GROUP BY user_id
    ORDER BY count() DESC
    LIMIT 100
);

Step-by-step explanation.

The broken query fans out the outer SELECT to every shard. Each shard then independently runs the inner subquery against its own data — producing 3 different "top 100 by local count" lists.
The outer WHERE on each shard checks user_id IN (local_top_100), which excludes users whose events happen to be on a different shard.
The GLOBAL IN fix changes the execution: the coordinator runs the inner subquery once (which itself fans out to every shard for the aggregation), collects the top 100 globally, then broadcasts that list to every shard for the outer WHERE.
Now every shard filters by the same global top-100 list. The result is what the user actually wanted.
GLOBAL JOIN is the analogous fix for joins where the right side needs to be computed once and broadcast.

Output.

Query	Result
Naive `IN`	each shard's local top 100, no global consistency
`GLOBAL IN`	events by the true global top 100 users

Rule of thumb. Whenever a subquery on a Distributed table should produce a single result for the whole cluster (not per-shard), use GLOBAL IN or GLOBAL JOIN. Without GLOBAL, every shard re-runs the subquery against its own partial data.

Worked example — measuring replication lag

Detailed explanation. Replication in ClickHouse is asynchronous — writes commit locally then propagate. Under normal load, lag is sub-second; under heavy bulk inserts, it can climb to seconds or tens of seconds. Monitoring the gap is the first step toward debugging "the dashboard shows stale data on one replica" tickets.

Question. Write a system-table query that reports per-shard replication lag in seconds. Explain the columns it reads.

Input. A running cluster with events_local on every node.

Code.

SELECT
    database,
    table,
    replica_name,
    is_leader,
    absolute_delay,
    queue_size,
    log_max_index,
    log_pointer
FROM system.replicas
WHERE table = 'events_local'
ORDER BY absolute_delay DESC;

Step-by-step explanation.

system.replicas is the live view of replication health. It exposes one row per replicated table per node.
absolute_delay (seconds) is the time since the most recent unmerged log entry was generated on the leader. Anything > 30 is worth investigating.
queue_size is the count of pending log entries waiting for this replica to apply. A growing queue with steady log_max_index means the replica is falling behind.
log_max_index is the most recent log entry index globally; log_pointer is this replica's local pointer. The difference is the count of pending log entries.
is_leader rotates between replicas of the same shard. Routine reads can hit any replica; some DDL (mutations, drops) goes through the leader.

Output.

database	table	replica_name	is_leader	absolute_delay	queue_size
default	events_local	node-1a	1	0	0
default	events_local	node-1b	0	2	4
default	events_local	node-2a	1	0	0
default	events_local	node-2b	0	15	87

Rule of thumb. Alert on absolute_delay > 30 seconds per replicated table. Alert on queue_size growing for more than 60 seconds. Both indicate that the replica is not keeping up with writes — either because of network, disk, or merge backlog.

Senior interview question on cluster scaling

A senior interviewer often asks: "Your ClickHouse cluster is 3 shards × 2 replicas. Write QPS is doubling every quarter and a single shard is now hitting CPU saturation. Walk me through how you scale, what breaks, and how you keep the dashboard online."

Solution Using a four-step horizontal scale-out plan

-- Step 1: stand up new shards (4 and 5) ON CLUSTER prod
CREATE TABLE events_local ON CLUSTER prod_v2  -- new cluster def includes 5 shards
(
    ts         DateTime,
    user_id    UInt64,
    event_type LowCardinality(String),
    value      Float64
)
ENGINE = ReplicatedMergeTree(
    '/clickhouse/tables/v2/{shard}/events_local',
    '{replica}'
)
PARTITION BY toYYYYMMDD(ts)
ORDER BY (event_type, ts, user_id);

-- Step 2: new Distributed table over the 5-shard cluster
CREATE TABLE events_v2 ON CLUSTER prod_v2 AS events_local
ENGINE = Distributed(
    prod_v2,
    default,
    events_local,
    cityHash64(user_id)
);

-- Step 3: backfill (or dual-write from the Kafka MV bridge)
-- Option A: backfill from old cluster using remote() function
INSERT INTO events_v2
SELECT * FROM remote('prod_old_clusters', default.events_local)
WHERE ts >= '2026-01-01';

-- Option B: dual-write at the MV bridge layer (Kafka -> both clusters)
-- by adding a second MV that targets the new cluster.

-- Step 4: cut the dashboard over to events_v2 and decommission v1

Step-by-step trace.

Step	Action	Risk
1	Stand up 2 new shards with their replicas	new nodes; no data yet
2	Define `events_v2` as Distributed over 5-shard cluster	router only; no traffic yet
3	Backfill or dual-write to populate the new shards	I/O-heavy; do in batches
4	Cut dashboards to `events_v2`, decommission `events`	requires app-config push

The migration is online: the old cluster keeps serving until the dashboard is cut over. The hard part is Step 3 — the backfill must respect the new sharding function so that cityHash64(user_id) % 5 routes rows to the right new shard.

Output (after migration):

Cluster	Shards	Replicas	Write throughput	Dashboard latency
`prod` (old)	3	2	150K events/sec	200–500ms
`prod_v2`	5	2	250K events/sec	100–300ms

Why this works — concept by concept:

Add shards to scale write throughput — each new shard owns a slice of the hash space. Write throughput scales linearly because each shard handles its own partition's inserts and merges.
Add replicas to scale read concurrency and HA — each replica can independently serve reads. Two replicas tolerate one node failure; three tolerate two.
Distributed table is a thin router — it owns no data, so reshaping the cluster (adding shards) does not lose any of the cluster's data when done correctly. The migration risk is in the backfill, not in the router.
Dual-write at the MV bridge — if the Kafka → ClickHouse MV is the only writer, adding a second MV that targets the new cluster gives you dual-write for free during migration. Cut the dashboard, then drop the old MV.
Cost — O(rows × N_replicas) write amplification per shard; O(touched_partitions / shards) read latency reduction per added shard. The migration itself is O(historical_rows / network_throughput).

SQL
Topic — design
System design problems (DE)

Practice →

Cheat sheet — ClickHouse recipes

Default time-series schema. ENGINE = MergeTree PARTITION BY toYYYYMMDD(ts) ORDER BY (entity_id, toStartOfHour(ts), ts) — coarse partition, sort by the most-filtered column then time.
Real-time roll-up. AggregatingMergeTree target + materialized view with -State aggregate functions. Read with *Merge and GROUP BY at query time.
Dedup on CDC. ReplacingMergeTree(version_col) with argMax(col, version_col) GROUP BY pk for hot queries; reserve FINAL for low-QPS dashboards.
Distributed table. ENGINE = Distributed(cluster, db, local_table, cityHash64(shard_key)) — co-locate the most-grouped column on one shard.
Backfill a new MV. Two-step: (1) create the MV (captures new inserts via trigger), (2) INSERT INTO target SELECT ... FROM source WHERE ts < cutoff for history. Avoid POPULATE on live tables.
Schema evolve an MV. ALTER MATERIALIZED VIEW mv MODIFY QUERY ... after adding the column to source and target with ADD COLUMN.
Test cardinality before partitioning. SELECT uniq(col) FROM table LIMIT 1 — if a candidate partition column has > 1000 distinct values, it is too fine-grained for PARTITION BY.
Compress for time-series. CODEC(DoubleDelta, LZ4) on monotonic timestamps; CODEC(T64, LZ4) on bounded integers; CODEC(LZ4HC(9)) for cold data.
Replication health. SELECT replica_name, absolute_delay, queue_size FROM system.replicas WHERE absolute_delay > 30.
Inspect parts. SELECT partition, count() FROM system.parts WHERE active GROUP BY partition — too many parts per partition (>50) indicates merge pressure.
Force a merge (test only). OPTIMIZE TABLE x PARTITION 'YYYYMMDD' FINAL — never run unconditionally in production at scale.
Insert via Kafka engine. Kafka table (ENGINE = Kafka) + MV (TO target) + MergeTree target — the canonical three-object pipeline.
GLOBAL IN for cross-shard subqueries. Whenever the subquery should yield one global result, write WHERE col GLOBAL IN (subquery).
Dictionaries for joins. Define small dimension tables as Dictionary with LAYOUT(HASHED()) + LIFETIME(MIN 300 MAX 600); read with dictGet('dict', 'col', key) for sub-millisecond lookups.

Frequently asked questions

What is ClickHouse used for?

ClickHouse is an open-source columnar OLAP database designed for sub-second interactive analytics over billions of rows. It is the default answer for real-time dashboards, log analytics, event-stream aggregation, ad-tech metrics, and any workload where the read pattern is "aggregate over a column" and the write pattern is "append from a stream or a bulk file." Major deployments at Cloudflare, Uber, ByteDance, and Yandex run ClickHouse at the multi-petabyte scale. It is not a replacement for Postgres / MySQL (no row-level transactions, no point updates) or for Snowflake (slower at heavy multi-join batch). It sits between the stream and the dashboard as the sub-second serving tier.

What is the difference between MergeTree and ReplacingMergeTree?

MergeTree is the base columnar engine — it writes immutable on-disk parts and merges them in the background according to the ORDER BY key. ReplacingMergeTree adds a dedup semantic to the merge: when two rows share the same ORDER BY key, the merge keeps only one of them (the one with the greatest value in an optional version column, otherwise an arbitrary one). Use MergeTree for append-only event streams where every row is unique; use ReplacingMergeTree for CDC sinks where you want "the latest version of every row by primary key." Note that between merges, both versions may exist on disk — production queries pair ReplacingMergeTree with argMax(col, version) GROUP BY pk or with SELECT ... FINAL for the dedup at read time.

How do materialized views work in ClickHouse?

ClickHouse materialized views are insert-time triggers, not refresh-on-schedule snapshots. When you create a materialized view with CREATE MATERIALIZED VIEW mv TO target AS SELECT ... FROM source, the engine fires the SELECT over each insert batch into the source table and writes the result into the target table. There is no schedule, no cron, no full-table refresh. For real-time roll-ups, the target is typically an AggregatingMergeTree that stores partial aggregate states (countState, uniqState, sumState), and the dashboard reads with the matching *Merge functions to finalize the states. The "Refreshable Materialized View" feature added in 2024 is a separate construct that does run on a schedule — but in interviews "materialized view" almost always refers to the insert-time trigger variant.

How does ClickHouse handle updates and deletes?

ClickHouse does not have OLTP-style row updates. The closest equivalents are (1) ALTER TABLE ... UPDATE / DELETE mutations, which rewrite entire affected on-disk parts in the background — fine at low volume but unsuitable for high-frequency point updates; (2) ReplacingMergeTree with a version column, which lets the writer emit a new row per version and the merge dedupes at the sort-key level; (3) CollapsingMergeTree, which collapses paired +1 / -1 rows during merge; and (4) ALTER TABLE ... DROP PARTITION, which is the cheapest way to delete a coarse range (e.g. GDPR-driven cohort deletion at month granularity). If the workload demands frequent point updates, you are using the wrong tool — Postgres or a key-value store is the right answer, and ClickHouse becomes the analytical mirror downstream via CDC.

ClickHouse vs Snowflake — which one for real-time analytics?

For interactive sub-second dashboards over append-heavy data, ClickHouse is the strong default — its columnar storage, vectorised execution, and materialized-view roll-ups land query latencies in the 50–500ms range that Snowflake's compute-warehouse model cannot match without aggressive caching. For batch ELT, long-tail analytics, complex multi-join workloads, and ad-hoc SQL across many domains, Snowflake is the strong default — its separation of storage and compute, mature dbt integration, and seconds-to-minutes latency budget fit the batch pattern. The most common production deployment is both: ClickHouse for the real-time speed lane (kept hot for 30–90 days), Snowflake for the batch warehouse (kept for 5+ years). Pick the one that matches the latency contract, not the cost contract.

Do I need ZooKeeper to run ClickHouse?

For single-node ClickHouse (development, ETL scratch space, small dashboards) — no. For any production cluster with Replicated*MergeTree tables — yes, you need either ZooKeeper or ClickHouse Keeper as the coordination service. ClickHouse Keeper is a Raft-based, C++-implemented drop-in replacement that ships with ClickHouse and is the recommended choice for new clusters since 2023; it can be deployed standalone or co-located on ClickHouse nodes. ZooKeeper remains supported and is the right choice if your organisation already operates a ZooKeeper ensemble. Either way, the coordination service sequences replication log entries, DDL queries, and distributed leadership — without it, replication, ON CLUSTER DDL, and distributed mutations cannot function.

Practice on PipeCode

Drill the real-time analytics practice library → for the dashboard-latency and roll-up family of problems.
Rehearse on aggregation problems → when the interviewer wants GROUP BY with multiple aggregates.
Sharpen the time-axis with time-series practice drills → for toStartOfHour / toStartOfDay and the partition-pruning patterns.
Layer the data aggregation library → for materialized-view-style roll-up problems.
Stack the streaming pipeline library → for Kafka → sink contract questions.
For the broader surface, read top data engineering interview questions →.
Stack the prerequisites with the only 5 skills you need to become a data engineer →.
Sharpen the SQL axis with the SQL for data engineering interviews course →.
For long-form schema craft, work through data modelling for DE interviews →.
For the ELT system-design axis, study the ETL system design course →.

Pipecode.ai is Leetcode for Data Engineering — every ClickHouse recipe above ships with hands-on practice rooms where you write the MergeTree table definition, the AggregatingMergeTree roll-up MV, and the Distributed-table sharding key against graded inputs that mirror real production schemas. PipeCode pairs every reading with 450+ DE-focused problems and a real-time scoring engine, so you never have to wonder whether your `cityHash64(user_id)` choice actually balances the shards or whether your `uniqState` / `uniqMerge` pairing returns the correct DAU.

Practice real-time analytics now →
Time-series drills →

Trino vs Presto vs Athena: Federated SQL Engines for the Modern Lakehouse

Gowtham Potureddi — Wed, 17 Jun 2026 13:02:43 +0000

trino vs presto looks like a single-word product comparison to a junior — interviewers know it is actually a twelve-year lineage question that splits Facebook's 2012 Presto into PrestoDB (Meta + Linux Foundation) and Trino (Starburst), with AWS Athena straddling both engines and Starburst layering commercial governance on top. The result is the most under-explained question in the modern lakehouse stack: four engine names, one shared architecture, and four very different operational, cost, and connector profiles that every senior data engineer is expected to defend at an interview whiteboard.

This guide is the cheat sheet that decodes the entire family. It walks through the trino presto athena lineage, the coordinator + workers + connectors architecture that every distributed sql engine shares, the connector ecosystem that drives real query federation, the athena vs presto cost-versus-utilisation decision, and the trino interview questions interviewers love to probe — the prestodb vs trino rename, predicate pushdown, cross-source joins, and when fault-tolerant execution actually earns its keep. Each section pairs a teaching block with a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the SQL practice library →, rehearse on joins practice →, and stack the query-shape muscles with aggregation problems →.

On this page

Why federated SQL became the lakehouse default
The Presto → Trino → Athena lineage
Architecture compared — coordinator, workers, connectors
Connector ecosystem & federation patterns
Cost, performance & when to pick which
Cheat sheet — Trino vs Presto vs Athena recipes
Frequently asked questions
Practice on PipeCode

1. Why federated SQL became the lakehouse default

Federated SQL is the answer to "the data lives in seven places" — the lakehouse needs a query engine, not another database

The one-sentence invariant: a federated SQL engine pushes a single SQL statement across many storage backends without copying data into one warehouse first — that is the precise property the lakehouse pattern depends on, and the precise reason Trino, PrestoDB, and Athena exist as a family rather than as competitors to Snowflake or BigQuery. Once you internalise that "query travels to data, not data to query," the entire trino vs presto interview surface stops being a feature checklist and becomes a single architectural argument.

The three pressures that produced federated SQL.

Data lake explosion. By 2018 every large company had at least one petabyte-scale object store (S3, ADLS, GCS) that no traditional warehouse could read natively without a copy step. The "load it into Redshift first" answer stopped fitting in the budget and the SLO.
Multi-source analytics. Product analytics, finance ledgers, CRM exports, event streams, and ML feature stores started living in different engines (Postgres, MySQL, Kafka, Iceberg, Elasticsearch). Stitching them into one report meant either a nightly ETL run or a query engine that could fan out reads in flight.
Decoupled storage and compute. The S3-as-table-storage thesis (table formats: Hive, Iceberg, Delta, Hudi) turned storage into a flat layer that any compute engine could attach to. The natural next step was a thin SQL engine on top — no storage, no transactions, just plan-execute-return.

Federated SQL vs traditional MPP — the contract.

MPP warehouse (Snowflake, BigQuery, Redshift). Owns its storage format, its statistics, its transaction log, its compute cluster, and its catalog. Optimal when every byte you query lives inside that warehouse.
Federated SQL engine (Trino, PrestoDB, Athena). Owns only the query coordinator and the compute. Storage, statistics, transactions, and catalogs all live in the connector — the engine asks each connector "what can you push down, what statistics do you have, how do I split this scan?" and dispatches accordingly. This is why a single Trino SELECT can join a Postgres customers table to an Iceberg events table to a Kafka clicks topic in one statement.

Why "query engine, not database" is the right framing.

No durable storage of its own. The engine has memory and scratch disk only. Catastrophic worker loss costs the in-flight query, not the data.
No transactional commit semantics. Writes are delegated to the connector (the Iceberg connector writes Iceberg snapshots; the Hive connector writes Parquet files into Hive partitions).
No proprietary statistics. Cost-based optimisation reads stats from the catalog (Hive metastore, Glue, Iceberg metadata) — there is no "ANALYZE the warehouse" step the engine owns.

The four engines you actually compare in 2026.

Trino. The post-2019 fork led by Starburst-affiliated maintainers. Monthly release cadence. Most aggressive connector and execution roadmap. The default open-source answer.
PrestoDB. The Meta + Linux Foundation continuation of the 2012 Facebook code base. Quarterly releases. Strong on Spark integration (Presto on Spark) and RaptorX hot-data caching.
AWS Athena. Serverless managed engine. Engine v2 was PrestoDB; engine v3 (2023+) is Trino. No cluster to run. Per-query pricing.
Starburst. Commercial Trino distribution with governance, caching (Warp Speed), and an enterprise control plane. Often the "we want Trino with a support contract" answer.

The 2026 reality.

Trino has won the open-source mindshare but Athena is the deployment majority for AWS-anchored shops because the per-query pricing kills the operational tax for spiky workloads.
Iceberg has become the assumed table format for new lakehouses — every engine in this family ships first-class Iceberg connectors.
Fault-tolerant execution (Trino) and RaptorX caching (PrestoDB) finally make "long-running ETL on a query engine" feasible, narrowing the gap with Spark for analytical workloads.

Worked example — when federated SQL is the wrong choice

Detailed explanation. A common interview opener is "would you put Trino in front of an OLTP database for low-latency dashboards?" The correct answer is no — every federated SQL engine pays a coordinator-planning tax that makes sub-100ms queries unrealistic, and every backing connector becomes a bottleneck when the workload pattern is "thousands of small reads per second." Federated SQL was designed for analytical workloads — large scans, complex joins, multi-source — not for transactional workloads.

Question. A team wants to run a customer-facing dashboard with 200 ms p95 latency over a Postgres OLTP database. Should they front Postgres with Trino? Why or why not?

Input — workload characteristics.

Workload	Query rate	Rows scanned	Latency target
Customer dashboard	5,000 qps	50 rows / query	< 200 ms p95
Analyst exploration	5 qps	1B rows / query	< 30 s p95

Code — what not to do.

-- Anti-pattern: putting Trino in front of OLTP
-- A user-facing dashboard query hits Trino, which hits Postgres via JDBC connector.
-- Every request pays: coordinator planning + JDBC round trip + result marshalling.
SELECT customer_id, plan, last_login_at
FROM postgres.public.customers
WHERE customer_id = ?;

Step-by-step explanation.

The Trino coordinator parses the SQL, plans it, dispatches a single split to one worker — every step costs tens of milliseconds even when the underlying query is trivial.
The worker opens a JDBC connection to Postgres (or borrows one from the connector pool) and issues the same SELECT. The Postgres query itself returns in 5 ms.
The result is serialised back through the worker to the coordinator, then to the client. The hop adds another 10 ms.
At 5,000 qps you are running thousands of JDBC connections per second and burning coordinator CPU on planning overhead for tiny queries.
The right answer is "skip Trino — let the dashboard read Postgres directly, or read a materialised aggregate cache (Redis, ClickHouse, a denormalised replica)."

Output.

Approach	Median latency	p95	Notes
Direct Postgres	5 ms	15 ms	Native, designed for this
Trino in front	80 ms	250 ms	Coordinator + JDBC overhead
ClickHouse cache	8 ms	25 ms	Best for high-qps analytics

Rule of thumb. Federated SQL engines are analytical engines. If the workload is "thousands of small queries per second with sub-100 ms latency," reach for the source database directly or a purpose-built serving layer. Trino, PrestoDB, and Athena all shine when query rate is low and scan volume is high — exactly the inverse profile.

Worked example — the "before federated SQL" pipeline

Detailed explanation. A second favourite interview probe: "what did teams do before Trino existed?" The honest answer is "they ran nightly ETL jobs that copied every source into a single warehouse, then queried the warehouse." Recognising the pattern that federated SQL replaces makes the value of the engines obvious, and surfaces the cost trade-off you accept by adopting one.

Question. Sketch the data flow for a single dashboard that needs Postgres customer data joined to S3 event data, before and after a federated SQL engine. What changes?

Input — what the dashboard needs.

Source	Live in	Daily volume
customers	Postgres OLTP	50 GB
events	S3 / Iceberg	5 TB

Code.

-- BEFORE — nightly ETL approach
-- 1) Extract Postgres customers via CDC into S3 staging
-- 2) Run Spark ETL to land into the warehouse (Snowflake)
-- 3) Run Spark/Glue to land events into the warehouse
-- 4) Run the join inside Snowflake
SELECT c.plan, COUNT(*) AS events
FROM warehouse.customers c
JOIN warehouse.events e ON e.customer_id = c.customer_id
GROUP BY c.plan;

-- AFTER — federated SQL via Trino
-- Same SQL, no ETL — two connectors, one statement
SELECT c.plan, COUNT(*) AS events
FROM postgres.public.customers c
JOIN iceberg.lakehouse.events e ON e.customer_id = c.customer_id
GROUP BY c.plan;

Step-by-step explanation.

The BEFORE pipeline pays an "ETL latency tax" — yesterday's events join yesterday's customers because both arrive in the warehouse on a nightly schedule.
It also pays a "storage duplication tax" — every byte lives once in Postgres or S3 and once in Snowflake.
The AFTER pipeline drops both taxes. The Trino coordinator reads customers live from Postgres and events live from Iceberg, joins them in flight, and returns the aggregate.
The cost trade-off: every dashboard hit re-reads Postgres and S3. If the dashboard runs thousands of times a day, the cumulative read cost on the source systems may exceed the cost of materialising the join nightly.
The right hybrid is "federated SQL for ad-hoc exploration, materialised tables for hot dashboards" — Trino is the discovery tool, the warehouse is the serving tool.

Output.

Pattern	Freshness	Source load	Storage cost
Nightly ETL	24 h stale	low (one read/day)	2x (copy in warehouse)
Federated SQL	live	high (per query)	1x (no copy)

Rule of thumb. Federated SQL trades fresh data and storage savings for source-system read load. Use it where freshness matters and read volume is low; materialise the result where read volume is high and freshness can lag.

SQL interview question on federated SQL fundamentals

A senior interviewer often opens with: "Explain when you'd reach for Trino instead of Snowflake. Walk me through one concrete workload where Trino wins and one where Snowflake wins, and explain the architectural reason — not just the price tag."

Solution Using the federated-vs-MPP decision frame

-- Federated SQL wins: live cross-source join, no ETL latency
-- Trino reads two backends in flight, joins in compute, returns aggregate
SELECT
    c.plan,
    DATE_TRUNC('day', e.event_ts) AS day,
    COUNT(*)                       AS events,
    COUNT(DISTINCT e.customer_id)  AS active_customers
FROM postgres.crm.customers c
JOIN iceberg.lake.events    e
       ON e.customer_id = c.customer_id
WHERE e.event_ts >= CURRENT_DATE - INTERVAL '7' DAY
GROUP BY c.plan, DATE_TRUNC('day', e.event_ts);

-- MPP warehouse wins: 50 concurrent users hitting the same dashboard
-- Snowflake's pre-aggregated micro-partitions + result cache are the right primitive
-- This query would punish Trino by re-reading both backends per request
SELECT plan, total_revenue, active_customers
FROM analytics.daily_dashboard_v1
WHERE day = CURRENT_DATE;

Step-by-step trace.

Workload	Engine	Why
Live multi-source join, 1 query	Trino	No ETL needed; engine fans out to sources
Pre-aggregated dashboard, 50 qps	Snowflake	Result cache + micro-partition prune
1B-row ad-hoc on Iceberg	Trino	Native Iceberg, no warehouse load step
Hourly KPI batch	Snowflake	Tasks + streams own the warehouse pattern

The trace highlights the architectural divide: Trino is the right primitive when the query plan crosses backends or the data lives in S3 and must not be copied; the warehouse is the right primitive when the workload is stable, high-concurrency, and pre-aggregated.

Output:

Pattern	Winner	Reason
Cross-source ad-hoc	Trino	Federation, no copy
Iceberg ad-hoc	Trino	First-class connector
High-concurrency BI	Snowflake	Result cache, MPP
Hourly batch	Snowflake	Tasks, streams, transactions

Why this works — concept by concept:

Federated SQL trades freshness for source load — every Trino query is a live read against the backend, so you pay source-system load for every dashboard hit. Materialising the result trades freshness for cost.
MPP storage owns statistics — Snowflake's micro-partition statistics live inside the warehouse, which is what powers its concurrency. A query engine without owned storage relies on the connector's stats — often weaker.
Result cache vs cluster cache — Snowflake's result cache returns identical queries in milliseconds; Trino's per-cluster cache (or PrestoDB RaptorX) caches data blocks, not results, so warming the cache requires similar scans.
No transactional contract — Trino does not own a transaction log, so writes happen at the connector layer (Iceberg snapshots, Hive directory renames). Cross-source writes are not atomic.
Cost — Trino: O(rows scanned per query × source cost). Snowflake: O(rows × credit) + cache hit factor. The crossover is workload concurrency and read frequency, not "engine speed."

SQL
Topic — SQL fundamentals
SQL practice library

Practice →

2. The Presto → Trino → Athena lineage

One Facebook project became three engine names — the lineage is the most-asked Trino interview question

The mental model in one line: Presto was born at Facebook in 2012; the original maintainers forked it in 2019 as PrestoSQL (renamed Trino in 2020); Meta and the Linux Foundation kept the original code base under the PrestoDB name; AWS Athena moved from PrestoDB (engine v2) to Trino (engine v3). Once you can say that fluently, the rest of the prestodb vs trino interview surface is a Q&A on motivation, licensing, and roadmap rather than a knowledge gap.

The 12-year lineage in one paragraph.

2012 — Facebook ships Presto. Built to run interactive SQL over Hive warehouses where Hive was too slow. Open-sourced in 2013. The early connectors target Hive and a handful of relational sources.
2019 — the fork. The original founders (Martin Traverso, Dain Sundstrom, David Phillips) leave Facebook and form Starburst Data. They fork the project as PrestoSQL and continue active development. Facebook keeps the original code base under the PrestoDB name and donates it to the Linux Foundation as the Presto Foundation.
2020 — PrestoSQL becomes Trino. A trademark dispute resolves with the fork rebranding to Trino. The codebase, releases, and community move under the new name.
2021 — Athena engine v2. AWS Athena (launched 2016 on PrestoDB) is still on PrestoDB at this point.
2023 — Athena engine v3. AWS moves Athena to Trino. The migration is transparent to the SQL surface, but unlocks faster release cadence and the Trino connector ecosystem.
2024–2026 — Trino dominates the open-source narrative. Monthly release cadence vs PrestoDB's quarterly cadence; far broader connector coverage; Starburst's commercial offering grows around governance and caching.

Why the rename matters in interviews.

License clarity. Trino and PrestoDB are both Apache 2.0 — same OSS license — but they are different code bases with different release cadences and roadmaps. A query that works on Trino 435 may not parse on PrestoDB 0.288.
Connector parity. Trino tends to ship connector enhancements months earlier (Iceberg writes, Delta UniForm support, materialised views). PrestoDB catches up on the major ones eventually.
Cloud distribution. Athena's move to engine v3 (Trino) is a strong industry signal — the largest managed deployment of either engine chose Trino as its forward platform.

The release cadence comparison.

Engine	Cadence	License	Governance
Trino	Monthly	Apache 2.0	Trino Software Foundation (Starburst-led)
PrestoDB	Quarterly	Apache 2.0	Presto Foundation (Linux Foundation)
Athena	Continuous (managed)	Proprietary AWS service	AWS
Starburst Enterprise	Quarterly	Commercial (Trino core)	Starburst Data

Common interview probes on the lineage.

"What's the difference between Presto and Trino?" — same DNA, different forks since 2019. Trino is the active fork; PrestoDB is the Meta-anchored continuation. Trino has the faster release cadence and broader connector ecosystem.
"Is Athena Presto or Trino?" — both, depending on era. Engine v2 was PrestoDB; engine v3 (2023+) is Trino.
"Why did the fork happen?" — governance disagreement and the founders wanting to ship faster than Facebook's review cycle allowed. The trademark dispute later forced the PrestoSQL → Trino rename.
"Is Starburst the same as Trino?" — Starburst Enterprise is a commercial distribution built on top of Trino, with added governance, caching (Warp Speed), and support. Trino is the core; Starburst is the wrapper.

Worked example — picking the right engine name in an interview answer

Detailed explanation. The classic mistake is to say "Presto" when you mean Trino, or "PrestoDB" when you mean Athena engine v3. Interviewers track the precision because it correlates with real platform choices — saying "we use Presto" without specifying which fork tells them you have not actually compared the engines. The fix is to always name the fork and the version.

Question. A platform team says "we run Presto on AWS." Which of the four engines do they actually mean, and what follow-up question would you ask to disambiguate?

Input — what "we run Presto on AWS" can mean.

Phrase used	Likely engine	Disambiguating follow-up
"Presto on EMR"	PrestoDB (EMR's default)	"EMR version? PrestoDB or Trino flavour?"
"Presto on EKS"	Trino (community charts target Trino)	"What's the connector for the lake?"
"Athena"	Trino (engine v3)	"Are you on engine v2 or v3?"
"Starburst Galaxy / Enterprise"	Trino + Starburst	"Self-hosted or Galaxy SaaS?"

Code — making the difference real with a version probe.

-- Trino — version probe
SELECT node_version FROM system.runtime.nodes LIMIT 1;
-- Returns: 435 (or similar three-digit Trino release)

-- PrestoDB — version probe
SELECT version() AS presto_version;
-- Returns: 0.288 (or similar 0.xxx PrestoDB release)

-- Athena — version probe in the AWS console
-- Workgroup settings show "Engine version 2" or "Engine version 3"

Step-by-step explanation.

Ask "what is the version string?" — Trino versions look like 435; PrestoDB versions look like 0.288. The format alone disambiguates.
If the answer is "Athena," ask "engine v2 or v3?" — that maps directly to PrestoDB vs Trino under the hood and changes everything from connector support to SQL dialect quirks.
If the answer is "Starburst," ask "Galaxy or Enterprise?" — Galaxy is the SaaS multi-tenant; Enterprise is self-hosted. Both wrap Trino, but operational responsibility differs.
With the engine pinned, you can speak confidently about connector availability, feature flags, and known issues — without the engine pinned, every answer is a guess.

Output.

Engine	Version format	Fork
Trino	three-digit (e.g. 435)	Starburst-led OSS
PrestoDB	0.xxx (e.g. 0.288)	Meta + Linux Foundation
Athena v3	"Engine version 3"	Trino under the hood
Athena v2	"Engine version 2"	PrestoDB under the hood

Rule of thumb. Every time someone says "Presto," ask "which one?" — the conversation is more productive after that. In interview answers, name the fork and the major version every time you reference a feature; that one habit signals "I have actually shipped on these engines" louder than any benchmark number.

Worked example — what the fork unlocked

Detailed explanation. A frequent interview probe is "what did the Trino fork actually enable that PrestoDB does not have?" The honest answer is release velocity plus a more aggressive execution roadmap — dynamic filtering, fault-tolerant execution (Project Tardigrade), and a much faster cadence on table-format support (Iceberg, Delta, Hudi writes). PrestoDB has its own innovations (RaptorX caching, Presto on Spark) but ships them quarterly.

Question. Sketch a side-by-side feature comparison for Trino vs PrestoDB as of 2026, focusing on the three or four features interviewers commonly probe.

Input — feature axes that matter in practice.

Axis	Trino	PrestoDB
Release cadence	monthly	quarterly
Iceberg writes	mature (table maintenance, MERGE)	available, slower roadmap
Fault-tolerant execution	yes (Project Tardigrade)	partial (Presto on Spark)
Caching	community plugins + Starburst Warp Speed	RaptorX (built-in)
Connector breadth	wider	narrower
MPP join enhancements	dynamic filtering, AQE	AQE-style improvements

Code — a feature flag check.

-- Trino: enable fault-tolerant execution at session level
SET SESSION retry_policy = 'TASK';

-- Run a long ETL query; failed tasks are retried instead of failing the query
INSERT INTO iceberg.lake.daily_kpis
SELECT day, plan, COUNT(*) AS events
FROM iceberg.lake.events
WHERE event_ts >= CURRENT_DATE - INTERVAL '1' DAY
GROUP BY day, plan;

-- PrestoDB equivalent: route the query through Presto on Spark for resiliency
-- This is a deployment switch, not a session setting

Step-by-step explanation.

The Trino SET SESSION retry_policy = 'TASK' enables Project Tardigrade — a worker failure no longer kills the whole query; the failed task retries on another worker.
PrestoDB's nearest equivalent is to run long queries on the "Presto on Spark" runtime, which inherits Spark's task-level resilience.
The trade-off is operational: Trino's TFE is in-process; Presto on Spark is a separate execution path with its own driver setup.
For a typical ETL workload that runs 10–60 minutes, TFE is the easier setup; for hours-long jobs that already have a Spark pipeline, Presto on Spark is the natural fit.

Output.

Scenario	Trino	PrestoDB
5-min interactive query	sub-second feature parity	sub-second feature parity
30-min ETL with worker churn	TFE retries tasks	needs Presto on Spark
Iceberg writes	first-class MERGE / OPTIMIZE	available, less polished

Rule of thumb. For greenfield builds in 2026, Trino is the default open-source pick unless you have a specific reason to stay on PrestoDB (existing investment, EMR contract, RaptorX cache savings). For managed deployments, Athena is the right starting point on AWS unless cluster utilisation is high enough to justify self-hosting Trino on EKS.

SQL interview question on the Trino / Presto / Athena lineage

A senior interviewer often frames this as: "Tell me the lineage of Trino, Presto, and Athena. Why does it matter for SQL portability? Walk me through one query that behaves differently on each."

Solution Using a SQL dialect probe across the family

-- A query that exposes the three dialect quirks
SELECT
    user_id,
    -- 1) ARRAY indexing — Trino & PrestoDB are 1-indexed; Athena inherits whichever engine version
    tags[1] AS first_tag,
    -- 2) DATE arithmetic — Trino prefers INTERVAL '7' DAY; PrestoDB allows both DATE + 7 forms
    CURRENT_DATE - INTERVAL '7' DAY AS week_ago,
    -- 3) Lambda / higher-order functions — same syntax across all three
    REDUCE(amounts, 0, (s, x) -> s + x, s -> s) AS total_spend
FROM events_table;

Step-by-step trace.

Quirk	Trino	PrestoDB	Athena v2	Athena v3
`array[1]`	1-indexed	1-indexed	1-indexed	1-indexed
`DATE - INTERVAL '7' DAY`	yes	yes	yes	yes
`DATE - 7`	no	yes	yes	no
`REDUCE(...)`	yes	yes	yes	yes
`LISTAGG`	yes (since 421)	no	no	yes (v3)
`MERGE` on Iceberg	yes	newer	no (v2)	yes (v3)

The trace highlights that the common subset of SQL works everywhere, but the long tail (LISTAGG, MERGE on table formats, certain window-function frame syntaxes) diverges. The senior signal is to write portable SQL by default and reach for engine-specific sugar only when you must.

Output:

Query family	Portable answer
Aggregations	`SUM`, `AVG`, `COUNT(DISTINCT ...)` — works everywhere
Window functions	`ROW_NUMBER() OVER (PARTITION BY ... ORDER BY ...)` — universal
Date arithmetic	`INTERVAL '7' DAY` — portable; avoid `DATE - integer`
Array index	`arr[1]` — portable; avoid `element_at` quirks

Why this works — concept by concept:

Same lineage = same SQL core — every engine in the family parses the same ANSI-style SQL surface for SELECT / JOIN / WHERE / GROUP BY / ORDER BY plus the Presto-specific extensions (UNNEST, lambda, complex types).
Divergence lives at the edges — date arithmetic, certain aggregates (LISTAGG, GROUP_CONCAT), and MERGE / UPSERT semantics on table formats are where the engines drift apart.
Athena inherits its engine version — v2 quirks match PrestoDB of the same era; v3 quirks match Trino. Re-test queries when AWS announces engine upgrades.
Connector quirks compound__ — even when the SQL parses, the Iceberg connector on Trino may support a write mode the same connector on PrestoDB does not. Always check connector docs against the engine version.
Cost — engine-portable SQL has zero runtime cost; the cost is the discipline of not reaching for non-portable sugar.

SQL
Topic — case expression
CASE expression problems (SQL)

Practice →

3. Architecture compared — coordinator, workers, connectors

Every federated SQL engine is a coordinator + N workers + N connectors — the differences live in execution and ops, not topology

The mental model in one line: one coordinator parses and plans the SQL; many workers execute the splits; many connectors translate plan fragments into reads against external storage — Trino, PrestoDB, and Athena are all built that way, and naming the topology cold is the table-stakes architecture answer interviewers expect. The differences live in how tasks are scheduled, how failures are handled, and how the operator deploys the cluster — not in the topology itself.

The shared core in detail.

Coordinator. Single JVM (per cluster) that owns: SQL parsing, semantic analysis, query planning, cost-based optimisation, split generation, task scheduling, and the metadata API the catalog talks to. Every client connection lands at the coordinator.
Workers. Stateless JVMs that execute the plan fragments handed to them. A worker holds the in-flight data for its share of the query in memory and on local scratch disk; if the worker dies, the in-flight data dies with it (unless fault-tolerant execution is enabled).
Connectors. Pluggable Java modules implementing the SPI (Service Provider Interface). Each connector knows how to list tables in its catalog, fetch column metadata, generate splits, push down predicates, and read / write rows. The connector is the only thing that knows about S3, Hive, Iceberg, MySQL, Kafka, etc. — the engine itself is storage-agnostic.

Trino-specific architecture.

Dynamic filtering. When a small join build side is materialised, the coordinator broadcasts the inferred predicate ("only customer_ids in this set") to the probe side's scan; the scan uses it as a runtime filter to skip rows before reading. Huge wins on star-schema joins.
Fault-tolerant execution (Project Tardigrade). Long queries can run with retry_policy = 'TASK' (task retries) or 'QUERY' (whole-query retries). Intermediate data spills to a shared "exchange" (S3 / HDFS / Azure) so a failed worker no longer kills the query.
Adaptive Query Execution (AQE). Some query reshaping happens after the first stage materialises — the planner re-decides whether to broadcast or shuffle a join based on actual row counts.
Connector cadence. Trino is the first engine to land most new connector features (Iceberg writes, Delta UniForm, Hudi read-on-write, materialised views over external catalogs).

PrestoDB-specific architecture.

RaptorX (hierarchical caching). Built-in caching layer with file-list cache, file-handle cache, file fragment cache, and metastore cache. Strong for repeated scans against the same Hive/Iceberg partition set.
Presto on Spark. A deployment mode where Presto's planner emits a query that runs as a Spark application — inheriting Spark's task-level retry and elasticity. Best for very long ETL queries.
Native execution (Velox / Prestissimo). The C++ vectorised execution engine, originally a Meta project, that PrestoDB is gradually adopting as its worker runtime for big perf wins on CPU-bound workloads.

Athena-specific architecture.

Serverless coordinator + workers. There is no cluster — AWS manages the pool of coordinators and workers and routes the query to whichever capacity is available. The user sees only a workgroup, a database catalog, and a per-query bytes-scanned charge.
Per-query pricing. $5 per terabyte of compressed data scanned (varies by region) — that single metric replaces "how big is my cluster?"
Workgroups. Logical buckets for billing, query limits, and engine version pinning. A v2 workgroup runs PrestoDB; a v3 workgroup runs Trino.
Athena Federation. A Lambda-backed extension that adds connectors AWS does not natively support — implemented as user-deployed Lambda functions the engine calls. Slower than native connectors, but extends the federation surface.

The split — open-source vs managed vs SaaS.

Tier	Example	Who runs the cluster	Who picks the version
Open-source self-host	Trino on EKS	you	you
Open-source self-host	PrestoDB on EMR	you	you
Managed serverless	AWS Athena	AWS	AWS (you pick v2 or v3)
Commercial managed	Starburst Galaxy	Starburst	Starburst
Commercial self-host	Starburst Enterprise	you	you (Starburst rebuilds)

Worked example — sizing a Trino cluster on EKS

Detailed explanation. A common interview probe is "how would you size a Trino cluster for an analytics team of 50 engineers running 200 queries / hour?" The senior answer is concurrency × per-query memory, plus headroom — not "throw more CPUs at it." Knowing the back-of-envelope numbers tells the interviewer you have actually run the engine.

Question. Size a Trino cluster on EKS for 50 engineers, ~200 queries/hour peak, average query touches 50 GB of Parquet (after partition pruning), p95 query duration target of 15 seconds.

Input — workload profile.

Metric	Value
Concurrent queries (p95)	8
Average data scanned / query	50 GB
Average query memory	12 GB
Coordinator	1
Required p95 latency	15 s

Code — Helm values sketch.

# values.yaml for trino-helm-chart (simplified)
coordinator:
  resources:
    requests:
      cpu: 8
      memory: 32Gi
    limits:
      cpu: 16
      memory: 64Gi

worker:
  replicas: 12
  resources:
    requests:
      cpu: 16
      memory: 64Gi
    limits:
      cpu: 32
      memory: 128Gi

server:
  config:
    query.max-memory-per-node: 50GB
    query.max-memory: 600GB
    query.max-total-memory-per-node: 60GB
    discovery-server.enabled: true

Step-by-step explanation.

Coordinator: single node, sized for plan compilation and task dispatch — 8 CPU / 32 GB is plenty for 200 queries/hour as long as the planner is not the bottleneck.
Workers: 12 nodes at 16 vCPU / 64 GB each = 192 vCPU and 768 GB across the cluster. With p95 concurrency of 8 and average memory 12 GB, that gives ~12 GB worker memory headroom per query.
query.max-memory-per-node and query.max-memory cap individual queries — set them so a runaway query cannot OOM the whole cluster.
For p95 latency of 15 s on 50 GB scans, the math is "50 GB / (12 nodes × ~1 GB/s S3 read per worker)" ≈ 4 s of pure scan time. The remaining 11 s budget covers planning, join shuffles, and aggregate.
Add 30% headroom for partition skew and unexpected load. The right answer is "round up to 15 workers" if the SLO is tight.

Output.

Sizing variable	Value
Workers	12 (× 16 vCPU / 64 GB)
Coordinator	1 (× 8 vCPU / 32 GB)
Concurrent query slots	~8
Per-query memory ceiling	50 GB
Cluster memory ceiling	600 GB

Rule of thumb. Size workers by concurrent_queries × per_query_memory + 30% headroom. Size the coordinator for plan rate, not data — even 1000 queries/hour rarely needs more than 16 vCPU on the coordinator. Always cap query.max-memory so one bad query cannot blast the cluster.

Worked example — Athena workgroup vs self-hosted Trino — same SQL, different ops

Detailed explanation. A second favourite interview probe: "if you can run the same SQL on Athena and on self-hosted Trino, why pick one over the other?" The answer is operational responsibility plus cost shape, not "performance." Same query, same data, very different bill and incident pager.

Question. A team runs a daily ETL that scans 500 GB / day across 30 minutes. Compare the operational and cost profile on Athena vs self-hosted Trino on EKS.

Input — daily workload.

Metric	Value
Daily scan	500 GB
Daily queries	200
Average wall-clock / query	30 s
Peak concurrent queries	6

Code — same SQL, two deployments.

-- Athena (engine v3 / Trino) workgroup query
-- Runs in the AWS service; you see only the bytes-scanned charge
SELECT day, plan, COUNT(*) AS events
FROM iceberg.lake.events
WHERE day = CURRENT_DATE
GROUP BY day, plan;

-- Self-hosted Trino (Trino on EKS) — same query, you run the JVMs
SELECT day, plan, COUNT(*) AS events
FROM iceberg.lake.events
WHERE day = CURRENT_DATE
GROUP BY day, plan;

Step-by-step explanation.

Athena cost: 500 GB × $5/TB = ~$2.50 per day in scan. Zero cluster cost. The bill scales with scan volume.
Self-hosted Trino cost: 6 workers × m5.4xlarge × 24 h = ~$70 / day on EC2 (with reservations / Spot, less). The cluster runs whether or not it is busy.
Cost crossover: at this load, Athena is dramatically cheaper. The crossover happens around steady-state utilisation > 30% of a self-hosted cluster — then the fixed cost amortises.
Operational profile: Athena hands you a "queries failed" CloudWatch metric and zero cluster pager. Self-hosted Trino hands you JVM tuning, pod restarts, version upgrades, and connector configuration.
The right answer is "Athena for spiky / low-volume; self-host Trino when utilisation is steady and the team can pay the ops tax for the savings."

Output.

Approach	Daily cost	Ops effort	Best fit
Athena v3	~$2.50	low	spiky, low scan
Self-host Trino	~$70 fixed	medium	steady, high scan
Starburst Galaxy	~$X (variable)	low	when you want Trino with support

Rule of thumb. Compute "Athena equivalent cost" = total daily bytes scanned × $5 / TB. Compare against your cluster fixed cost. If the cluster cost is less than half the Athena cost, self-hosting earns its keep; if the cluster cost is more than the Athena cost, you are paying for capacity you do not need.

SQL interview question on engine architecture

A senior interviewer might frame this as: "Walk me through what happens from SELECT * FROM iceberg.lake.events WHERE day = '2026-06-10' AND plan = 'gold' arriving at the coordinator, to rows returning to my client. Be specific about coordinator vs worker vs connector responsibilities."

Solution Using a stage-by-stage trace

-- The query under trace
SELECT *
FROM iceberg.lake.events
WHERE day = DATE '2026-06-10'
  AND plan = 'gold';

-- Inspect what the engine actually does:
EXPLAIN (TYPE DISTRIBUTED)
SELECT *
FROM iceberg.lake.events
WHERE day = DATE '2026-06-10'
  AND plan = 'gold';

Step-by-step trace.

Stage	Component	What happens
1. Parse	Coordinator	SQL string → AST → logical plan
2. Plan	Coordinator	Logical plan → optimised distributed plan
3. Push down	Iceberg connector	`day = ...` and `plan = ...` translated into Iceberg partition + column filters; connector returns only matching data files
4. Split	Coordinator	Generate one split per data file (or per row group)
5. Schedule	Coordinator	Assign splits to workers based on data locality / fair share
6. Scan	Workers + connector	Read Parquet row groups from S3; apply residual predicates per row
7. Exchange	Workers	(For non-trivial queries) shuffle rows between stages
8. Return	Coordinator	Stream rows back to client over JDBC / HTTP

The trace highlights where each component owns work. The Iceberg connector eliminates entire partitions and entire files via metadata before any worker reads a byte. Workers do the I/O-heavy scan; the coordinator never reads data, it just plans and dispatches.

Output:

Component	Owns
Coordinator	Parse, plan, schedule, return
Workers	Scan, exchange, aggregate
Connector	Metadata, splits, pushdown, read/write

Why this works — concept by concept:

Pushdown happens at the connector — the engine asks the connector "what can you do with these predicates?" and the Iceberg connector translates day = ... into a partition filter, skipping entire S3 prefixes.
Splits are the unit of parallelism — each data file (or row group) becomes one split, dispatched to one worker. More splits = more parallelism, up to the worker count.
Workers are stateless and short-lived — without TFE, a worker death kills the query. With TFE, intermediate data persists in the exchange and the failed split retries.
Coordinator never reads data — its job ends at "the rows have been streamed back to the client." This is why coordinator sizing is about CPU for planning, not memory for scans.
Cost — Parse + plan: O(query size). Scan: O(rows after pushdown). Exchange: O(rows × join fan-out). The "magic" of pushdown reduces the dominant term to "only the partitions you asked for."

SQL
Topic — joins
JOIN problems for federated SQL

Practice →

4. Connector ecosystem & federation patterns

Connectors are how a federated SQL engine reaches the lakehouse — and how cross-source joins actually work

The mental model in one line: a connector is a Java module that implements the engine's SPI for one external system — Hive, Iceberg, Delta, Hudi, MySQL, Postgres, Kafka, Elasticsearch, MongoDB — exposing it as a SQL catalog that you can SELECT from and (sometimes) INSERT into. Once you internalise that "every external system is just another catalog," the entire query federation interview surface collapses to "which connectors exist, which support pushdown, and how do cross-source joins behave?"

The four connector families.

Lakehouse / table-format connectors. Hive, Iceberg, Delta, Hudi. These read columnar files (Parquet, ORC) on object storage with table-format metadata for partitioning, schema evolution, and snapshot isolation. The lakehouse default.
JDBC connectors. Postgres, MySQL, SQL Server, Oracle, Redshift, Snowflake. These wrap the source database's own SQL engine over a JDBC connection. Predicate pushdown converts WHERE clauses into native SQL the source database executes.
NoSQL / specialty connectors. Kafka, Elasticsearch, MongoDB, Redis, Cassandra. Each translates SQL semantics onto a non-relational API as best it can; pushdown is partial.
Cloud-native / system connectors. system (engine introspection), jmx, tpch / tpcds (generated test data), Iceberg REST catalog, Glue catalog, Unity catalog.

The lakehouse default — Hive vs Iceberg.

Hive connector. Reads Hive-managed warehouses: external tables backed by a Hive metastore (HMS) or AWS Glue catalog, with partition discovery in the catalog. Mature, ubiquitous, but lacks atomic writes, schema evolution beyond renames, and time-travel.
Iceberg connector. Reads Iceberg tables — same Parquet files on S3, but with a manifest-based metadata layer that gives atomic writes, full schema evolution, hidden partitioning, time-travel, and table maintenance commands (OPTIMIZE, EXPIRE_SNAPSHOTS).
Delta connector. Reads Delta Lake tables — Databricks' table format with a transaction log (_delta_log) on S3. Conceptually similar to Iceberg; lock-in to Databricks pipelines is the differentiator.
Hudi connector. Reads Apache Hudi tables — built for streaming ingest patterns (copy-on-write vs merge-on-read storage modes).

Pushdown — the most important property.

Predicate pushdown. The engine translates WHERE clause filters into something the connector can apply at the source. Hive/Iceberg: partition + column statistics filtering. JDBC: native WHERE in the remote SQL. Kafka: usually no pushdown (offsets, not columns). Elasticsearch: partial (term filters).
Projection pushdown. Only the requested columns are read from columnar storage. Critical for wide tables on Iceberg / Delta / Hudi.
Aggregate pushdown. Some connectors push SUM / COUNT / MAX down to the source. The Iceberg connector pushes count and min/max from statistics; JDBC connectors push the aggregate as native SQL.
Limit pushdown. A LIMIT 100 at the engine becomes a LIMIT 100 at the source — invaluable for "find me one row" queries on huge tables.

Federated join behaviour — the gotcha.

Pushdown stops at the join. Each side of a cross-source join is read separately; the rows are shipped to the engine workers and joined there. A 1B-row Iceberg table joined to a 10B-row MySQL table is not a 10B-row MySQL scan — it is a full read of both tables across the network.
Always filter before the join. Push the most-selective predicate as deep into each side as possible. If you can pre-aggregate a side in the source (with a CTE that the connector compiles to native SQL), do it.
Beware of large JDBC pulls. A SELECT * FROM jdbc.huge_table will saturate the source database's network. Push a WHERE and a column list every time.

Common interview probes on connectors.

"Which connectors support predicate pushdown?" — the lakehouse and JDBC connectors do, fully. Kafka and Elasticsearch do partially. Mongo and Cassandra are partial.
"How do I join across connectors in one SQL statement?" — SELECT ... FROM cat1.schema1.t1 JOIN cat2.schema2.t2 ON ... — the engine plans and shuffles between the two source reads.
"What is the difference between the Hive and Iceberg connectors?" — Hive reads partitioned Parquet/ORC with metastore catalogs; Iceberg adds atomic writes, schema evolution, snapshot isolation, and time-travel via manifest metadata.
"Why does the Kafka connector seem slow?" — Kafka is not a queryable table; the connector scans topics by offset, with limited pushdown. Use it for low-volume ad-hoc joins, not large scans.

Worked example — federated join: Iceberg events × Postgres customers

Detailed explanation. The canonical federation example: a customer profile lives in a Postgres OLTP database; the event stream lives as Iceberg tables in S3. A single Trino query joins them — the engine reads each side with its own connector and joins in the worker layer. The gotcha is which side ships across the network and how to keep the join fast.

Question. Write a Trino query that joins Iceberg events to Postgres customers to produce per-plan event counts for the last 7 days. Show the EXPLAIN output and explain why the join uses dynamic filtering.

Input — events (Iceberg).

event_id	customer_id	event_ts
1	100	2026-06-10 09:01
2	101	2026-06-10 09:02
3	100	2026-06-11 12:30

Input — customers (Postgres).

customer_id	plan
100	gold
101	silver
102	gold

Code.

-- Federated join in one SQL statement
SELECT
    c.plan,
    COUNT(*) AS events_7d
FROM postgres.public.customers c
JOIN iceberg.lake.events     e
       ON e.customer_id = c.customer_id
WHERE e.event_ts >= CURRENT_TIMESTAMP - INTERVAL '7' DAY
  AND c.is_active = true
GROUP BY c.plan;

Step-by-step explanation.

The coordinator plans the query and pushes the is_active = true filter into the Postgres connector (compiled as native Postgres SQL WHERE is_active = true). Postgres returns only active customers.
The coordinator pushes the event_ts >= ... filter into the Iceberg connector (compiled as a partition + column filter on the event table). Iceberg returns only files within the last 7 days.
Dynamic filtering kicks in: once the Postgres side has materialised (small build side), the coordinator broadcasts the customer_id set to the Iceberg scan as a runtime filter. The Iceberg scan now reads only matching files / row groups.
The two sides are shuffled by customer_id and joined in worker memory.
The GROUP BY aggregates the joined rows per plan.

Output.

plan	events_7d
gold	2
silver	1

Rule of thumb. Federated joins live or die on dynamic filtering plus selective predicates on both sides. Always filter both sides as deeply as you can before the join; let dynamic filtering on the larger side ride on the smaller side's predicate set. If neither side filters meaningfully, the engine ships the world across the network and you wait.

Worked example — predicate pushdown check with EXPLAIN

Detailed explanation. A senior interviewer often asks "how would you verify pushdown actually happened?" The answer is EXPLAIN — every federated SQL engine exposes a plan command that shows whether a predicate landed at the connector or at the engine. The senior signal is reading the plan output and naming the operators.

Question. Given a JDBC connector over Postgres, write an EXPLAIN and identify the pushed-down predicate in the plan output. Why does this matter?

Input — Postgres connector configured as pg catalog.

customer_id	created_at	plan
100	2026-01-15	gold
101	2026-03-01	silver

Code.

-- Inspect the plan
EXPLAIN (TYPE DISTRIBUTED, FORMAT TEXT)
SELECT customer_id, plan
FROM pg.public.customers
WHERE created_at >= DATE '2026-06-01'
  AND plan IN ('gold', 'silver');

-- Sample (abbreviated) output:
-- - Output[customerId, plan]
--   - RemoteSource[1]
-- Fragment 1
--   - TableScan
--       table = pg:public.customers
--       columns = [customer_id, plan]
--       predicate pushdown = (created_at >= DATE '2026-06-01' AND plan IN ('gold','silver'))

Step-by-step explanation.

The EXPLAIN output shows a TableScan node with a predicate pushdown line listing the predicates the connector accepted. Any predicate listed there is being applied by Postgres, not by the engine.
If a predicate is not listed, the engine applies it after reading every row — a sign you should rewrite the predicate to be pushdown-friendly (use simple comparisons, avoid casts on the column side).
The columns = [...] line shows projection pushdown — only the requested columns are read.
For Iceberg, the equivalent plan shows splits = N and a dynamic filter ID reference once dynamic filtering attaches.
The senior interview habit: run EXPLAIN before merging any production query against a federated source.

Output.

Plan element	What it tells you
`predicate pushdown = ...`	which WHERE went to the source
`columns = [...]`	projection pushdown
`splits = N`	parallelism on lakehouse scans
`dynamic filter`	join-build-side filter on probe scan

Rule of thumb. EXPLAIN is the source-of-truth for pushdown. Run it on every cross-source query; if a WHERE clause did not push down, refactor the SQL (cast on the literal side, avoid wrapping the column in a function) until it does. The runtime difference is often 10–100x.

Worked example — cross-source join gotcha: shipping rows over the network

Detailed explanation. A frequent senior interview question: "why is my cross-source join slow even though both tables have indexes?" The honest answer is because the engine reads both sides over the network into worker memory; the source indexes only help with the initial filter, not with the join itself. Naming the cost gives the senior signal.

Question. A team complains a Trino query joining 50M Postgres rows to 5B Iceberg rows takes 40 minutes. Diagnose the bottleneck and propose two fixes.

Input — symptom.

Side	Rows	Storage
customers (Postgres)	50M	OLTP table
events (Iceberg)	5B	S3 Parquet

Code — the slow query.

-- Slow: 40-minute runtime
SELECT c.plan, COUNT(*) AS events
FROM pg.public.customers c
JOIN iceberg.lake.events  e ON e.customer_id = c.customer_id
WHERE e.event_ts >= CURRENT_DATE - INTERVAL '7' DAY
GROUP BY c.plan;

-- Diagnosis: the Postgres side has no selective filter,
-- so 50M rows ship over JDBC. The Iceberg side reads the
-- 7-day window (~50M rows) and joins. The join itself is fast;
-- the JDBC pull is the bottleneck.

-- Fix 1: filter the Postgres side more aggressively
SELECT c.plan, COUNT(*) AS events
FROM pg.public.customers c
JOIN iceberg.lake.events  e ON e.customer_id = c.customer_id
WHERE e.event_ts >= CURRENT_DATE - INTERVAL '7' DAY
  AND c.is_active = true
  AND c.created_at >= CURRENT_DATE - INTERVAL '180' DAY
GROUP BY c.plan;

-- Fix 2: pre-aggregate the Iceberg side, then look up customers
WITH event_counts AS (
    SELECT customer_id, COUNT(*) AS event_count
    FROM iceberg.lake.events
    WHERE event_ts >= CURRENT_DATE - INTERVAL '7' DAY
    GROUP BY customer_id
)
SELECT c.plan, SUM(ec.event_count) AS events
FROM pg.public.customers c
JOIN event_counts ec ON ec.customer_id = c.customer_id
GROUP BY c.plan;

Step-by-step explanation.

The slow query asks the engine to ship 50M Postgres rows (the whole customers table) across JDBC into worker memory. The JDBC pull is single-stream-per-connection and slow on wide tables.
Fix 1 filters the Postgres side down to active customers in the last 180 days. The pushed-down WHERE reduces the JDBC pull from 50M to perhaps 5M rows — 10x faster fetch.
Fix 2 pre-aggregates the Iceberg side first (5B rows → maybe 1M customer-event-count rows), then joins to the smaller pre-aggregate. The Iceberg scan is fast (parallel S3 reads); the join is now tiny.
The pattern "filter both sides, prefer pre-aggregating the lakehouse side, keep JDBC pulls small" is the federation playbook.

Output.

Strategy	Time
Original	40 min
Fix 1 (filter Postgres)	6 min
Fix 2 (pre-aggregate Iceberg)	3 min

Rule of thumb. In a cross-source join, the slow side is whichever side has the smallest selective predicate. JDBC connectors stream sequentially over a single (or pooled) socket — a 50M-row JDBC pull will dominate the runtime even on a fast cluster. Iceberg / Hive scans are parallel — Spark them harder is cheap; the JDBC pull is the bottleneck to hunt.

SQL interview question on connector ecosystem

A senior interviewer often asks: "Design a single SQL query that produces a daily customer activity report by joining Postgres customers, Iceberg events, and MySQL subscription_revenue. How would you tune it for federation?"

Solution Using three-connector federation with selective pushdown

-- Federated SELECT across three connectors
WITH
events_today AS (
    SELECT customer_id, COUNT(*) AS event_count
    FROM iceberg.lake.events
    WHERE event_ts >= CURRENT_DATE
      AND event_ts <  CURRENT_DATE + INTERVAL '1' DAY
    GROUP BY customer_id
),
revenue_today AS (
    SELECT customer_id, SUM(amount) AS revenue
    FROM mysql.billing.subscription_revenue
    WHERE charge_date = CURRENT_DATE
    GROUP BY customer_id
)
SELECT
    c.customer_id,
    c.plan,
    COALESCE(e.event_count, 0) AS events,
    COALESCE(r.revenue,     0) AS revenue
FROM postgres.crm.customers c
LEFT JOIN events_today  e ON e.customer_id = c.customer_id
LEFT JOIN revenue_today r ON r.customer_id = c.customer_id
WHERE c.is_active = true;

Step-by-step trace.

Step	Connector	Action
1	Iceberg	partition-filter on `event_ts`, GROUP BY pushed partially
2	MySQL JDBC	`WHERE charge_date = CURRENT_DATE`, SUM pushed as native MySQL
3	Postgres JDBC	`WHERE is_active = true` pushed as native Postgres
4	Engine	hash join (Postgres ⨝ events_today) and (Postgres ⨝ revenue_today) on `customer_id`
5	Engine	final SELECT with COALESCE — returns one row per active customer

The trace shows three CTEs that each pre-aggregate at the source. The final join is between three small, pre-aggregated streams — not three full table scans.

Output:

customer_id	plan	events	revenue
100	gold	12	49.99
101	silver	3	9.99
102	gold	0	49.99

Why this works — concept by concept:

Predicate pushdown reduces each side first — each CTE filters at the connector, so each side ships only the rows that pass the WHERE.
Aggregate pushdown shrinks the JDBC pull — the MySQL SUM(amount) runs natively on MySQL; only one row per customer crosses the wire.
LEFT JOIN preserves customers with zero activity — Postgres customers without matching events or revenue still appear in the report with COALESCE(0).
Final aggregate is in the engine — once the three sources are pre-aggregated, the federated join is cheap and runs in memory on the workers.
Cost — Postgres: O(active customers). MySQL: O(today's chargers). Iceberg: O(today's events) parallel S3 reads. Join: O(active customers × max fan-out per side).

SQL
Topic — aggregation
Aggregation problems (SQL)

Practice →

5. Cost, performance & when to pick which

Engine choice is a utilisation question, not a benchmark question — Athena, Trino, and Starburst each win at a different cluster utilisation band

The mental model in one line: Athena charges $5 per TB scanned with no cluster cost; self-hosted Trino charges fixed EC2 + ops cost regardless of scan volume; Starburst charges a commercial license on top of either — so the right engine is whichever one's cost curve matches your workload's utilisation profile. Once you can draw that crossover by hand, the entire athena vs presto and distributed sql engine cost interview collapses into a single arithmetic question.

The Athena pricing model in detail.

Per-query scan charge. ~$5 per terabyte of compressed data scanned (US-East rates; varies by region). Rounded up to 10 MB per query.
No cluster cost. Zero capacity charge. The bill is purely a function of data scanned.
DDL is free. CREATE TABLE, ALTER TABLE, schema discovery on Glue — no scan charges.
Optimisation levers. Compression (Parquet + Snappy / ZSTD), partition pruning (eliminate scanned partitions), columnar projection (avoid SELECT *), file size (256 MB–1 GB Parquet files for the right scan parallelism).

Self-hosted Trino / PrestoDB pricing model.

EC2 / EKS compute. Coordinator + workers running 24/7 (or scaled with autoscaling). Cost is roughly workers × instance hourly × utilization_hours.
Object storage egress. Reading S3 from EC2 in the same region is free; cross-region or cross-cloud incurs egress.
Ops cost. Engineering time for upgrades, alerting, JVM tuning, connector configuration. Often the dominant "real" cost for small teams.
Optimisation levers. Right-sizing the cluster, Spot / Graviton instances, autoscaling, result caching plugins, fault-tolerant execution to avoid restart cost on long queries.

Starburst (commercial) pricing model.

License fee. Per-vCPU or per-worker subscription on top of self-hosted Trino. Galaxy SaaS adds a per-cluster-hour or per-query fee.
Value-adds. Warp Speed caching, query result cache, role-based access control, lineage, materialised views, support contract.
Right fit. Enterprises with governance / compliance requirements that open-source Trino does not meet, and the engineering budget to justify the license.

The performance comparison in detail.

Athena cold start. The first query in a workgroup pays a small warmup tax (sub-second). Subsequent queries benefit from query plan cache and scaling capacity.
Trino warm cluster. A pre-warmed cluster runs identical queries faster than Athena because no scheduling latency exists between client and worker. Caching plugins (RaptorX in PrestoDB; Starburst Warp Speed) close the data-locality gap.
Concurrency profile. Athena scales horizontally for free; Trino's concurrency is capped by cluster size. For 50+ concurrent users, Athena often wins on absolute throughput.
Long-running ETL. Trino + TFE outperforms Athena for queries that run > 10 minutes because Athena imposes a 30-minute query timeout (raised, but still bounded).

Workload fit matrix.

Workload	Athena	Trino self-host	Starburst
Ad-hoc exploration (5 qps)	excellent	overkill on cost	overkill on cost
Scheduled BI (200 qps)	depends on scan	excellent	excellent
Interactive BI (50 concurrent)	excellent	needs sizing	excellent
Long ETL (30+ min)	tight (timeout)	excellent (TFE)	excellent
ML feature gen	excellent on scan	excellent on cost if utilised	excellent
Multi-source federation	excellent (v3)	excellent	excellent

Decision framework.

Utilisation < 20%. Athena. The per-query model dominates anything you'd self-host.
Utilisation 20–50%. Both viable. Athena for ops simplicity; Trino for steady cost.
Utilisation > 50%. Self-host Trino (or Starburst Enterprise). The fixed cost amortises and Athena's scan charges add up.
Governance / compliance hard requirement. Starburst (Enterprise or Galaxy).
AWS-only stack, AWS-anchored team. Athena (v3) unless ETL exceeds 30 minutes.

Cost optimisation tactics — universal.

Partition pruning. Filter on partition columns (date, region) so the connector skips entire directories. The single biggest scan-reduction lever.
Columnar formats. Parquet + ZSTD or Snappy. Wide tables with SELECT col1, col2 should scan only those columns.
File sizing. 256 MB to 1 GB Parquet files. Too small wastes file-list time; too big serialises scans.
Result caching. Athena query result cache (per query string); Starburst result cache; Trino + Hazelcast plugins.
EXPLAIN before you SELECT. Every cost optimisation starts with reading the plan.

Worked example — Athena cost reduction by partition pruning

Detailed explanation. A team queries an Iceberg events table on Athena and the bill is $40/day. The query selects event_ts filters across the whole table. The fix is partition pruning — Iceberg supports partition transforms (days(event_ts)) that let the query skip entire days of data.

Question. Given a 30 TB Iceberg events table partitioned by days(event_ts), a query filters the last 7 days. What is the scan reduction and the cost reduction?

Input — table profile.

Field	Value
Total table size (Parquet, ZSTD)	30 TB
Partition strategy	`days(event_ts)`
Days retained	365
Query filter	last 7 days

Code.

-- Without partition pruning awareness — accidentally scans all 30 TB
SELECT plan, COUNT(*) AS events
FROM iceberg.lake.events
WHERE CAST(event_ts AS DATE) >= CURRENT_DATE - INTERVAL '7' DAY
GROUP BY plan;
-- The CAST on the left side defeats partition pruning;
-- the connector cannot translate the predicate into a partition filter.

-- With partition pruning — scans only 7 days of data
SELECT plan, COUNT(*) AS events
FROM iceberg.lake.events
WHERE event_ts >= CURRENT_TIMESTAMP - INTERVAL '7' DAY
GROUP BY plan;
-- The predicate is on the raw column; Iceberg's hidden partitioning
-- transforms `event_ts` → day partitions and skips the other 358 days.

Step-by-step explanation.

The buggy query wraps event_ts in CAST(... AS DATE). The Iceberg connector cannot translate that into a partition filter because the transform on the column side is opaque.
The Iceberg connector falls back to a full scan, reading all 30 TB to apply the residual predicate at row level.
The fixed query keeps event_ts on the left of the predicate without a function wrap. The connector matches the partition transform days(event_ts) and prunes 358 days.
Scan reduction: 30 TB × (7 / 365) ≈ 0.575 TB. On Athena: 30 TB × $5 = $150 per query vs 0.575 TB × $5 = $2.88 per query — 52x cost reduction.
The 30 ms code change buys $147 per query. On 200 queries / day, that is $29,400 per day saved.

Output.

Query form	Scan	Athena cost
`CAST(event_ts AS DATE) >= ...`	30 TB	$150
`event_ts >= ...`	0.575 TB	$2.88

Rule of thumb. Never wrap a partition column in a function on the left side of a predicate. Always cast the literal instead. The partition pruning savings are typically 10–100x, and they are deterministic — no engine tuning required.

Worked example — choosing Athena vs Trino by utilisation

Detailed explanation. A common interview probe: "given a workload, how do you choose between Athena and self-hosted Trino?" The senior answer computes both bills and reports the crossover utilisation — not "Trino is faster" or "Athena is cheaper." Knowing the math makes you defensible at the architecture review.

Question. A team scans 200 TB / month across 1500 queries. Compare Athena vs a 10-worker Trino cluster on EKS (m6i.4xlarge, $0.77/hr each). At what monthly scan volume does Trino become cheaper?

Input — cost components.

Cost component	Athena	Trino self-host
Per-TB scan	$5	$0
Worker hourly	$0	$0.77 × 10
Coordinator hourly	$0	$0.50
Hours / month	n/a	24 × 30 = 720

Code — the crossover calculation.

# Athena monthly cost
athena = 200 * 5  # = $1000

# Trino monthly cost (fixed)
trino_worker_cost = 10 * 0.77 * 720  # = $5544
trino_coord_cost  = 0.50 * 720       # = $360
trino_ops_cost    = 1500             # ~$1500/mo for upgrades, alerting, etc.
trino_total       = trino_worker_cost + trino_coord_cost + trino_ops_cost
# = $7404 / month

# Crossover: solve  scan_tb * 5 = 7404
crossover_tb = 7404 / 5  # ≈ 1481 TB / month

Step-by-step explanation.

At 200 TB / month, Athena costs $1000 and Trino costs ~$7,400 — Athena wins decisively.
The crossover happens around 1500 TB / month — at that scan volume, Athena's per-query bill matches the fixed cluster bill.
Real-world considerations push the crossover lower in practice: Spot / Graviton instances cut the Trino bill by 30–50%; reserved capacity discounts another 20–40%; result caching pushes effective utilisation up.
The same math with Spot pricing and reservations puts the realistic crossover around 600–800 TB / month for an utilisation-conscious team.
The senior signal is to quote the crossover number, then say "at our actual 200 TB / month, Athena is dramatically cheaper — let's revisit when we hit 500 TB / month."

Output.

Monthly scan	Athena	Trino	Winner
200 TB	$1000	$7400	Athena
500 TB	$2500	$7400	Athena
1500 TB	$7500	$7400	Tie
3000 TB	$15000	$7400	Trino

Rule of thumb. Compute Athena cost as monthly_scan_tb × $5. Compute Trino cost as cluster_fixed_monthly + ops_overhead. The crossover is cluster_fixed_monthly / 5 TB. Below that, Athena wins; above that, Trino wins — if you can keep utilisation steady.

Worked example — Trino fault-tolerant execution for a 90-minute ETL

Detailed explanation. A common interview probe: "you have a 90-minute ETL on Trino that fails 1 in 4 runs due to spot interruption. What changes?" The answer is fault-tolerant execution (TFE) with a shared exchange — Trino retries failed tasks against new workers and recovers from spot interruption without losing the whole query.

Question. Configure Trino TFE for a daily ETL that builds an Iceberg snapshot from 3 TB of upstream events, with spot worker churn.

Input — workload.

Property	Value
Input	3 TB Iceberg events
Output	Iceberg snapshot
Wall-clock target	< 90 min
Worker type	Spot
Expected spot churn	~25% of workers / hour

Code.

-- Enable fault-tolerant execution for this session
SET SESSION retry_policy = 'TASK';
SET SESSION exchange_compression = true;

-- Run the ETL: failed tasks retry on other workers
INSERT INTO iceberg.lake.daily_aggregates
SELECT
    DATE(event_ts) AS day,
    plan,
    COUNT(*)       AS events,
    SUM(amount)    AS revenue
FROM iceberg.lake.events
WHERE event_ts >= CURRENT_DATE - INTERVAL '1' DAY
  AND event_ts <  CURRENT_DATE
GROUP BY DATE(event_ts), plan;

Step-by-step explanation.

SET SESSION retry_policy = 'TASK' activates TFE for the query — intermediate task output is spilled to a configured exchange (S3 bucket or HDFS path) instead of held in worker memory.
When a Spot worker is interrupted mid-query, the coordinator detects the failure and re-dispatches the failed task to another worker, which reads its inputs from the exchange and re-runs.
The query continues. The wall-clock impact is bounded — each failed task costs only its own re-run, not the whole query.
The exchange S3 / HDFS path adds writeback latency (typically 5–15% overhead) but eliminates the "spot interruption kills the 89-minute job" failure mode.
For interactive queries, TFE is overkill — leave it off. For ETL > 10 minutes, especially on Spot, turn it on.

Output.

Setting	Without TFE	With TFE
Spot interruption survival	no	yes
Wall-clock overhead	0	5–15%
Cost of interruption	full re-run	failed task only

Rule of thumb. Turn TFE on for any query expected to run more than 10 minutes, especially on Spot workers. The 5–15% overhead is a cheap insurance premium against losing 80+ minutes of work to a single Spot interruption.

SQL interview question on engine selection and cost tuning

A senior interviewer often asks: "Walk me through a cost optimisation for an Athena workload that suddenly doubled in spend last month. What do you look at first?"

Solution Using a layered cost audit

-- 1) Find the heaviest queries by bytes scanned
SELECT
    query_string,
    bytes_scanned,
    ROUND(bytes_scanned / 1024.0 / 1024 / 1024 / 1024 * 5, 2) AS est_cost_usd,
    execution_time_ms
FROM information_schema.queries
WHERE date(start_time) >= DATE '2026-06-01'
ORDER BY bytes_scanned DESC
LIMIT 20;

-- 2) Find queries missing partition pruning (full-table scans)
SELECT
    query_id,
    query_string,
    table_name,
    bytes_scanned
FROM query_audit
WHERE bytes_scanned > 1024 * 1024 * 1024 * 100  -- > 100 GB
  AND query_string NOT LIKE '%CURRENT_DATE%'
  AND query_string NOT LIKE '%WHERE % = DATE %';

-- 3) Find tables that should be re-partitioned or compacted
SELECT
    table_name,
    AVG(bytes_scanned)         AS avg_scan,
    COUNT(*)                   AS query_count,
    SUM(bytes_scanned) / 1024.0 / 1024 / 1024 / 1024 * 5 AS month_cost_usd
FROM query_audit
WHERE date(start_time) >= DATE '2026-06-01'
GROUP BY table_name
ORDER BY month_cost_usd DESC
LIMIT 10;

Step-by-step trace.

Step	What you check	What you find
1	top queries by scan	`events` table dominates 60% of bill
2	queries missing pruning	dashboard query has `CAST(event_ts AS DATE)` — pruning broken
3	tables by aggregate scan	`events` is the big spender — compact / re-partition
4	fix and re-run	scan drops 30x; bill drops accordingly

The trace highlights the audit pattern: top queries → broken pruning → table-level review. Most cost regressions in Athena come from a small number of unbounded queries or partition-pruning misses.

Output:

Action	Effect
Fix `CAST(...)` predicate	30x scan reduction on dashboard
Compact small files	1.5x speedup, modest cost reduction
Add result cache	repeat-query cost → $0
Move ETL to Trino self-host	when ETL is daily and 500 GB+

Why this works — concept by concept:

Bytes scanned is the bill — every Athena cost optimisation maps to "reduce bytes scanned per query." Reading the audit table by scan volume surfaces the top offenders directly.
Partition pruning is the biggest lever — a single missed pruning rewrite is often 10–50x scan inflation. Hunt these first.
Result cache turns repeat queries free — Athena caches by query string; identical dashboards re-running every minute pay one query, not 60.
Self-host crossover — once a workload is steady and high-volume, the audit will surface a candidate to move off Athena to self-hosted Trino — but never before the utilisation justifies it.
Cost — the audit itself is cheap (system tables, no scan charges). The fixes are deployed as code review nudges + table maintenance jobs.

SQL
Topic — optimization
Query optimization problems (SQL)

Practice →

Cheat sheet — Trino vs Presto vs Athena recipes

Federated join across two sources. SELECT ... FROM cat1.s1.t1 JOIN cat2.s2.t2 ON ... — the engine fans out reads through both connectors, joins in the worker layer, and returns rows. Always filter both sides selectively before the join to avoid shipping the world.
Predicate pushdown check. Run EXPLAIN (TYPE DISTRIBUTED) SELECT .... Look for predicate pushdown = (...) on each TableScan node. Any predicate not listed there is being applied by the engine, not the source.
Athena partition projection. TBLPROPERTIES ('projection.enabled' = 'true', 'projection.day.type' = 'date', ...) — skips Glue catalog lookups for partitions, dramatically faster on time-partitioned tables.
Iceberg time-travel. SELECT * FROM iceberg.lake.events FOR VERSION AS OF 12345 or FOR TIMESTAMP AS OF TIMESTAMP '2026-06-01 00:00' — read a historical snapshot without restoring it.
Trino fault-tolerant execution. SET SESSION retry_policy = 'TASK' — long ETL queries spill intermediate data to a shared exchange and survive worker churn. Trade 5–15% wall-clock for spot resilience.
PrestoDB RaptorX caching. Hierarchical cache on data files + metadata. Configure cache.enabled=true on the connector to warm repeat scans on the same partitions.
Cross-source join performance. Pre-aggregate the lakehouse side in a CTE, then JOIN to a small filtered JDBC side. JDBC pulls are sequential — keep them small.
Partition pruning safety. Never wrap a partition column in a function on the left side of a predicate. Use event_ts >= TIMESTAMP '...', not CAST(event_ts AS DATE) >= DATE '...'.
EXPLAIN ANALYZE for runtime stats. EXPLAIN ANALYZE SELECT ... runs the query and annotates each operator with rows / bytes / wall time. The senior debugging primitive.
Catalog organisation. Name catalogs by source semantics: iceberg_lake, postgres_crm, mysql_billing. Don't name them by storage tier — the next migration will leak the wrong name into every dashboard.
Engine version pinning in dbt. Set target.engine_version = 'trino-435' (or equivalent) so model SQL is regression-tested against the actual deployed engine.
Athena vs Trino choice. Athena for < 20% utilisation; Trino for > 50%; both viable in the middle. Compute the crossover before you migrate.
Starburst Enterprise fit. Add governance, lineage, materialised views, and Warp Speed caching when those are a hard requirement; do not adopt Starburst purely for "performance."

Frequently asked questions

What's the difference between Trino and PrestoDB?

Trino and PrestoDB are two forks of the same project — Facebook Presto (2012). In 2019, the original maintainers left Facebook and forked the project as PrestoSQL, later renamed Trino in 2020 after a trademark dispute. Facebook (now Meta) kept the original code base under the PrestoDB name and donated it to the Linux Foundation. Both are Apache 2.0 licensed. The practical differences in 2026 are release cadence (Trino monthly, PrestoDB quarterly), connector breadth (Trino is wider), execution features (Trino has dynamic filtering and Project Tardigrade fault-tolerant execution), and ecosystem mindshare (most new tools and SaaS distributions target Trino first).

Is Athena Trino or Presto?

Both, depending on the engine version. Athena launched in 2016 on PrestoDB and remained on PrestoDB through engine version 2. In 2023 AWS introduced engine version 3, which is built on Trino. Workgroups can be pinned to v2 or v3; new workgroups default to v3. The SQL surface is mostly compatible across versions, but certain functions (LISTAGG, MERGE on Iceberg) and connector behaviours differ — always check the engine version in your workgroup settings before deploying a new query.

Can Trino query multiple data sources in one SQL statement?

Yes — that is the entire point of a federated SQL engine. A single Trino SELECT can read from Iceberg tables on S3, a Postgres OLTP database, a MySQL billing schema, a Kafka topic, and an Elasticsearch index in one statement: SELECT ... FROM iceberg.lake.events e JOIN postgres.crm.customers c ON ... JOIN kafka.events.clicks k ON .... The engine plans each source-side read through that source's connector (with predicate pushdown), reads them in parallel into worker memory, and joins them on the workers. The classic gotcha is that cross-source joins ship rows over the network — always filter selectively before the join so each side is small.

Is Starburst the same as Trino?

No — Starburst Enterprise is a commercial distribution built on top of Trino, not the same product. It bundles Trino with added enterprise features: governance, row- and column-level security, materialised views, Warp Speed caching, query result cache, vendor support, and a managed control plane. Starburst Galaxy is the SaaS multi-tenant version; Starburst Enterprise is self-hosted. The Trino engine is identical or near-identical to open-source Trino at any given release; the value of Starburst is the enterprise wrapper, not a different engine.

When should I pick Athena over Trino?

Pick Athena when your workload is spiky, ad-hoc, AWS-anchored, or below ~30% cluster utilisation if you were to self-host. The per-query pricing ($5/TB scanned) dominates anything you'd run on a 24/7 cluster at low utilisation. Pick self-hosted Trino when utilisation is steady and high (above ~50% of a sized cluster) — the fixed compute cost amortises and the per-query Athena charges become more expensive. The middle band (20–50% utilisation) is operational preference: Athena for simplicity, Trino for control over JVM, connectors, and feature flags. Above 30-minute query runtimes, Trino (with fault-tolerant execution) wins because Athena imposes per-query timeouts.

How does Trino handle joins across federated connectors?

Trino reads each side of the join through its own connector — with predicate, projection, and (where supported) aggregate pushdown — into worker memory. The two sides are then shuffled by the join key and combined with a hash join on the workers. Dynamic filtering further optimises the larger side: once the smaller (build) side is materialised, Trino broadcasts the set of join keys to the larger (probe) side's scan as a runtime filter, so the probe scan skips files / row groups that cannot match. The cost model is: O(rows read from each source after pushdown) for I/O, plus O(rows after dynamic filtering) for the join. The interview gotcha is that cross-source joins ship rows — JDBC pulls are sequential and dominate runtime, so pre-aggregate the JDBC side or filter it tightly before the join.

Practice on PipeCode

Drill the SQL practice library → for the SELECT / JOIN / GROUP BY / WHERE surface that every federated engine assumes.
Rehearse on joins problems → for the cross-source join shapes you'll write against Trino, PrestoDB, and Athena.
Sharpen aggregation drills → for the GROUP BY / SUM / COUNT patterns the federation playbook leans on.
Layer the window functions library → for the OVER (PARTITION BY ...) patterns that ship identically across Trino and PrestoDB.
Stack the CTE practice library → for the WITH-clause pre-aggregation pattern that keeps federated joins fast.
For the broader surface, read top data engineering interview questions →.
Stack the prerequisites with the only 5 skills you need to become a data engineer →.
Sharpen the dialect axis with the SQL for data engineering interviews course →.
For long-form schema craft, work through data modelling for DE interviews →.

Pipecode.ai is Leetcode for Data Engineering — every federated SQL recipe above ships with hands-on practice rooms where you write the cross-source join, the predicate-pushdown rewrite, and the partition-pruning fix against real graded inputs. PipeCode pairs every reading with 450+ DE-focused problems and a real-time scoring engine, so you never have to wonder whether your Trino query, your PrestoDB rewrite, or your Athena cost optimisation actually holds up under interview pressure.

Practice SQL now →
JOIN drills →

Feature Stores Compared: Feast vs Tecton vs Hopsworks for Production ML

Gowtham Potureddi — Wed, 17 Jun 2026 12:56:37 +0000

feature store is the piece of the ML platform almost every team underestimates until the second production model ships, two pipelines compute "user 7-day spend" with subtly different definitions, and the on-call ticket reads "training accuracy 92%, production accuracy 71%." That gap has a name — training-serving skew — and a feature store is the boring, opinionated piece of infrastructure that closes it by making one canonical feature definition the single source of truth for both the offline training dataset and the online serving lookup.

This guide is the side-by-side reference you actually want when your team is evaluating feature stores compared to one another. It walks through why feature stores exist, the role they play in a modern ML platform, the offline feature store vs online feature store split, the point-in-time join semantics that keep historical features honest, the feast vs tecton vs hopsworks vendor matrix, and the full training-to-serving lifecycle with feature serving SLAs, materialization, and drift monitoring for production ml features. Each section pairs a teaching block with a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the streaming practice library → where most feature pipelines live, rehearse on ETL problems → to internalise the offline → online materialization shape, and stack the platform-design muscles with system-design drills →.

On this page

Why feature stores exist — training/serving skew and feature reuse
The feature store's role in a modern ML platform
Online vs offline store — two stores, one truth
Feast vs Tecton vs Hopsworks — vendor comparison
Training-to-serving lifecycle in production
Cheat sheet — feature store recipes
Frequently asked questions
Practice on PipeCode

1. Why feature stores exist — training/serving skew and feature reuse

Training-serving skew is the silent killer of production models — a feature store fixes it by making one feature definition the contract between data pipelines and ML services

The one-sentence invariant: a feature store is the system that owns a feature's definition, its historical values for training, and its latest value for serving — so that the model sees the exact same thing in production that it saw during training. Once you internalise "one definition, two stores," every downstream architectural decision (materialization, point-in-time joins, online TTLs, drift monitoring) becomes a corollary.

The two real problems a feature store solves.

Training-serving skew. Data scientists prototype features in Pandas / Snowflake notebooks, then a separate engineer reimplements the same feature in a Flink job for serving. Two implementations, two bugs, one silent accuracy loss. A feature store makes the one definition compile down to both the offline and the online path so the gap cannot exist.
Duplicated feature logic across teams. Three teams independently compute "user 7-day order count." Three names (user_7d_orders, u_orders_7d, recent_orders_count_7d), three slightly different windows (rolling 7 days vs trailing-week vs ISO-week), three slightly different SLAs. A feature store centralises the definition, the owner, and the lineage so the second team discovers and reuses instead of rebuilding.

The "research notebook → production service" gap.

A research notebook reads from the warehouse: cheap latency, point-in-time joinable, all of history. A production service reads from a low-latency key-value store: millisecond budget, single-row by entity, fresh-only. Without a feature store, somebody hand-translates between the two and the translation is where the skew lives. With a feature store, both reads compile from the same logical feature view — and the registry guarantees they are the same.

When you DON'T need a feature store.

One model, one team, all-batch scoring. If you score offline on a schedule, the same warehouse query that built the training data builds the scoring data. Adding a feature store buys you nothing this quarter.
Sub-10 features, sub-1k QPS, single dialect. A handful of features and a single Redis instance with hand-rolled hydration code is faster to ship than a feature store deployment.
Pure NLP / vision models with raw inputs. If the model consumes raw text or pixel buffers, "features" are really embeddings produced inside the model. A vector store is the right tool, not a feature store.

The 2026 reality.

Feature stores are now a data engineering concern, not a model concern. The DE owns the offline → online materialization, the freshness SLA, and the registry. The data scientist consumes via get_historical_features() and get_online_features() — they never write to either store directly.
Streaming features are table-stakes. Tecton, Hopsworks, and recent Feast releases all support a streaming materialization path that consumes Kafka / Kinesis and pushes per-entity updates to the online store on sub-second timescales.
Point-in-time correctness is non-negotiable. Every modern feature store ships an AS-OF join semantic so that the training row labelled "fraud at T = 2026-04-01 09:00:13" sees feature values frozen at T, not the latest values.
Open-source baselines are mature. Feast 0.40+ and Hopsworks 4.x both ship production-grade self-hosted deployments. Tecton remains the velocity-and-managed-streaming option but is no longer the only credible answer.

Worked example — the training-serving skew bug in one diagram

Detailed explanation. New ML teams write a feature once in Python (Pandas window over the warehouse) for training, then again in production-serving code (Redis hash lookup or a Flink rolling counter) for serving. The two implementations slowly drift — a holiday-rule fix here, a timezone fix there — and the model's production accuracy drops without anyone noticing because the offline test set still passes.

Question. A team trains a churn model on user_7d_orders. Training shows AUC 0.91. Production AUC is 0.74. How would a feature store have prevented the gap? Walk through the offending architecture and the fixed architecture.

Input.

Path	Implementation	Where it runs	Bug surface
Training	Snowflake SQL window	offline notebook	timezone = UTC
Serving	Flink rolling counter	streaming service	timezone = local
Net effect	two definitions of "7 days"	drift between offline and online	training-serving skew

Code.

# WITHOUT a feature store — two divergent implementations

# Offline (training)
training_features_sql = """
SELECT
    user_id,
    COUNT(*) AS user_7d_orders
FROM orders
WHERE order_ts >= DATEADD(day, -7, CURRENT_TIMESTAMP())   -- UTC
GROUP BY user_id
"""

# Online (serving) — different system, different timezone semantics
def user_7d_orders_online(user_id: int) -> int:
    # Flink rolling state, keyed by user, 7-day window in *local* time
    return flink_state.get(("u7o", user_id))

# WITH a feature store — one definition compiled to both paths
@on_demand_feature_view(
    sources=[orders_stream, orders_warehouse],
    entities=[user],
    schema=[Field("user_7d_orders", Int64)],
    ttl=timedelta(hours=1),
)
def user_7d_orders(orders: DataFrame) -> DataFrame:
    cutoff = orders["event_ts"] - pd.Timedelta(days=7)   # UTC, single source of truth
    return (orders
            .filter(orders["event_ts"] > cutoff)
            .groupby("user_id")
            .agg(user_7d_orders=("order_id", "count")))

Step-by-step explanation.

In the bug architecture, the SQL DATEADD(day, -7, CURRENT_TIMESTAMP()) evaluates in UTC because Snowflake's CURRENT_TIMESTAMP is UTC-zoned. The Flink state, however, was keyed in the JVM's default timezone (often the deploy region's local time). On UTC-vs-PT clusters, the 7-day window slides by 8 hours.
Eight hours of window drift is enough to include or exclude an entire weekend of orders for west-coast users. Training-serving skew silently amplifies on edge users — the very users a churn model cares about most.
The fixed architecture defines user_7d_orders once as a feature view. The materialization layer compiles the same logic to both the offline SQL (Snowflake / BigQuery / Spark) and the online incremental update (Flink / structured streaming). Timezone is pinned UTC at the definition layer; both paths inherit it.
The model now reads features through get_historical_features() at training time and get_online_features() at inference time. Both APIs hit the same logical feature view; the offline path scans the offline store and the online path looks up the online store, but the definition is identical by construction.

Output.

Path	Pre-feature-store AUC	Post-feature-store AUC
Offline test set	0.91	0.91
Production (live)	0.74	0.90

Rule of thumb. If two different services compute the same feature, the only question is how long until they diverge — not whether. A feature store collapses the two implementations into one definition and recovers most of the production accuracy gap in a single quarter.

Worked example — feature reuse: three teams, three implementations, one bug

Detailed explanation. Without a feature store registry, every team rebuilds the wheel. Three teams across fraud, recommendations, and growth all need "user lifetime order count." They each write it, they each ship it, and they each carry the bug when the orders table gets a new soft-delete column that none of the three rolls into their query.

Question. A platform team audits the warehouse and finds three user_lifetime_orders columns in three different schemas, all subtly different. How does adding a registry-first feature store help, and what does the migration plan look like in one quarter?

Input.

Team	Column	Definition
Fraud	`fraud.user_orders_lifetime`	COUNT(orders) — ignores soft-deleted
Recs	`recs.lifetime_orders`	COUNT(orders) — includes soft-deleted
Growth	`growth.user_total_orders`	COUNT(orders) WHERE status != 'cancelled'

Code.

# The one canonical definition — registered once, consumed three ways
name: user_lifetime_orders
entity: user
description: "Count of completed (non-cancelled, not-soft-deleted) orders ever."
owner: platform-de@pipecode.ai
source:
  warehouse: orders
  event_ts: order_placed_at
transform: |
  SELECT
      user_id,
      COUNT(*) AS user_lifetime_orders
  FROM orders
  WHERE status != 'cancelled'
    AND deleted_at IS NULL
  GROUP BY user_id
ttl: 24h
tags: [user, lifetime, finance-grade]

Step-by-step explanation.

The registry record fixes the definition (the exclusion rules: non-cancelled, not-soft-deleted), the owner (platform DE team — accountable for changes), and the TTL (how stale the online value is allowed to be). Each team now references the registry instead of writing their own SQL.
The migration plan is two-step: (a) deploy the canonical feature view as user_lifetime_orders_v1, populate it from a backfill, and have fraud / recs / growth read from it in shadow mode for one week; (b) cut over each consumer and decommission the per-team columns.
The "shadow week" surfaces the silent disagreements. Fraud was over-counting because it included cancelled orders during the COVID rollback; recs was under-counting on the new soft-delete column. Both bugs were sitting in production unnoticed because nobody compared the three columns against each other.
Net result: one definition, one owner, one number — the company-wide "lifetime order count" goes from three contradictory numbers in three reports to one number that every team trusts.

Output.

State	Definitions	Owner	Discrepancy across reports
Pre-feature-store	3	unclear	4–11% drift
Post-feature-store	1	platform DE	0%

Rule of thumb. The first deliverable of a new feature store is not a new feature — it is the deprecation of three existing duplicate features. Reuse is what justifies the platform cost; novelty comes second.

Worked example — when NOT to deploy a feature store

Detailed explanation. It is just as important to know when a feature store is overkill. A small team running a single batch model — a churn scorecard scored once a week — gains almost nothing from a feature store and pays the operational cost of running registry + offline + online services for a use case that never needs the online path.

Question. A startup with one DE, one DS, and one fraud model ("score every transaction within 30 seconds of arrival") asks whether they need a feature store. What is the smallest viable architecture?

Input.

Constraint	Value
Models in prod	1
Features	14
Scoring latency budget	30 seconds (not milliseconds)
Team size	3

Code.

# Smallest viable — single Python service, no feature store
def score_transaction(txn: dict) -> float:
    # 1. Build all 14 features inline from a single Snowflake query.
    features = snowflake.fetchone("""
        SELECT
            user_7d_orders,
            user_lifetime_orders,
            ...
        FROM mart.user_features
        WHERE user_id = %s
    """, (txn["user_id"],))
    # 2. Call the model.
    return model.predict_proba([features])[0][1]

Step-by-step explanation.

The 30-second scoring budget is two orders of magnitude looser than the typical 25 ms online-store SLA. A single Snowflake query per scoring event is fast enough; you do not need Redis / DynamoDB on the critical path.
With only 14 features and 1 model, there is no "reuse" deliverable to justify the registry. A YAML / Markdown table in the repo serves the same governance need.
The same Snowflake query builds both the training set (over historical rows) and the live score (over the latest row). Training-serving skew is structurally avoided because there is one query, one engine, one definition.
The reassessment trigger is concrete: when the team adds a second model, OR a sub-second scoring SLA, OR more than 50 features — at any of those points, the feature store starts paying for itself.

Output.

Trigger	Adopt feature store?
1 model, batch scoring, <50 features	No
2+ models sharing features	Yes
Sub-second serving SLA	Yes
50+ features across teams	Yes

Rule of thumb. Adopt a feature store the moment your second model wants to share features with the first, OR the moment serving latency drops below 1 second. Below those thresholds, a single warehouse query and good naming discipline are cheaper than the platform.

Interview question on when to introduce a feature store

A senior interviewer often opens with: "Your team has shipped one batch model and is now greenlighting a real-time fraud detector. Walk me through whether to introduce a feature store, what the migration would look like, and which features go offline-only versus online + offline."

Solution Using a tier-by-tier adoption plan and a feature classification

# Step 1 — classify every feature by access pattern
features:
  - name: user_lifetime_orders
    access: [offline, online]    # used by both batch churn and real-time fraud
    freshness_sla: 1h
    materialize: streaming-from-orders
  - name: user_avg_basket_30d
    access: [offline]            # batch churn only
    freshness_sla: 24h
    materialize: nightly-batch
  - name: txn_velocity_60s
    access: [online]             # real-time fraud only, useless for batch churn
    freshness_sla: 1s
    materialize: streaming-only

# Step 2 — deploy registry + offline store first, online store second
phases:
  - phase: registry-and-offline
    weeks: 1-2
    deliverable: every feature has a canonical definition; batch churn reads from offline
  - phase: online-store
    weeks: 3-5
    deliverable: redis online store, materialization job for online-tagged features
  - phase: cutover
    weeks: 6-8
    deliverable: real-time fraud reads from online store; deprecate ad-hoc Redis hashes

Step-by-step trace.

Week	Action	Risk
1–2	Stand up registry; classify 30+ features	low — paperwork only
1–2	Backfill offline store from warehouse	low — read-only
3–5	Provision Redis / DynamoDB online store	medium — production infra
3–5	Build materialization job	medium — data freshness depends on it
6–8	Cut over real-time fraud reads	high — production scoring affected
6–8	Decommission ad-hoc Redis hashes	low — but irreversible, do last

The trace highlights that the registry-and-offline phase is structurally safer than the online cutover. The plan reflects that asymmetry by running classification and backfill in parallel up front and serialising the online infra and cutover behind it.

Output:

Phase	Models supported	Features served
Week 2	batch churn (no change)	30+ via offline only
Week 5	batch churn + shadow fraud	30+ offline, 12 online
Week 8	batch churn + live fraud	30+ offline, 12 online (cutover)

Why this works — concept by concept:

Access-pattern classification — every feature is tagged offline, online, or both. The tag decides which infra (just warehouse, or warehouse + KV store + materialization) needs to be paid for. Cheap insurance against over-provisioning.
Freshness SLA — pins how stale a feature is allowed to be at serve time. Drives the materialization cadence (nightly batch vs streaming) and the online store TTL. Surfaces up-front the cost of "I want this fresh to the second."
Registry-first phasing — registry is the safest deliverable; deploy it first to surface the duplicated features without touching production. Online infra comes only after the inventory is clean.
Shadow before cutover — run the real-time fraud model in shadow mode reading from the online store for a week before flipping the decision path. Catches lookup-latency and TTL bugs before they touch production decisions.
Cost — registry + offline are nearly free (object store + Snowflake / BigQuery). Online store is the recurring cost: ~$0.50–$5 per million reads on Redis / DynamoDB, plus the streaming infra. Quote it explicitly when the platform tax is questioned.

DE
Topic — design
System design problems (DE)

Practice →

2. The feature store's role in a modern ML platform

A feature store is the contract between data pipelines and ML services — registry + offline store + online store + serving APIs + monitoring, in five tightly-coupled pieces

The mental model in one line: a feature store is one registry (definitions, owners, versions), two stores (offline historical, online latest), two APIs (get_historical_features, get_online_features), and one monitor (drift, freshness, fill rate) — and every ML pipeline either writes to it or reads from it, never around it. Once you can name those five pieces and what each one owns, the platform diagram fits on a napkin.

The five pieces in detail.

Registry. The catalogue of every feature definition — its name, owner, source, transform, freshness SLA, TTL, and tags. Lives in a small SQL database (Postgres / SQLite) or sometimes in object storage (Feast's registry.db). Acts as the source of truth for what a feature is.
Offline store. The point-in-time-correct historical archive of every feature, keyed by (entity, event_timestamp). Backed by the warehouse (Snowflake / BigQuery / Redshift) or the lakehouse (Delta / Iceberg / Hudi). Optimised for analytical scans during training — not lookups.
Online store. The low-latency single-entity lookup store for serving. Backed by Redis / DynamoDB / Cassandra / Bigtable. Optimised for sub-25 ms reads keyed by entity. TTLs bound staleness and recycle storage.
Serving APIs. Two functions on the SDK: get_historical_features(entity_df, features=[...]) — does a point-in-time join against the offline store; get_online_features(entities, features=[...]) — does an entity-keyed lookup against the online store. Both compile from the same feature view definition.
Monitoring. Surface area for feature drift (offline distribution vs online sample), freshness (lag between source event and online value), fill rate (% of entities that have a non-null value), and read latency. Without monitoring, the feature store is a black box once production hits.

The inputs.

Warehouse tables. Snowflake / BigQuery / Redshift / Databricks SQL. The source for batch features (aggregates over multi-day windows, slowly changing dimensions, historical labels).
Streams. Kafka / Kinesis / Pub/Sub. The source for streaming features (sub-second windows, per-entity rolling counters, "last seen" timestamps).
Feature pipelines. Spark / Flink / dbt jobs that read source events and compute feature values. Their output lands in both the offline store (for training) and the online store (for serving).

The consumers.

Training jobs. Call get_historical_features(training_labels_df, features=[...]) to materialise a point-in-time training dataset. Run on Spark / Pandas / Polars; produce a model artifact.
Serving services. Call get_online_features(entities=[user_id], features=[...]) per inference request to hydrate the model input vector. P99 budget typically 25 ms.

The lineage and governance flow.

Every feature has an owner recorded in the registry. Changes to the definition require owner approval (PR review on the YAML / Python definitions in source control).
Schema evolution is non-breaking by default — add a new feature; never re-purpose an existing column. Deprecations follow a 30-day shadow window: mark tombstone: 2026-08-01, dual-write for a month, then drop.
Tags (finance-grade, pii, internal-only) drive ACLs and let downstream consumers filter the registry catalogue.

Worked example — registering a feature view (Feast)

Detailed explanation. A feature view declares the entity, the source, the transformation, the output schema, and the TTL. Once registered, the same view backs both the offline get_historical_features and the online get_online_features paths. The framework code stays small — most of the file is metadata.

Question. Register a user_7d_orders feature view in Feast that reads from a Snowflake source, keys by user, surfaces a single integer feature, and has a 1-hour online TTL. Show what a downstream caller does next.

Input.

Element	Value
Entity	`user_id` (Int64)
Source	Snowflake `mart.orders` with `event_ts` timestamp
Output	`user_7d_orders` (Int64)
TTL (online)	1 hour

Code.

from datetime import timedelta
from feast import Entity, FeatureView, Field
from feast.types import Int64
from feast.infra.offline_stores.contrib.snowflake_source import SnowflakeSource

user = Entity(name="user", join_keys=["user_id"])

orders_source = SnowflakeSource(
    name="orders",
    database="MART",
    schema="ANALYTICS",
    table="orders",
    timestamp_field="event_ts",
)

user_7d_orders = FeatureView(
    name="user_7d_orders",
    entities=[user],
    ttl=timedelta(hours=1),
    schema=[Field(name="user_7d_orders", dtype=Int64)],
    source=orders_source,
    tags={"owner": "platform-de", "freshness": "1h"},
)

Step-by-step explanation.

Entity(name="user") declares the join key column. Every feature view that targets users keys by user_id; the registry enforces the type so a join between two user-keyed features never silently joins on INT64 vs STRING.
SnowflakeSource points at the table that generates the feature. The timestamp_field is critical — it is the column Feast uses for point-in-time joins on the offline side.
FeatureView declares the schema and the TTL. The TTL is online-store-only: it tells the online store to drop entity rows whose newest timestamp is older than 1 hour. The offline store keeps history forever.
tags are arbitrary key-value pairs; consumers filter the registry by tag (e.g. "show me every finance-grade feature this user owns").
After feast apply, the same view backs both APIs. The training job reads it as a point-in-time-joined column in the training DataFrame; the serving service reads it as a Redis hash lookup keyed by user_id.

Output.

Caller	API	Latency	Result
Training job	`get_historical_features(spine_df, ["user_7d_orders"])`	minutes (Spark)	training DataFrame
Serving service	`get_online_features({"user_id": 123}, ["user_7d_orders"])`	<25 ms (Redis)	single integer

Rule of thumb. Every feature view is two metadata sections (entity + source) and one schema. Resist the urge to put business logic in the view itself — the source is where SQL / Spark lives. Views are the contract, not the computation.

Worked example — calling the two serving APIs

Detailed explanation. Once the feature view is registered, the same materialised feature surfaces through two API calls — get_historical_features for training (joins against the offline store with point-in-time correctness) and get_online_features for serving (looks up the latest value in the online store). Knowing the SDK shape is half the interview.

Question. Show the exact SDK calls a training job and a serving service make for the same feature view. Include the entity DataFrame for training and the entity dict for serving.

Input.

Caller	Entity input	Time semantics
Training	spine DataFrame with `user_id` + `event_ts`	AS-OF `event_ts`
Serving	`{"user_id": 123}`	latest value, subject to TTL

Code.

from feast import FeatureStore
import pandas as pd

store = FeatureStore(repo_path="feast_repo")

# Training — point-in-time join against the OFFLINE store
spine = pd.DataFrame(
    {"user_id": [1, 2, 3],
     "event_ts": pd.to_datetime(["2026-05-01", "2026-05-02", "2026-05-03"])}
)
training_df = store.get_historical_features(
    entity_df=spine,
    features=["user_7d_orders:user_7d_orders"],
).to_df()

# Serving — single-entity lookup against the ONLINE store
online_features = store.get_online_features(
    features=["user_7d_orders:user_7d_orders"],
    entity_rows=[{"user_id": 123}],
).to_dict()

Step-by-step explanation.

get_historical_features takes a spine DataFrame — one row per (entity, event_ts) — and returns the same DataFrame with the requested feature columns appended. For each spine row, Feast does an AS-OF join against the offline store: the value returned is the most recent feature value with feature_event_ts <= spine.event_ts.
The qualified feature name user_7d_orders:user_7d_orders is feature_view_name:feature_name. The redundancy is intentional — a single view can produce multiple features, and the SDK needs both to disambiguate.
get_online_features takes one or more entity rows (a list of dicts; one dict per entity). For each entity, the SDK hits the online store, fetches the latest value, and returns a {"user_id": [123], "user_7d_orders": [42]} shape.
The serving call is wire-compatible across Redis, DynamoDB, Cassandra, and Bigtable — the materialization layer abstracts away the backend. Swapping online stores does not change the serving service code.

Output.

Call	Returns
`get_historical_features`	DataFrame with `user_id`, `event_ts`, `user_7d_orders` (point-in-time)
`get_online_features`	dict with `user_id`, `user_7d_orders` (latest, ≤1h TTL)

Rule of thumb. Never read the offline store directly with raw SQL inside a training job — always go through get_historical_features. The SDK is what guarantees point-in-time correctness; bypassing it is how silent label leakage sneaks back in.

Worked example — monitoring drift, freshness, and fill rate

Detailed explanation. A feature store without monitoring is a black box. The three metrics every production deployment exposes are drift (offline vs online distribution mismatch — usually a KS test), freshness (lag between source event and online value), and fill rate (fraction of entities with a non-null value). Each catches a different failure mode.

Question. Define minimal SQL / pseudo-code monitors for drift, freshness, and fill rate on the user_7d_orders feature. Show the alert thresholds you would deploy.

Input.

Metric	What it catches
Drift (KS test)	offline-vs-online distribution mismatch
Freshness (lag)	upstream pipeline stalled
Fill rate	upstream join broken / new entities have no features

Code.

-- 1) Drift — KS distance between offline and online distributions
WITH offline_sample AS (
    SELECT user_7d_orders AS v FROM mart.offline_features
    WHERE event_ts BETWEEN now() - interval '7 days' AND now() - interval '1 day'
    SAMPLE (10000 rows)
),
online_sample AS (
    SELECT user_7d_orders AS v FROM mart.online_audit
    WHERE log_ts BETWEEN now() - interval '1 hour' AND now()
)
SELECT ks_distance(o.v, l.v) AS ks_offline_vs_online
FROM offline_sample o CROSS JOIN online_sample l;

-- 2) Freshness — lag between source event and online write
SELECT
    MAX(online_log_ts - source_event_ts) AS max_lag_seconds,
    PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY online_log_ts - source_event_ts) AS p95_lag_seconds
FROM mart.online_audit
WHERE log_ts > now() - interval '15 minutes';

-- 3) Fill rate — fraction of entities with a non-null feature
SELECT
    COUNT(*) FILTER (WHERE user_7d_orders IS NOT NULL) * 1.0 / COUNT(*) AS fill_rate
FROM mart.online_audit
WHERE log_ts > now() - interval '15 minutes';

Step-by-step explanation.

The drift monitor samples the offline distribution (one week of history, excluding today's potentially incomplete partition) and compares it to the last hour of online reads via a Kolmogorov-Smirnov test. A KS distance above 0.1 typically warrants investigation; above 0.2, page on-call.
The freshness monitor watches the gap between when the source event happened (source_event_ts) and when the online store recorded the new value (online_log_ts). Both p95 and max are tracked because a stalled streaming worker shows up as a slow-creeping max before the p95 budges.
The fill rate monitor catches the "new entity, no feature value" failure mode. If a launch pushes a million new users into the system and the materialization job hasn't caught up, the model serves NULL features and silently degrades. Fill rate falling below 99% on a stable population is a paging signal.
All three monitors run on a 15-minute cadence and write to the same telemetry table that powers the on-call dashboard. The alert thresholds are stored next to the feature view definition so they version with the feature.

Output.

Monitor	Threshold	Action
Drift (KS)	>0.2	page on-call
Freshness p95	>2x SLA	page on-call
Fill rate	<99% on stable population	page on-call

Rule of thumb. Every production-grade feature ships with the three monitors at the moment of registration, not bolted on after the first incident. The cost of three queries on a 15-minute cron is negligible; the cost of a silent feature regression is not.

Interview question on the platform diagram

A senior interviewer often asks: "Draw the feature store's place in your ML platform on a whiteboard. What feeds it, what reads from it, where does monitoring sit, and which pieces would you build vs buy?"

Solution Using the five-piece platform diagram and a build/buy tier

                +-------------------+
                |  REGISTRY (build) |
                |  feature views,   |
                |  owners, tags     |
                +---------+---------+
                          |
   +---------------+      |      +------------------+
   |  WAREHOUSE    |---+  |  +---|  STREAMS (Kafka) |
   |  Snowflake    |   |  |  |   |  Kinesis         |
   +---------------+   |  |  |   +------------------+
                       v  v  v
                +-----------------------+
                |  FEATURE PIPELINES    |
                |  Spark / Flink / dbt  |
                +----+--------------+---+
                     |              |
            (point-in-time)    (streaming)
                     |              |
       +-------------v---+   +------v----------+
       | OFFLINE STORE   |   |  ONLINE STORE   |
       | warehouse /     |   |  Redis / DDB /  |
       | lakehouse (buy) |   |  Cassandra(buy) |
       +-------+---------+   +------+----------+
               |                    |
   get_historical_features    get_online_features
               |                    |
       +-------v-------+    +-------v---------+
       | TRAINING JOB  |    |  SERVING SERVICE|
       +---------------+    +-----------------+
                          |
                +---------v----------+
                |  MONITORING (build)|
                |  drift, freshness, |
                |  fill rate         |
                +--------------------+

Step-by-step trace.

Piece	Build or buy	Why
Registry	build (thin)	open-source frameworks (Feast / Hopsworks) give you 80% of it
Warehouse	buy	Snowflake / BigQuery / Databricks — never build
Streams	buy	MSK / Confluent / Kinesis — never build
Feature pipelines	build	the business logic; cannot outsource
Offline store	buy	sits on the warehouse — pay per query
Online store	buy	Redis / DynamoDB / Cassandra — fully managed
Serving SDK	reuse	Feast / Tecton / Hopsworks SDKs are battle-tested
Monitoring	build (thin)	hooks into your existing telemetry stack

The trace highlights that most of the platform is buy, some of it is reuse, and a few thin pieces are build. The build pieces are exactly where your business logic lives — the rest is infrastructure that scales with your wallet, not your team size.

Output:

Diagram piece	Owner	Pager rotation
Registry + serving SDK	platform DE	weekday business hours
Online store	platform SRE	24/7
Streams + warehouse	platform DE	weekday business hours
Monitoring	platform DE	24/7 (drift + freshness pages)

Why this works — concept by concept:

Registry as the contract — every feature lives in source control as a YAML / Python file. Pull-request review on the registry is what catches "Alice and Bob are about to register two flavours of the same feature."
Two stores, two latencies — the offline store optimises for analytical scans; the online store optimises for single-row lookups. Trying to use one for both is the most common architecture anti-pattern.
Materialization is the bridge — the pipeline that moves data from offline to online runs on the same schedule as your freshness SLA. Nightly batch for 24h-freshness features; streaming for sub-second features.
Monitoring closes the loop — drift, freshness, and fill rate are the three signals that say "the feature store is alive and producing correct values." Page on the second one; the first surfaces in dashboards but rarely pages.
Cost — the recurring spend is warehouse compute (training joins), online store reads (serving QPS), and streaming compute (materialization). Each scales with usage; the registry and the monitoring add a fraction of a percent on top.

DE
Topic — ETL
ETL pipeline design problems (DE)

Practice →

3. Online vs offline store — two stores, one truth

One feature definition, two stores — the offline store answers "what was the value at this moment in the past?" and the online store answers "what is the value right now?"

The mental model in one line: the offline store is time-indexed and history-deep (point-in-time join, used for training); the online store is entity-indexed and freshness-bounded (single-row lookup, used for serving) — and the materialization job is the single bridge that guarantees both stores agree on the same feature definition. Once you can hold that asymmetry in your head, the rest of feature store engineering is plumbing.

The contrast in five bullets.

Offline store. Backed by Snowflake / BigQuery / Databricks SQL / Delta / Iceberg / Parquet-on-S3. Optimised for full-table scans, multi-day aggregations, and point-in-time joins. Holds every historical feature value forever (or for the regulatory retention window).
Online store. Backed by Redis / DynamoDB / Cassandra / Bigtable. Optimised for single-row GETs keyed by entity. Holds only the latest feature value per entity, bounded by a TTL.
Materialization. The pipeline that reads computed feature values and writes them to both stores. Batch materialization runs nightly or hourly; streaming materialization runs continuously. Same logic, two destinations.
Point-in-time correctness. Offline reads are AS-OF — given a training row labelled at T, the join returns the feature value with the largest event_ts ≤ T. This prevents label leakage from future feature values.
TTL on the online store. Bounds staleness. A 1-hour TTL says "if the online value is older than 1 hour, do not serve it" — the SDK returns NULL or raises, depending on configuration. Drives the materialization cadence.

Why point-in-time correctness matters.

A naive training join (SELECT label, features FROM ... JOIN features ON user_id) silently grabs the latest feature value for every label row. Labels in March end up joined with features computed in July — the model gets to "see the future," and training accuracy is artificially inflated.
The point-in-time join fixes this: for every label row (user_id, label_ts), the join picks the feature row with the largest event_ts ≤ label_ts. The model never sees a feature value that did not exist at label time.
This is the single most-tested concept in any feature-store interview. If you cannot explain why JOIN ON user_id is wrong and how AS-OF fixes it, you are not running production ML.

Why TTLs matter.

Without a TTL, a feature value computed yesterday could be served indefinitely. If the upstream pipeline silently stalls, the model serves stale features and slowly degrades.
A TTL on the online store says "this value is only valid for X hours; after that, treat it as missing." Combined with a freshness monitor, this surfaces a stalled pipeline within an hour instead of after a week.
TTL choice is a per-feature decision. user_lifetime_orders can have a 24h TTL; txn_velocity_60s needs a 60-second TTL. Encode the TTL in the feature view definition.

Why both stores must read the same definition.

The whole point of the architecture is one definition, two stores. If the offline computation and the online computation come from different code paths, you are back to training-serving skew.
Modern feature stores (Tecton, Hopsworks) compile a single feature view into a Spark batch job (for offline) and a Flink streaming job (for online) — same expression, two compilers. Feast asks you to write the transformation as a SQL or Python expression that runs against the source on both paths.
The materialization job is the enforcement of this property. If you ever find yourself writing two transformations (one for training, one for serving), the architecture has broken — go back and unify.

Worked example — the naive training join silently leaks the future

Detailed explanation. A team builds a churn model. The training table joins labels (one row per user-day, with a churn flag at day T) to a user_features table on user_id. The query forgets to scope the features table by time, so every label row is joined with the latest feature row — including features computed after the label day. The model "predicts" churn with 0.97 AUC; production AUC is 0.66. Classic label leakage.

Question. Given the schema below, write the buggy naive join and the correct point-in-time join. Show on a sample row why the buggy version is wrong.

Input — labels.

user_id	label_ts	churned
1	2026-03-01	0
1	2026-04-01	1
2	2026-03-01	0

Input — user_features.

user_id	event_ts	user_7d_orders
1	2026-02-25	5
1	2026-03-15	1
1	2026-04-15	0
2	2026-02-25	3

Code.

-- BROKEN — naive join joins on user_id only, grabs LATEST features
SELECT
    l.user_id,
    l.label_ts,
    l.churned,
    f.user_7d_orders
FROM labels l
JOIN user_features f ON l.user_id = f.user_id;   -- leak!

-- CORRECT — point-in-time join (Snowflake / Databricks / DuckDB syntax)
SELECT
    l.user_id,
    l.label_ts,
    l.churned,
    f.user_7d_orders
FROM labels l
ASOF JOIN user_features f
     MATCH_CONDITION (l.label_ts >= f.event_ts)
     ON l.user_id = f.user_id;

Step-by-step explanation.

The naive join multiplies each label row by every feature row for the same user. The query returns 6 rows (3 label rows, average 2 feature rows per user) instead of 3 — silently fans out, and every downstream aggregate is now wrong by a factor.
Even if the team adds DISTINCT or MAX, the result is the latest feature value at training time — i.e. a value from after the label day. The model sees user_7d_orders = 0 (computed on 2026-04-15) joined with the label from 2026-03-01. The model "learns" that low recent orders predict already-known churn — accuracy 0.97, useful 0.
The point-in-time ASOF JOIN (Snowflake / Databricks 2024+; equivalent in Postgres via lateral joins, in DuckDB natively, in Spark via as_of_join) picks the feature row with the largest event_ts ≤ label_ts per user_id. Label row (1, 2026-03-01) gets the 2026-02-25 features (user_7d_orders=5), not the 2026-03-15 or 2026-04-15 ones.
The corrected query returns exactly one feature row per label row, with values that were known at label time. The model now trains on the same view the serving service sees — production accuracy aligns with offline.

Output (correct, point-in-time).

user_id	label_ts	churned	user_7d_orders
1	2026-03-01	0	5
1	2026-04-01	1	1
2	2026-03-01	0	3

Rule of thumb. Never join labels to features on entity-key alone. Always use ASOF JOIN (Snowflake / Databricks / DuckDB), a LATERAL subquery (Postgres), or the feature store SDK's get_historical_features. If your training join does not have a time predicate, your model has leaked.

Worked example — materialization moves features offline → online

Detailed explanation. The materialization job is the bridge between the two stores. It reads the most recent feature values per entity from the offline store (or directly from the source) and writes them to the online store keyed by entity. Batch materialization runs on a schedule; streaming materialization runs continuously off Kafka.

Question. Show the two materialization shapes — batch (nightly) and streaming (continuous) — for the same user_7d_orders feature. Include the entity-keyed write.

Input.

Materialization	Source	Cadence	Online TTL
Batch	Snowflake `mart.orders`	nightly @ 02:00 UTC	24h
Streaming	Kafka `orders.events`	continuous	1h

Code.

# 1) Batch materialization — Feast nightly job
from feast import FeatureStore
from datetime import datetime, timedelta

store = FeatureStore(repo_path="feast_repo")
store.materialize(
    feature_views=["user_7d_orders"],
    start_date=datetime.utcnow() - timedelta(days=1),
    end_date=datetime.utcnow(),
)
# Internally: SELECT user_id, user_7d_orders, event_ts FROM mart.user_features
#             WHERE event_ts BETWEEN <start> AND <end>
# then for each row: redis.hset(f"user:{user_id}", "user_7d_orders", value)

# 2) Streaming materialization — Tecton-style continuous push
@stream_feature_view(
    source=orders_kafka_source,
    entities=[user],
    schema=[Field("user_7d_orders", Int64)],
    online=True,
    offline=True,
    aggregation_interval=timedelta(seconds=10),
)
def user_7d_orders_streaming(orders: Stream) -> Stream:
    return (orders
            .window(timedelta(days=7))
            .group_by("user_id")
            .agg(user_7d_orders=count()))

Step-by-step explanation.

The batch materialization is a scheduled job that scans the offline source for the time window since the last run, computes the feature values per entity, and writes them to the online store. It is cheap (one warehouse query + a bulk Redis write) but staleness is bounded only by the cadence.
The streaming materialization defines the same logic as a continuously-running aggregation over a Kafka stream. The framework (Tecton / Hopsworks / Feast-with-Bytewax) maintains the per-entity rolling state and writes updates to the online store every aggregation_interval.
Both paths write to both stores by default: the streaming job dual-writes (offline for history, online for serving); the batch job's source IS the offline store, and it writes to the online store as the materialization step.
The trade-off is freshness vs cost. Batch materialization is essentially free if the warehouse query already runs nightly; streaming materialization is a continuously-running Flink / Bytewax / Spark Streaming cluster that costs $X/month per feature view. Pick streaming only for features where the freshness SLA demands it.

Output.

Path	Freshness	Cost (per feature view)
Batch	24h	~$0 (warehouse already runs)
Streaming	10s	~$100–$500/mo (Flink cluster slice)

Rule of thumb. Default to batch materialization with a 24h cadence. Promote individual feature views to streaming only when (a) the model SLA explicitly demands sub-hour freshness, AND (b) the feature's value materially changes inside the hour. Otherwise the streaming budget is wasted.

Worked example — TTLs bound staleness and surface stalled pipelines

Detailed explanation. The TTL on the online store is a circuit breaker. If the materialization job stalls and a feature's online value goes stale, the SDK detects that now - feature_event_ts > TTL and either returns NULL or raises. The serving service treats NULL as "missing feature" — typically imputes a default or gates the model — rather than serving silently stale values.

Question. Configure a TTL on a feature view, then walk through what happens at serve time when the materialization job stalls for 4 hours on a 1-hour TTL feature.

Input.

Feature view	TTL	Materialization cadence
`user_7d_orders`	1 hour	nightly batch
Stall scenario	4 hours since last write	T0 + 4h

Code.

# Feature view with explicit TTL
user_7d_orders = FeatureView(
    name="user_7d_orders",
    entities=[user],
    ttl=timedelta(hours=1),                     # <-- circuit breaker
    schema=[Field("user_7d_orders", Int64)],
    source=orders_source,
)

# Serving service — what happens at T0 + 4h
features = store.get_online_features(
    features=["user_7d_orders:user_7d_orders"],
    entity_rows=[{"user_id": 123}],
).to_dict()

# Stale-feature handling at the application layer
v = features.get("user_7d_orders", [None])[0]
if v is None:
    # serve fallback model, or impute default, or gate the request
    log.warning("user_7d_orders stale or missing", extra={"user_id": 123})
    v = DEFAULT_USER_7D_ORDERS

Step-by-step explanation.

At write time, the materialization job writes both the feature value AND its source event timestamp to the online store. The online store entry looks like {"user_7d_orders": 5, "event_ts": "2026-06-15T08:00:00Z"}.
At read time, the SDK fetches the entry and computes staleness = now - event_ts. If staleness > TTL, the SDK treats the value as missing.
At T0 + 4h with the pipeline stalled at T0, every read for an entity that has not been refreshed sees staleness = 4h > 1h and returns NULL. The serving service falls into its NULL-handling path.
The application can choose: serve a fallback model, impute a default, or gate the request entirely. The choice is per-feature and per-model — high-value features may gate; low-value features may impute.
The TTL also drives an automatic monitoring signal: the freshness monitor fires the moment staleness exceeds the TTL on more than X% of entities. The on-call gets paged within minutes of the stall, not after a week of degraded production accuracy.

Output.

Time	Materialization status	Online value visible?	Serving behaviour
T0	fresh	yes	normal model path
T0 + 30m	fresh	yes	normal model path
T0 + 1h 5m	stalled	no (TTL expired)	NULL → fallback
T0 + 4h	stalled	no	NULL → fallback + page

Rule of thumb. Every feature view ships with an explicit TTL. The TTL should be 2–3x the materialization cadence (so transient lag does not trigger false fallbacks) but no longer than the model's tolerance for staleness. Treat TTL = materialization cadence × 2 as a starting default.

Interview question on offline-vs-online architecture

A senior interviewer often asks: "Explain the difference between the offline and online stores, what materialization is, and why you cannot serve the offline store directly even if you wanted to."

Solution Using the storage class + access pattern framing

+----------------------------+      +----------------------------+
|  OFFLINE STORE             |      |  ONLINE STORE              |
|  warehouse / lakehouse     |      |  Redis / DynamoDB / Cass   |
+----------------------------+      +----------------------------+
| append-only history        |      | latest per entity (TTL'd)  |
| analytical columnar reads  |      | row-level GET / HGETALL    |
| seconds to minutes/query   |      | <25 ms p99 / read          |
| cost: per query (compute)  |      | cost: per request (storage |
| read shape: full scan      |      |   + read units)            |
| join semantics: AS-OF      |      | join semantics: none, just |
|   (point-in-time correct)  |      |   single-entity lookup     |
+--------------+-------------+      +-------------+--------------+
               ^                                  ^
               |                                  |
               |        +-----------------+       |
               +--------|  MATERIALIZATION|-------+
                        |  batch + stream |
                        +-----------------+

Step-by-step trace.

Dimension	Offline store	Online store
Backing storage	Snowflake / BigQuery / Delta	Redis / DynamoDB / Cassandra
Read latency	seconds–minutes (scan)	<25 ms (lookup)
Read shape	columnar full scan	single-entity GET
History	forever	latest per entity, TTL-bounded
Used by	training jobs	serving services
Join semantics	AS-OF (point-in-time)	none — direct lookup
Cost driver	compute per query	reads per request + storage

The trace makes it explicit: you cannot serve from the offline store because each scoring request would cost seconds of warehouse compute and hundreds of milliseconds of latency. You cannot train from the online store because it does not retain history. The materialization job is the bridge that lets one feature definition land in both shapes.

Output:

Use case	Reads from	Why
Train churn model	offline	needs point-in-time history
Score live fraud	online	needs <25 ms lookup
Backfill 6 months	offline	needs full history
Daily batch scoring	offline	latency tolerable, no online cost

Why this works — concept by concept:

Append-only history vs latest-per-entity — the offline store keeps every (entity, event_ts) row forever; the online store keeps one row per entity, overwritten on each materialization. The schemas are different by design.
Analytical vs transactional read shape — the offline store is columnar and scans cheaply across many rows; the online store is key-value and GETs cheaply by primary key. Mixing the access patterns is what makes warehouses bad at serving and Redis bad at history.
AS-OF join semantics — only the offline store supports it. The online store has no time dimension at read time — it only knows "the latest value." Point-in-time correctness lives entirely on the offline side.
TTL as freshness circuit breaker — bounds how stale the online store can serve, surfaces stalled pipelines, and turns a silent-degradation failure into a loud "feature missing" alert.
Cost — offline scans are pay-per-query; online lookups are pay-per-request + storage. The total platform cost is dominated by online reads at high QPS and offline scans at large training-set sizes. Right-size each one with the freshness SLA per feature.

DE
Topic — streaming
Streaming pipeline problems (DE)

Practice →

4. Feast vs Tecton vs Hopsworks — vendor comparison

Feast for DIY, Tecton for streaming velocity, Hopsworks for sovereignty — the three vendors compete on managed-ness, transformation responsibility, and deployment locus

The mental model in one line: Feast is the open-source skeleton that asks you to bring your own infra; Tecton is the managed end-to-end stack that owns transformations on Spark / Snowflake / Rift; Hopsworks is the open-source-plus-managed full data-and-ML platform with the strongest on-prem story — and the three differ less in what features they store than in who owns the transformation runtime and the cloud bill. Once you can name the three trade-off axes (transformations, streaming-strength, deployment model), the right choice for any team is mechanical.

The three vendors in one matrix.

Vendor	Hosting	Transforms	Streaming	Strongest fit
Feast	open-source, self-hosted	BYO (you write SQL / Python / Spark)	community contribs (Bytewax, Spark)	DIY teams, cost-sensitive shops, AWS / GCP-native
Tecton	managed SaaS	first-party (Spark, Snowflake, Rift compute)	first-class (sub-second Flink-grade)	streaming-heavy use cases, fast time-to-prod
Hopsworks	open-source + managed	first-party (Spark, Flink, Python)	first-class (Flink-native)	sovereign / on-prem deployments, EU data residency, full data+ML platform

Feast in detail.

Open-source, BYO infra. Feast is a Python library + a small registry database. You bring the offline store (Snowflake / BigQuery / Redshift / Delta), the online store (Redis / DynamoDB / Cassandra / Postgres), and the compute that runs the transformations (Spark / dbt / your warehouse).
No managed transformations. You write the feature logic as SQL or Python that runs against your source. Feast does not run Flink for you. This is a feature (you control everything) and a cost (you have to operate everything).
Lightweight. A Feast deployment is a Python SDK, a SQLite/Postgres registry, a feature server (FastAPI), and the BYO stores. The whole control plane fits on a single VM if you want it to.
Streaming support is community-driven. Stream ingestion via Bytewax or Spark Streaming is supported, but it is not as polished as Tecton or Hopsworks. If streaming is your dominant pattern, Feast adds work.
Where it wins. Teams that already operate Snowflake + Redis well and want to add a feature-store SDK without paying a managed-platform vendor. Teams that want to read the source code.

Tecton in detail.

Managed end-to-end SaaS. Tecton runs the registry, the transformations, the online store, and the serving SDK. You write feature definitions in Python; Tecton compiles them to Spark / Snowflake / their proprietary "Rift" compute engine.
First-class transformations. Tecton owns the compute that produces the features. The same definition compiles to a batch Spark job for offline backfills and a streaming Flink-equivalent job for online updates.
Streaming velocity. Tecton's sub-second streaming materialization is the fastest off-the-shelf option. If your model needs features that change every few seconds (real-time fraud, ad bidding), Tecton minimises the engineering work.
Higher cost, faster ROI. Managed pricing means you pay per feature view + storage + compute. For a team that does not want to own the streaming runtime, it is often the cheapest path to production.
Where it wins. Streaming-heavy teams, teams that want to skip the infra build, teams shipping into AWS / GCP with no on-prem constraint.

Hopsworks in detail.

Open-source AND managed. Hopsworks ships as a free open-source project (deployable on Kubernetes or on-prem) and as a managed SaaS. Same code, two consumption models.
Full data + ML platform. Beyond the feature store, Hopsworks includes a model registry, experiment tracking, a Jupyter cluster, and a serving layer. It is closer to a Databricks-shaped platform than to a single-purpose feature store.
Strong on-prem story. Hopsworks is the most credible option for EU sovereignty / GDPR data residency / air-gapped deployments. Tecton is SaaS-only; Feast is self-hosted but lacks the full platform.
Flink-native streaming. Hopsworks integrates Flink as a first-class transformation engine. Streaming features have parity with Tecton in many shops.
Where it wins. Teams with regulatory data-residency requirements, teams that want one platform for both data and ML, teams in EU finance / public sector / healthcare.

The transformation responsibility axis.

You compute (Feast). You write the transformation in SQL / Spark; Feast registers and serves the result. You operate the compute.
Vendor computes (Tecton, Hopsworks). You declare the transformation in Python / SQL; the vendor compiles and runs it on their (or your) Spark / Flink. They operate the compute.

The pricing / operational footprint axis.

Feast. Sub-$10/mo on a small VM for the control plane; the rest is your existing warehouse + Redis bill. The "tax" is the engineering time you spend operating it.
Tecton. Five-figure-per-month-and-up SaaS pricing. The "tax" is the wallet; the engineering hours go from operating to shipping.
Hopsworks (managed). Mid-four-figure-per-month-and-up SaaS pricing, slightly cheaper than Tecton at smaller scales. Open-source self-hosted is free at the license layer, expensive at the engineering layer.

Worked example — choosing the vendor for a 30-feature, 2-model platform

Detailed explanation. A fintech team has shipped one batch churn model and is greenlighting a real-time fraud model. They have 30 features (15 batch, 15 streaming), one DE, one DS, one MLE, and a Snowflake + Redis stack already in production. Pick the vendor and justify.

Question. Given the constraints, score Feast / Tecton / Hopsworks against the team's profile and recommend.

Input.

Constraint	Value
Models	2 (one batch, one streaming)
Features	30 (50/50 batch/streaming)
Existing infra	Snowflake, Redis, AWS
Team size	3 engineers
Budget for new SaaS	low (CFO is asking)
Data residency	US-only (no EU constraint)

Code.

score(vendor) =
    0.30 * managed_streaming
  + 0.25 * cost_efficiency
  + 0.20 * fit_with_existing_infra
  + 0.15 * platform_breadth
  + 0.10 * sovereignty

Feast:
  managed_streaming = 0.4 (community-driven)
  cost_efficiency   = 0.95 (BYO, sub-$100/mo)
  fit              = 0.9 (drops onto Snowflake + Redis)
  platform_breadth = 0.4 (registry only)
  sovereignty       = 0.7 (self-host anywhere)
  TOTAL ≈ 0.69

Tecton:
  managed_streaming = 0.95
  cost_efficiency   = 0.4 (SaaS pricing)
  fit              = 0.7 (works with Snowflake; replaces Redis with theirs)
  platform_breadth = 0.7 (feature store + serving)
  sovereignty       = 0.4 (SaaS only)
  TOTAL ≈ 0.66

Hopsworks (managed):
  managed_streaming = 0.85
  cost_efficiency   = 0.55
  fit              = 0.6 (different OLAP / OLTP than Snowflake-native)
  platform_breadth = 0.9 (full platform)
  sovereignty       = 0.85
  TOTAL ≈ 0.69

Step-by-step explanation.

The team's strongest constraints are cost efficiency and fit with existing infra. Feast scores highest on both because it drops onto the Snowflake + Redis they already operate and the new spend is ≤$100/mo.
Tecton scores highest on managed streaming, which matters for the fraud model — but the cost penalty against the CFO's low-budget signal is severe. Tecton is the right answer if streaming velocity is the hard constraint; the team can tolerate 30-second freshness, so it is not.
Hopsworks ties with Feast on the total score but its platform breadth is wasted (the team is not buying experiment tracking) and its non-Snowflake-native posture costs fit points. Hopsworks would dominate if the team needed EU residency; they do not.
Recommendation: Feast. Migrate the 30 features into a Feast registry over Q3, build a Bytewax streaming materialization for the 15 streaming features, and reassess in 6 months if the streaming SLA tightens.

Output.

Vendor	Score	Recommendation
Feast	0.69	Adopt
Tecton	0.66	Reconsider if streaming SLA tightens
Hopsworks	0.69	Adopt only if EU residency arrives

Rule of thumb. Feast wins on cost-sensitive, AWS / GCP-native teams. Tecton wins on streaming-heavy, time-to-prod-pressured teams. Hopsworks wins on sovereignty-constrained or full-platform-wanting teams. Score against your top three constraints, not against the marketing site.

Worked example — declaring the same feature view across vendors

Detailed explanation. All three vendors converge on a similar declarative shape: name, entity, source, transformation, output schema, online toggle. Reading the same feature view in three syntaxes is the fastest way to internalise that the concepts are universal — only the SDK noise differs.

Question. Show the same user_7d_orders feature view in Feast, Tecton, and Hopsworks declarative syntax. Highlight the structural commonalities.

Input.

Element	Value
Entity	`user`
Source	`orders` table / stream
Transform	rolling 7-day count
Online	yes
TTL	1 hour

Code.

# Feast
from feast import FeatureView, Field, Entity
from feast.types import Int64

user_7d_orders = FeatureView(
    name="user_7d_orders",
    entities=[user],
    ttl=timedelta(hours=1),
    schema=[Field("user_7d_orders", Int64)],
    source=orders_source,        # source is SQL / Parquet; transform lives in source
)

# Tecton
from tecton import batch_feature_view, Aggregation
from tecton.types import Int64

@batch_feature_view(
    sources=[orders_batch],
    entities=[user],
    mode="spark_sql",
    aggregations=[Aggregation(column="order_id", function="count",
                              time_window=timedelta(days=7))],
    online=True,
    offline=True,
    feature_start_time=datetime(2025, 1, 1),
    batch_schedule=timedelta(hours=1),
    ttl=timedelta(hours=1),
)
def user_7d_orders(orders):
    return f"SELECT user_id, order_id, event_ts FROM {orders}"

# Hopsworks
import hsfs
fs = hsfs.connection().get_feature_store()
fg = fs.create_feature_group(
    name="user_7d_orders",
    version=1,
    primary_key=["user_id"],
    event_time="event_ts",
    online_enabled=True,
    statistics_config={"enabled": True, "histograms": True},
)
# The transformation is a Spark / Flink job that writes into fg
fg.insert(spark.sql("""
    SELECT user_id, event_ts,
           COUNT(*) OVER (PARTITION BY user_id
                          ORDER BY event_ts
                          RANGE BETWEEN INTERVAL 7 DAYS PRECEDING AND CURRENT ROW
                         ) AS user_7d_orders
    FROM orders
"""))

Step-by-step explanation.

All three definitions name the feature view, declare the entity (user), point at the source (orders), and toggle online serving. The structural shape is identical; the SDK noise differs.
Feast keeps transformations out of the framework — the source query is what defines the feature. This is consistent with Feast's "you own compute" stance.
Tecton keeps transformations inside the framework — the @batch_feature_view decorator + aggregations= argument declares the transform, and Tecton compiles it to Spark / Snowflake / Rift. This is consistent with Tecton's "managed compute" stance.
Hopsworks sits between the two — the framework owns the registry and the storage, but the actual transformation is a Spark / Flink job you write yourself and insert into the feature group. The trade-off is more code, more control.
Once registered, every downstream call (get_historical_features / get_online_features or the vendor equivalent) returns the same logical column with the same point-in-time semantics. The vendor lock-in is the SDK, not the data shape.

Output.

Vendor	Lines of declarative code	Who runs the transform
Feast	~10	you
Tecton	~15	Tecton
Hopsworks	~10 + Spark job	you (managed runtime available)

Rule of thumb. When evaluating vendors, write the same 3–5 representative feature views in each SDK. The volume difference is small; the cognitive-load difference (do I write the transform, or do they) is decisive.

Worked example — when each vendor breaks down

Detailed explanation. Every vendor has a breaking point — a use case where the trade-offs cut the wrong way. Recognising these up front avoids the worst category of platform decision: the one that looks good on day 1 and traps the team on day 365.

Question. Walk through one realistic failure mode for each vendor and the migration path out.

Input.

Vendor	Realistic break point
Feast	streaming SLA tightens to <10s
Tecton	CFO cuts SaaS spend by 50%
Hopsworks	team forks the OSS too aggressively

Code.

Feast breaks when streaming SLA tightens:
  - Bytewax / Spark Streaming materialization tops out around 30–60s per-feature freshness
    in a cost-effective configuration.
  - Migration out: keep Feast as the registry + offline; introduce a side-car streaming
    layer (Flink / Materialize) for the 2–3 sub-second features only.

Tecton breaks when SaaS spend gets cut:
  - You cannot self-host Tecton. If the budget disappears, the platform disappears.
  - Migration out: every Tecton feature view has a YAML export; rebuild them as Feast feature
    views on top of your existing Snowflake + Redis. Plan for 8–12 weeks for a 50-feature shop.

Hopsworks breaks when the team forks the OSS:
  - The OSS is fully usable but easy to over-customise. A heavy fork drifts from upstream
    and the managed-platform upgrade path closes.
  - Migration out: rebase the fork onto upstream every minor release; reserve customization
    for plugins / hooks, not core changes.

Step-by-step explanation.

Feast's breaking point is streaming SLA. If the model needs features that change every 1–10 seconds across hundreds of feature views, Feast forces you to operate a streaming runtime yourself. The migration out is partial: keep Feast as the registry, add a streaming side-car for the 2–3 features that need it.
Tecton's breaking point is budget volatility. The SaaS lock-in is real — there is no "drop the bill and keep running" path. The migration out is rebuilding on Feast over a quarter; do not let the team forget that this option exists.
Hopsworks's breaking point is over-customisation of the OSS. The platform is generous with hooks, and teams sometimes patch core code instead of using plugins. The fork then cannot upgrade. Discipline at the PR level is the fix.
All three vendors are credible at the 30-feature, 2-model scale. The breaking points only matter at the 1000-feature, 50-model scale — but the architectural decision is made on day 1, not day 1000.

Output.

Vendor	Break point	Mitigation
Feast	streaming SLA tightening	hybrid: Feast + side-car streaming
Tecton	SaaS budget cut	export YAML; rebuild on Feast
Hopsworks	OSS fork drift	rebase quarterly; plugin-only customisation

Rule of thumb. Pick the vendor whose breaking point is least likely in your roadmap. If you cannot predict the next 18 months of constraints, pick Feast — its breaking point has the cheapest mitigation.

Interview question on the vendor decision

A senior interviewer often asks: "You are joining a team that has not picked a feature store yet. What is your decision tree to choose between Feast, Tecton, and Hopsworks?"

Solution Using a four-question decision tree

Q1. Is streaming the dominant pattern (>50% of features) AND is sub-second freshness required?
  yes -> Q2
  no  -> Q3

Q2. Is the budget for SaaS at least $10k/month and there are no on-prem constraints?
  yes -> TECTON
  no  -> HOPSWORKS (managed or self-hosted), or hybrid Feast + side-car streaming

Q3. Are there EU residency / on-prem / air-gap constraints?
  yes -> HOPSWORKS (self-hosted, full platform)
  no  -> Q4

Q4. Does the team already operate Snowflake / BigQuery + Redis / DynamoDB well?
  yes -> FEAST (drops in, near-zero new operational cost)
  no  -> TECTON (managed; cheaper than building both stores from scratch)

Step-by-step trace.

Team profile	Path through tree	Pick
Real-time ad bidder, $50k budget, AWS-native	Q1 yes → Q2 yes	Tecton
Real-time ad bidder, $5k budget, OSS-friendly	Q1 yes → Q2 no	Hybrid Feast
EU bank, sovereignty required	Q1 no → Q3 yes	Hopsworks (self-hosted)
US fintech, Snowflake + Redis already in prod	Q1 no → Q3 no → Q4 yes	Feast
US startup, greenfield stack	Q1 no → Q3 no → Q4 no	Tecton

The trace makes the trade-offs explicit: streaming-heavy + cash-rich → Tecton; sovereign → Hopsworks; existing infra → Feast; greenfield without on-prem → Tecton. Every other team is some shade of one of those four.

Output:

Pick	When	Common follow-up
Feast	DIY-capable team with existing stores	"how do we handle streaming?"
Tecton	streaming-heavy or greenfield team with SaaS budget	"what is our exit plan?"
Hopsworks	sovereignty-constrained or full-platform-wanting	"do we self-host or buy managed?"

Why this works — concept by concept:

Streaming dominance as the first cut — sub-second streaming is the single largest cost differential between vendors. Asking it first prunes the tree fastest.
Budget as the gating filter — SaaS pricing is not negotiable below certain volumes. A blank "we can pay anything" answer is rare; the budget conversation belongs in the second question, not the last.
Sovereignty as the third cut — EU / on-prem / air-gap constraints eliminate Tecton entirely and steer toward Hopsworks. If the constraint exists, every other dimension is secondary.
Existing infra fit as the tiebreaker — for the median team (no streaming dominance, no sovereignty constraint, modest budget), the deciding factor is "what do you already operate well?" Snowflake + Redis → Feast; nothing yet → Tecton.
Cost — the lifetime cost of the platform is dominated by ops headcount on Feast, SaaS bills on Tecton, and license + ops on Hopsworks. Quote each in the decision deck.

DE
Topic — design
Platform design problems (DE)

Practice →

5. Training-to-serving lifecycle in production

The lifecycle is a triangle — training reads the offline store, materialization moves features into the online store, serving reads the online store, and monitoring closes the loop with drift + freshness + fill-rate signals

The mental model in one line: training builds the model artifact from offline features; materialization keeps the online store fresh; serving hydrates inference inputs from the online store; monitoring watches the offline-vs-online distribution; backfills replay history through the same view; deprecations follow a 30-day shadow window — six stages, one feature definition — and a senior data engineer can walk through each stage at the whiteboard without notes. Once you can name the six stages and the artifact each produces, the production-ML interview is mostly done.

The six stages in detail.

Training. A point-in-time join of labels to features produces the training DataFrame; the model is fit and the artifact (pickled, ONNX, or vendor format) is registered. Reads from the offline store; touches no production infra.
Materialization. A scheduled batch job and/or a continuous streaming job pushes the latest feature value per entity into the online store. Same feature view; cadence per feature.
Serving. The inference service receives an entity key, calls get_online_features to hydrate the input vector, and runs the model. Latency budget is the model's SLA minus the online lookup minus the network — typically <100 ms end to end.
Monitoring. Three signals run continuously: drift (offline vs online distribution), freshness (lag between source and online write), fill rate (% of entities with non-null values). All three feed dashboards; the second two page on-call.
Backfills. Replay historical data through the same feature view to compute features for a new label window. Critical when a new model needs features that were never materialised before, or when a bug is fixed and history must be recomputed.
Governance. Feature ownership is recorded in the registry; schema evolution is additive; deprecations follow a 30-day shadow window (mark tombstone: 2026-08-01, dual-write for a month, then drop). Lineage is queryable from the registry.

The training path in three steps.

Step 1 — build the spine. A DataFrame of (entity, event_ts, label) rows. One row per training example. Comes from the labels table and the chosen time window.
Step 2 — point-in-time join. Call get_historical_features(spine, [features]). The SDK does an AS-OF join against the offline store for each feature, returns the spine + feature columns.
Step 3 — fit and register. Train the model; register the artifact with the feature-view list it depends on. The registration creates the lineage edge: model_v3 → uses → user_7d_orders v2.

The serving path in three steps.

Step 1 — receive entity. The inference request arrives with an entity key (user_id, txn_id, etc).
Step 2 — online lookup. Call get_online_features([entities], [features]). The SDK GETs the online store, returns a dict of feature values.
Step 3 — infer. The model consumes the feature dict, returns the prediction. The serving service writes the request + features + prediction to a log table for audit and for the drift monitor.

The closure: serving logs feed the monitor.

The serving service writes every (entity, features, prediction, ts) tuple to an audit log.
The drift monitor samples the audit log and compares it to the offline distribution every 15 minutes.
When drift exceeds the threshold, the monitor pages on-call and posts to the model's incident channel.

Worked example — building the training dataset with a point-in-time join

Detailed explanation. The spine + AS-OF join is the only correct way to build a training dataset that matches the serving-time data distribution. The SDK does the heavy lifting; what you have to get right is the spine — every (entity, event_ts) for which a label exists.

Question. Build a training DataFrame for a fraud model using txn_id as entity, the label "is_fraud" at txn_ts, and three features (user_7d_orders, user_lifetime_orders, txn_velocity_60s). Show the spine and the SDK call.

Input.

txn_id	user_id	txn_ts	is_fraud
100	1	2026-05-01 10:00	0
101	1	2026-05-15 11:00	1
102	2	2026-05-20 09:00	0

Code.

import pandas as pd
from feast import FeatureStore

store = FeatureStore(repo_path="feast_repo")

# Step 1 — build the spine
labels = pd.DataFrame([
    {"txn_id": 100, "user_id": 1, "txn_ts": "2026-05-01 10:00", "is_fraud": 0},
    {"txn_id": 101, "user_id": 1, "txn_ts": "2026-05-15 11:00", "is_fraud": 1},
    {"txn_id": 102, "user_id": 2, "txn_ts": "2026-05-20 09:00", "is_fraud": 0},
])
labels["txn_ts"] = pd.to_datetime(labels["txn_ts"])

# Step 2 — point-in-time join
training = store.get_historical_features(
    entity_df=labels.rename(columns={"txn_ts": "event_timestamp"}),
    features=[
        "user_features:user_7d_orders",
        "user_features:user_lifetime_orders",
        "txn_features:txn_velocity_60s",
    ],
).to_df()

# Step 3 — fit
X = training.drop(columns=["txn_id", "user_id", "event_timestamp", "is_fraud"])
y = training["is_fraud"]
model.fit(X, y)

Step-by-step explanation.

The spine is the labels table renamed so the timestamp column is called event_timestamp (Feast's convention). Each row is one training example with the timestamp at which the label is true.
get_historical_features does an AS-OF join per feature: for each spine row's (user_id, event_timestamp), it fetches the most recent feature value with feature_event_ts ≤ event_timestamp. Three features = three independent AS-OF joins, all anchored on the same spine.
The returned DataFrame has the spine columns plus the three feature columns. Drop the entity / timestamp / label columns to get X; the label is y.
The model fit happens entirely offline — no production infra is touched. The artifact is registered with a pointer to the feature views it depends on, so future schema changes can be flagged before deployment.

Output.

txn_id	event_timestamp	user_7d_orders	user_lifetime_orders	txn_velocity_60s	is_fraud
100	2026-05-01 10:00	5	47	2	0
101	2026-05-15 11:00	1	49	12	1
102	2026-05-20 09:00	3	21	1	0

Rule of thumb. The training spine is always (entity, event_ts, label). Never train on a "snapshot" of features at one time — that is the leakage pattern. Always let the SDK do the AS-OF join.

Worked example — backfilling a new feature through the same view

Detailed explanation. A new feature is added to an existing view. The historical values must be computed for every (entity, event_ts) in the offline store before the next training run can use it. This is a backfill — and it goes through the same feature view definition, so the historical values match what serving will see.

Question. Add a new user_avg_basket_30d feature to the existing user_features view, backfill 6 months of history, and verify that training queries can see it. Show the SDK calls and the verification step.

Input.

Step	Action
1	Add `user_avg_basket_30d` to the feature view definition
2	Apply the registry change
3	Backfill 6 months in 1-day chunks
4	Verify the feature is queryable on a spine

Code.

# Step 1 — extend the feature view
user_features = FeatureView(
    name="user_features",
    entities=[user],
    ttl=timedelta(hours=24),
    schema=[
        Field("user_7d_orders", Int64),
        Field("user_lifetime_orders", Int64),
        Field("user_avg_basket_30d", Float64),   # <-- new
    ],
    source=user_features_source,
)

# Step 2 — register
# CLI: $ feast apply

# Step 3 — backfill in 1-day chunks for 180 days
from datetime import datetime, timedelta
end = datetime(2026, 6, 15)
for d in range(180):
    chunk_end = end - timedelta(days=d)
    chunk_start = chunk_end - timedelta(days=1)
    store.materialize(
        feature_views=["user_features"],
        start_date=chunk_start,
        end_date=chunk_end,
    )

# Step 4 — verify
spine = pd.DataFrame({
    "user_id": [1, 1, 1],
    "event_timestamp": pd.to_datetime(
        ["2026-01-15", "2026-03-15", "2026-05-15"]),
})
df = store.get_historical_features(
    entity_df=spine,
    features=["user_features:user_avg_basket_30d"],
).to_df()
assert df["user_avg_basket_30d"].notna().all(), "backfill left gaps"

Step-by-step explanation.

Adding a feature to an existing view is an additive schema change — non-breaking. Existing consumers continue to read the old columns; new consumers can ask for the new column.
After feast apply, the registry knows about the new feature but the offline store has no historical values for it yet. Training queries that ask for it would return NULL — the backfill fixes that.
The backfill iterates in 1-day chunks, each one calling materialize for the new column over a small time window. Chunking limits the warehouse query size and lets the job resume on failure (process one day at a time, checkpoint per day).
The verification step does an AS-OF join on a 3-month-old spine and asserts the new column is non-NULL. Catches the "you forgot to backfill January" bug at the end of the migration instead of in a model's training run a week later.

Output.

user_id	event_timestamp	user_avg_basket_30d
1	2026-01-15	42.50
1	2026-03-15	51.20
1	2026-05-15	38.75

Rule of thumb. Backfill in chunks the same size as the source partition. If your source is daily-partitioned, backfill in 1-day chunks. Smaller chunks let you resume on failure; larger chunks waste compute on re-scans.

Worked example — deprecating a feature with a 30-day shadow window

Detailed explanation. A feature can almost never be deleted instantly — downstream models read it, and a delete is a production outage. The standard deprecation pattern is the 30-day shadow window: mark the feature deprecated, dual-write (or freeze writes) for 30 days, watch reads decay to zero, then delete.

Question. Deprecate user_avg_basket_legacy over 30 days while a new user_avg_basket_30d takes over. Show the registry tombstone, the reader audit, and the final delete.

Input.

Day	Action
0	Tombstone the feature; announce to consumers
0–30	Dual-read both; consumers migrate
30	Verify no reads; delete

Code.

# Day 0 — registry tombstone (Feast: tags; Tecton/Hopsworks: built-in deprecation field)
user_features = FeatureView(
    name="user_features",
    entities=[user],
    schema=[
        Field("user_avg_basket_legacy", Float64,
              tags={"tombstone_date": "2026-07-15"}),
        Field("user_avg_basket_30d", Float64),
    ],
    source=user_features_source,
)

# Day 0–30 — read audit: who is still calling for the deprecated column?
audit = pd.read_sql("""
    SELECT feature_view, feature_name, COUNT(*) AS reads
    FROM mart.feast_read_log
    WHERE log_ts >= now() - interval '7 days'
      AND feature_name = 'user_avg_basket_legacy'
    GROUP BY feature_view, feature_name
""", warehouse)

# Day 30 — verify, then delete
assert audit["reads"].sum() == 0, "still readers!"
# Remove the field from the FeatureView and re-apply

Step-by-step explanation.

Day 0: add a tombstone_date tag to the feature in the registry. Announce in the platform channel. Consumers see the tombstone in the registry UI; the SDK can be configured to log a deprecation warning when the feature is requested.
Day 0–30: the deprecated feature continues to write and serve as normal. The read audit (queryable from the feature server's log table) tracks who is still calling for it.
Each consumer migrates on its own schedule — drop the deprecated feature from their training spine, point at the new feature, and verify the next training run still converges.
Day 30: audit shows zero reads in the last 7 days. Remove the field from the feature view, re-apply, and the column is gone. The actual data in the offline store stays (it is cheap to keep history); only the registry binding is removed.
If reads are non-zero at day 30, extend the window. Hard-deleting a feature with live readers is a production outage — never worth the speed.

Output.

Day	Reads/week	Action
0	1,200	tombstone added
7	420	one consumer migrated
21	35	two more consumers migrated
30	0	safe to delete

Rule of thumb. 30 days is the minimum shadow window. Extend it (60 days, 90 days) if consumers are slow to migrate or if the feature is read by a model with quarterly retraining. The cost of keeping the feature is rounding error; the cost of an outage is not.

Interview question on the full lifecycle

A senior interviewer often asks: "Walk me through the lifecycle of a single feature from definition to deprecation, including every production system it touches."

Solution Using the six-stage lifecycle

Stage 1 — DEFINE
  - Author the feature view (YAML / Python) in source control.
  - Code review by the feature owner + the platform DE on-call.
  - Merge -> CI runs `feast apply --dry-run` to validate the registry.

Stage 2 — MATERIALIZE
  - Scheduled (batch) or continuous (streaming) job writes the feature
    to BOTH offline and online stores.
  - Materialization status surfaces in the platform dashboard.

Stage 3 — TRAIN
  - Training job builds a spine of (entity, event_ts, label).
  - get_historical_features() returns the AS-OF-joined training DataFrame.
  - Model artifact registered with feature-view lineage.

Stage 4 — SERVE
  - Inference service calls get_online_features() per request.
  - Online store lookup, model inference, prediction returned.
  - Audit log entry written (entity, features, prediction, ts).

Stage 5 — MONITOR
  - Drift monitor compares last-hour online sample to last-week offline sample.
  - Freshness monitor watches lag between source event and online write.
  - Fill-rate monitor watches % non-null per feature.
  - Alerts page on-call when thresholds breached.

Stage 6 — DEPRECATE
  - Mark tombstone_date on the feature in the registry.
  - Read audit tracks remaining consumers; nudge them to migrate.
  - After zero reads for a week (or 30 days, whichever later), delete.

Step-by-step trace.

Stage	Artifact	Production system touched
Define	feature view YAML / Python	git, registry DB
Materialize	offline rows + online row	warehouse, Redis / DynamoDB
Train	model artifact	model registry
Serve	prediction + audit log row	online store, audit log table
Monitor	drift / freshness / fill-rate metric	metrics store, paging system
Deprecate	tombstone tag + zero-read audit	registry, audit log

The trace highlights that every stage writes to a different production system — and that the registry is the single source of truth that ties them all together. A senior DE can name each system and what fails when it does.

Output:

Stage	Owner	On-call cadence
Define	feature author	code-review only
Materialize	platform DE	weekday business hours
Train	DS / MLE	as-needed
Serve	platform SRE	24/7
Monitor	platform DE	24/7 (drift + freshness page)
Deprecate	feature author + platform DE	as-needed

Why this works — concept by concept:

One feature definition across six stages — the registry is the contract. Every stage either writes through the registry or reads through it; nothing goes around.
Materialization as the bridge — without it, training and serving see different data. The bridge job is the only thing standing between the offline store and the online store, and that is why its monitoring is non-negotiable.
Audit log as the closure — the serving service's log table is what feeds the drift monitor. Without the log, drift is invisible. Without drift monitoring, the production model degrades silently.
Lineage as the safety net — every model artifact knows which feature views it depends on. Schema changes to a view automatically flag the dependent models for re-review.
Cost — define is free; materialize is recurring (warehouse + streaming compute); train is bursty (per-experiment); serve is per-request (online store reads); monitor is constant (15-minute cron); deprecate is free. The dominant line item is online reads at high QPS.

DE
Topic — time-series
Time-series aggregation problems (DE)

Practice →

Worked example — wiring the serving service to fall back when the online store misses

Detailed explanation. Online stores miss. The entity is new, the TTL has expired, the pipeline stalled. The serving service has to choose a fallback per missing feature — impute a default, fall back to a simpler model, or gate the request. The choice is part of the feature contract, not an afterthought.

Question. A scoring endpoint asks for three features. One returns NULL. Show the per-feature fallback policy and the gating logic.

Input.

Feature	Fallback
`user_7d_orders`	impute 0
`user_lifetime_orders`	impute 0
`txn_velocity_60s`	gate request (return 500) — required

Code.

def score(entity: dict) -> float:
    features = store.get_online_features(
        features=[
            "user_features:user_7d_orders",
            "user_features:user_lifetime_orders",
            "txn_features:txn_velocity_60s",
        ],
        entity_rows=[entity],
    ).to_dict()

    fallback = {
        "user_7d_orders": ("impute", 0),
        "user_lifetime_orders": ("impute", 0),
        "txn_velocity_60s": ("gate", None),
    }

    inputs = {}
    for name, (policy, default) in fallback.items():
        v = features.get(name, [None])[0]
        if v is None:
            if policy == "gate":
                raise FeatureMissing(name)
            v = default
            metrics.increment("feature.impute", tags={"feature": name})
        inputs[name] = v

    return model.predict_proba([list(inputs.values())])[0][1]

Step-by-step explanation.

Each feature is classified at the contract layer as either "imputable" (model still works without it; substitute a default) or "required" (model meaningfully degrades without it; gate the request).
The serving function checks each returned value. NULL with impute policy substitutes the default and logs a metric. NULL with gate policy raises FeatureMissing, which the API layer turns into a 500.
The metrics.increment call makes silent imputations visible. A spike in feature.impute for user_7d_orders is the on-call's first signal that materialization stalled.
Gating on a required feature surfaces immediately as a client-visible 500. The page is loud; the fix is upstream (restart the materialization job).

Output.

Feature value	Policy	Outcome
5 / 47 / 12	normal	scores
NULL / 47 / 12	impute	scores with imputed 0; metric logged
5 / 47 / NULL	gate	500 returned; on-call paged

Rule of thumb. Every feature ships with a documented fallback policy. "What happens if this feature is NULL at serve time?" is part of the registry, not an emergent property of the serving code.

Cheat sheet — feature store recipes

Define a feature view. Name + entity + source + schema + TTL. Keep transformations in the source (Feast) or in the framework (Tecton). Tag with owner + freshness.
Point-in-time training join. get_historical_features(spine_df, features=[...]) — always. Never JOIN ON entity_key alone; that leaks the future into the past.
Online materialization cadence. Nightly batch for 24h-freshness features; hourly batch for 1h-freshness; streaming (Bytewax / Flink / Tecton Rift) for sub-minute features. Pick per feature, not globally.
Online lookup SLA. Target P99 < 25 ms for a single-entity multi-feature read. Anything above 50 ms means the model's end-to-end SLA is breached on cold cache.
TTL on the online store. Set TTL = 2–3x materialization cadence. Bound staleness without triggering false fallbacks on transient lag.
NULL-handling at serve time. Classify each feature as impute or gate. Encode the fallback in the serving service; surface imputation rate as a metric.
Drift monitor. KS-distance (or PSI) between last-hour online sample and last-week offline sample. Alert at KS > 0.2; investigate at KS > 0.1.
Freshness monitor. Lag between source event timestamp and online write timestamp. Alert at p95 > 2x SLA.
Fill-rate monitor. % of entities with a non-null value. Alert at <99% on a stable population.
Backfill in chunks. One source-partition per chunk. Resume on failure; checkpoint per chunk. Verify on a 3-month-old spine before declaring done.
Deprecation shadow window. 30 days minimum. Tombstone in the registry; audit reads; delete only at zero reads.
Feast vs Tecton vs Hopsworks. Feast for DIY + cost. Tecton for streaming velocity + managed. Hopsworks for sovereignty + full platform. Decide by streaming dominance, budget, residency, and existing infra fit — in that order.
Vendor exit plan. Every vendor needs one. Tecton → Feast on Snowflake + Redis in 8–12 weeks for a 50-feature shop. Feast → side-car streaming for the 2–3 sub-second features. Hopsworks → rebase fork quarterly.
Lineage from model to view. Every model artifact records the feature views it depends on. Schema changes flag dependent models in CI before deployment.

Frequently asked questions

Do I need a feature store?

Adopt a feature store when (a) you have two or more models that share features, (b) your serving SLA drops below 1 second, OR (c) you have 50+ features across teams. Below those thresholds, a single warehouse query and good naming discipline are cheaper than a platform. The first deliverable should be the deprecation of duplicate features in the warehouse, not a brand new feature — reuse is what justifies the platform tax. If your team is a single DS + DE shipping one batch model with 14 features, you do not need one yet; revisit when the second model lands.

Feast vs Tecton vs Hopsworks — which fits my team?

Use a four-question decision tree: (1) is streaming the dominant pattern with sub-second freshness? — yes leads to Tecton (managed) or Hopsworks (sovereign / cost-sensitive); (2) is there an EU residency / on-prem constraint? — yes leads to Hopsworks; (3) does the team already operate Snowflake / BigQuery + Redis / DynamoDB well? — yes leads to Feast; (4) is the team greenfield with SaaS budget? — yes leads to Tecton. The vast majority of US fintech / SaaS teams land on Feast because they already operate the building blocks. The vast majority of regulated EU teams land on Hopsworks. The vast majority of streaming-velocity shops with SaaS budget land on Tecton. Score against your top three constraints, not against marketing copy.

What is a point-in-time join and why does it matter?

A point-in-time (AS-OF) join attaches feature values to label rows by picking the most recent feature value with feature_event_ts ≤ label_event_ts. Without it, a naive JOIN ON user_id grabs the latest feature value — usually computed after the label timestamp — and the model "sees the future" during training. Production accuracy then falls dramatically below the offline test set. Every feature store SDK (Feast get_historical_features, Tecton get_features_for_events, Hopsworks as_of) implements AS-OF semantics; modern engines (Snowflake ASOF JOIN, Databricks as_of_join, DuckDB ASOF) ship it natively. If your training join lacks a time predicate, your model has leaked — there is no exception to this rule.

How does the online store stay fresh?

Materialization. Either a scheduled batch job (nightly / hourly) scans the source for new feature values and writes them keyed by entity to Redis / DynamoDB / Cassandra / Bigtable, or a continuous streaming job (Flink / Bytewax / Spark Streaming) maintains per-entity rolling state and pushes updates every few seconds. The cadence is per-feature: 24h-freshness features cost nothing extra to materialize nightly off the warehouse query that already runs; sub-second features cost a continuously-running Flink slice. A TTL on the online store (typically 2–3x the materialization cadence) acts as the circuit breaker — when the pipeline stalls, the SDK starts returning NULL within the TTL window and the freshness monitor pages on-call.

Can I use a warehouse as my online store?

No, at any meaningful QPS. Warehouses (Snowflake / BigQuery / Redshift) are columnar and optimised for full-table scans; their single-row lookup latency is seconds, not milliseconds, and their cost per query is orders of magnitude above a Redis GET. The offline / online split exists precisely because no single storage class handles both access patterns well. The narrow exception is batch scoring (no real-time SLA): you can score a million rows offline by reading features directly from the warehouse — but that is not "serving," that is another batch job. The moment a model has a real-time inference path, you need an online store backed by a low-latency KV system.

How do I monitor feature drift in production?

Run three monitors continuously on every production feature: (1) drift — KS distance or PSI between a sample of the last hour of online reads and a sample of the last week of offline values; alert at KS > 0.2; (2) freshness — p95 lag between source event timestamp and online write timestamp; alert at 2x the freshness SLA; (3) fill rate — % of served entities with a non-null value; alert at <99% on a stable population. The drift monitor catches the "training-serving skew" failure mode; the freshness monitor catches stalled pipelines; the fill-rate monitor catches new entities arriving faster than materialization. All three feed the same on-call dashboard and version with the feature view definition in source control.

Practice on PipeCode

Drill the streaming practice library → for the per-entity rolling-window logic that powers most online features.
Rehearse on ETL pipeline problems → to internalise the offline → online materialization shape and the failure modes.
Sharpen system-design drills → for the registry + offline + online + serving + monitor whiteboard rounds.
Layer the aggregation library → for the COUNT / SUM / AVG patterns that show up inside every batch feature view.
Stack the time-series practice library → for the rolling-window and AS-OF semantics that drive point-in-time joins.
Work the window-functions library → for the SQL muscle behind LAG, LEAD, and rolling aggregates inside feature views.
For the broader surface, read top data engineering interview questions →.
Stack the prerequisites with the only 5 skills you need to become a data engineer →.
Sharpen the SQL axis with the SQL for data engineering interviews course →.
For long-form pipeline craft, work through ETL system design for DE interviews →.
For the platform context, layer Apache Spark internals for DE interviews →.

Pipecode.ai is Leetcode for Data Engineering — every feature store recipe above ships with hands-on practice rooms where you write the point-in-time join, the materialization job, and the online lookup against real graded inputs. PipeCode pairs every reading with 450+ DE-focused problems and a real-time scoring engine, so you never have to wonder whether your AS-OF join would actually behave the same on Snowflake as on Databricks — or whether your Feast feature view would survive a vendor migration to Tecton or Hopsworks.

Practice streaming features now →
ETL pipeline drills →

Reverse ETL with Hightouch, Census & RudderStack: Operational Analytics in Practice

Gowtham Potureddi — Wed, 17 Jun 2026 12:53:20 +0000

reverse etl is the discipline that closes the loop a data team starts the first time it lands raw events in a warehouse and then realises the warehouse, however beautiful, is invisible to the GTM team. Forward ETL moved source data into the warehouse so analysts could ask questions; reverse ETL ships the answers back out into the operational tools — Salesforce, HubSpot, Marketo, Intercom, Slack, Facebook Ads, Iterable — where the people and systems that act on customers actually live. It is the bridge between analytical truth and operational action, and in 2026 it is the single fastest-growing surface in the modern data stack.

This guide walks the practitioner's view of operational analytics end to end. It defines the data activation pattern (model → audience → sync → destination), compares the three production-grade reverse etl tools — Hightouch, Census, and RudderStack — across destinations, dbt integration, hosting, and pricing, deconstructs the sync architecture that turns a warehouse query into a queue of API calls absorbing 429s and dead letters, and lays out the governance and observability layer that distinguishes a real data product from a fragile pipeline. Each section pairs a teaching block with a Solution-Tail worked answer — code, a step-by-step trace, an output table, and a concept-by-concept breakdown of why it works.

When you want hands-on reps while reading, drill the ETL practice library →, layer in API integration drills →, and stack the warehouse muscles with dimensional modelling problems →.

On this page

Why reverse ETL exists — operational analytics as a discipline
The reverse ETL data model — models, audiences, syncs
Hightouch vs Census vs RudderStack — vendor comparison
Sync architecture — incremental detection, queues, rate limits
Governance, observability, and failure modes
Cheat sheet — reverse ETL recipes
Frequently asked questions
Practice on PipeCode

1. Why reverse ETL exists — operational analytics as a discipline

Forward ETL moves data INTO the warehouse so analysts can ask questions; reverse ETL moves data OUT so operational systems can act

The one-sentence invariant: forward ETL turns raw source data into warehouse rows that humans read on dashboards; reverse ETL turns those warehouse rows back into API calls that machines and SaaS tools execute against customers. Once you internalise that the warehouse is now the source of truth for every customer attribute, the question stops being "should we sync this?" and becomes "which destinations, which fields, how often, and with what governance?"

The data activation gap in three bullets.

Dashboards inform people; syncs inform systems. A lead score in Looker is a number a manager looks at on Monday. A lead score in Salesforce is a field a routing rule reads at midnight to assign the lead to the right rep. The two consumers want the same number but through different surfaces.
The warehouse aggregates across silos; SaaS tools cannot. Stripe knows about payments. HubSpot knows about emails. The product database knows about feature usage. Only the warehouse joins them. Reverse ETL ships that join back into every silo.
Manual CSV exports do not scale. A "send a CSV to ops once a week" workflow has zero observability, no schema contract, and breaks the first time a column is renamed. Reverse ETL turns the export into a versioned, scheduled, monitored data product.

Common destinations in 2026.

CRMs. Salesforce, HubSpot, Microsoft Dynamics, Pipedrive.
Marketing automation. Marketo, Iterable, Customer.io, Braze, Klaviyo, Mailchimp.
Support + success. Intercom, Zendesk, Gainsight, Vitally, ChurnZero.
Ad platforms. Facebook / Meta custom audiences, Google Ads customer match, TikTok audiences, LinkedIn matched audiences.
Collaboration + ops. Slack channels, Microsoft Teams webhooks, Notion databases, Asana tasks.
Product analytics. Amplitude cohorts, Mixpanel cohorts, Heap audiences.

Why the warehouse won as source of truth.

Compute and storage are now cheap. Snowflake, BigQuery, Databricks, Redshift — every cloud warehouse runs the joins at a price that makes "send the join result downstream" feasible at scale.
dbt made transformation governable. Once models/marts/customers.sql is the single SQL definition of "a customer," every downstream system can subscribe to its rows instead of recomputing them.
Data teams finally have leverage on the operational stack. Reverse ETL gives the data team a contract with marketing, sales, and CS without writing custom Python in five different SaaS APIs.

When NOT to use reverse ETL.

Sub-second latency requirements. Reverse ETL is a batch + micro-batch architecture. Hightouch ships syncs as fast as ~5 minutes; Census as fast as ~1 minute; RudderStack with streaming-event reverse ETL can hit seconds. Below that, you want event streaming (RudderStack event stream, Segment, Kafka → consumer) — not warehouse syncs.
True event streaming. "Page view fires → personalisation engine reacts in 200ms" is not a reverse ETL problem; it is a Kafka / Kinesis / event-bus problem.
One-off backfills. A 50k-row one-time list does not need a sync pipeline; a CSV import inside the destination is faster.

Worked example — the lead score sync that justifies reverse ETL

Detailed explanation. A B2B SaaS company computes a lead score in dbt by joining Salesforce contacts, product usage events, and marketing engagement. The score lives in marts.lead_scores. Sales wants the same score visible on the Salesforce Contact record so routing and prioritisation rules can act on it. Without reverse ETL the team writes a custom Python script, schedules it in Airflow, builds retries, builds dedupe, and rebuilds it every time the score model changes. With reverse ETL the team writes a one-page sync definition and inherits all of that infrastructure.

Question. Given the dbt model marts.lead_scores with columns (salesforce_contact_id, lead_score, last_engagement_at, churn_risk), how do you ship the row into Salesforce Contact.lead_score__c, Contact.last_engagement_at__c, and Contact.churn_risk__c so that routing rules can act on it?

Input — marts.lead_scores.

salesforce_contact_id	lead_score	last_engagement_at	churn_risk
003A1	87	2026-06-12	0.12
003A2	41	2026-05-30	0.55
003A3	92	2026-06-14	0.08
003A4	NULL	NULL	NULL

Code.

-- The dbt model that becomes the sync source.
-- File: models/marts/lead_scores.sql
SELECT
    c.salesforce_contact_id,
    ROUND(s.raw_score, 0)                       AS lead_score,
    s.last_engagement_at,
    s.churn_risk
FROM {{ ref('dim_contacts') }}            c
LEFT JOIN {{ ref('int_lead_scoring') }}   s
       ON s.contact_id = c.contact_id
WHERE c.salesforce_contact_id IS NOT NULL

# Hightouch sync definition (illustrative YAML).
# File: hightouch/syncs/salesforce_lead_score.yaml
model: marts.lead_scores
destination: salesforce_production
sync_mode: upsert
primary_key: salesforce_contact_id
schedule: "*/30 * * * *"   # every 30 minutes
mappings:
  - source: lead_score          -> Contact.lead_score__c
  - source: last_engagement_at  -> Contact.last_engagement_at__c
  - source: churn_risk          -> Contact.churn_risk__c

Step-by-step explanation.

The dbt model produces one row per Salesforce contact with a stable salesforce_contact_id primary key. The model is the contract — change the SQL, change every downstream consumer.
Hightouch reads the model on the cron schedule. On the first run it stores a snapshot; on every later run it diffs the current rows against the previous snapshot to find changes.
The sync_mode upsert tells the destination "insert if salesforce_contact_id does not exist, update otherwise." Salesforce External ID matching is configured in the Hightouch UI to map salesforce_contact_id to Salesforce's Id field.
The three field mappings turn warehouse columns into Salesforce custom fields. NULL lead_score for 003A4 becomes a blank update on the Salesforce field; the destination keeps any previous value if the sync setting is "do not overwrite with NULL."
The cron */30 runs every 30 minutes — far below Salesforce's daily API limit but fast enough for sales routing.

Output (Salesforce after sync).

Salesforce Contact Id	lead_score__c	last_engagement_at__c	churn_risk__c
003A1	87	2026-06-12	0.12
003A2	41	2026-05-30	0.55
003A3	92	2026-06-14	0.08
003A4	(unchanged)	(unchanged)	(unchanged)

Rule of thumb. Every operational team that asks for "a number on the record so we can route on it" is asking for reverse ETL. Push back when they ask for "a CSV every Monday" — propose the sync instead, because it ships with observability, history, and a schema contract for free.

Worked example — the dashboards-vs-syncs contrast

Detailed explanation. A common mistake is treating a dashboard and a sync as the same artefact with a different surface. They are not. A dashboard runs on demand and serves humans; a sync runs on a schedule and serves machines. Different SLA, different failure mode, different governance, different consumer.

Question. Given a churn-risk metric, write the two access patterns side by side — Looker dashboard query vs reverse ETL sync — and explain why both exist.

Input.

Surface	Cadence	Consumer	Failure mode
Looker dashboard	on demand	account manager	empty card
Reverse ETL sync	every 6h	Intercom tag automation	stale tag

Code.

-- Looker explore (shared view).
-- explore: account_health
-- view: marts.account_health
view: account_health {
  sql_table_name: marts.account_health ;;
  measure: avg_churn_risk {
    type: average
    sql: ${TABLE}.churn_risk ;;
  }
}

# Hightouch sync — same underlying model, machine surface.
model: marts.account_health
destination: intercom
sync_mode: mirror
primary_key: account_id
schedule: "0 */6 * * *"
mappings:
  - source: churn_risk                 -> Company.churn_risk_attr
  - source: CASE WHEN churn_risk > 0.7
            THEN 'at_risk' ELSE 'ok' END  -> Company.health_tag

Step-by-step explanation.

The same marts.account_health model feeds both surfaces. There is exactly one definition of "churn risk" in the company.
The dashboard query runs when a human opens it. The SLA is "the query returns in less than 10 seconds and the number is no older than the last warehouse refresh."
The Hightouch sync runs every 6 hours regardless of human attention. The SLA is "the Intercom tag reflects yesterday's risk score by the end of every 6-hour window."
Failure modes differ: a dashboard failure is loud (empty card, error toast); a sync failure is quiet (a stale tag still looks like data). Observability for the sync must be explicit.

Output.

Surface	Behaviour when warehouse fails
Looker dashboard	error visible immediately to the user
Hightouch sync	last successful tag persists; alert fires only if observability is set up

Rule of thumb. Treat the sync as a different product than the dashboard, even when both subscribe to the same model. Stamp a SLA on the sync, add an explicit row-error alert, and surface the sync as a dbt exposure so it shows up in lineage.

Worked example — when reverse ETL is the wrong tool

Detailed explanation. Reverse ETL has a lower bound on latency around a minute (Census) and a typical floor of 15–30 minutes for cost-efficient syncs (Hightouch on shared infrastructure). For sub-second personalisation, fraud-blocking, or in-session experiences, reverse ETL is the wrong tool — you need an event stream.

Question. Given a "personalise the homepage banner based on the user's churn risk" requirement, decide between reverse ETL and an event-stream architecture. Show the latency budget that drives the decision.

Input.

Requirement	Latency target	Architecture
Sales Salesforce score	30 minutes	reverse ETL
Marketing Intercom tag	6 hours	reverse ETL
Ad audience refresh	24 hours	reverse ETL
Homepage personalisation	< 500 ms	event stream
Fraud block at checkout	< 200 ms	online ML feature store

Code.

Decision rubric (pseudo-code):

if latency_target >= 5_minutes:
    use reverse_etl (Hightouch / Census / RudderStack)

elif latency_target >= 30_seconds:
    use event_stream_reverse_etl (RudderStack event stream)

elif latency_target >= 100_ms:
    use online_feature_store + low_latency_api (Tecton, Feast, custom)

else:
    use in_request_compute (edge function, cached cache lookup)

Step-by-step explanation.

The latency floor for batch reverse ETL is a function of warehouse query time + diff computation + destination API throughput. On a shared tenant in Hightouch this typically lands at 5–15 minutes.
RudderStack's event-stream reverse ETL closes the loop in seconds for individual event triggers but still cannot serve a single-millisecond synchronous API call.
Online ML feature stores (Tecton, Feast) maintain a serving layer separate from the warehouse precisely for sub-100ms reads. Reverse ETL pre-materialises features into that layer on a slower cadence.
The rubric ranks tools by the actual latency budget the use case requires. Picking the wrong tier wastes either money (using a feature store for a daily ad audience) or signal (using reverse ETL for sub-second personalisation).

Output.

Use case	Tool	Why
Lead score in Salesforce	Hightouch upsert	30-min cadence, batch fine
Churn risk tag in Intercom	Census sync	6h cadence, batch fine
Homepage banner	Edge feature read	sub-500ms, batch insufficient
Fraud rule at checkout	Online feature store	sub-200ms, must be pre-materialised

Rule of thumb. Sketch the latency budget first. Anything above 5 minutes is a reverse ETL problem. Anything below 5 minutes is a streaming or feature-store problem. Mixing the two architectures costs more than picking the right one from the start.

Reverse ETL interview question on the lift-up from forward ETL

A senior interviewer often asks: "Walk me through what changes in the data team's responsibility model when reverse ETL enters the stack. What dbt practices have to harden? What new SLAs do you accept?"

Solution Using the data activation contract

The data team takes on three new responsibilities the day reverse ETL ships:

1. Model stability is now an operational SLA.
   - Every sync model needs a stable primary key (renaming it
     breaks identity resolution downstream).
   - Column renames now break SaaS-tool fields that humans rely on.
   - Type changes can silently corrupt destination fields.
   - Solution: dbt contract tests + dbt exposures + protected branch
     for any model with downstream syncs.

2. Freshness is now a destination-level SLA.
   - Warehouse "fresh as of midnight" is no longer enough.
   - Each destination has its own freshness contract (Salesforce: 30m,
     Intercom: 6h, Facebook ads: 24h).
   - Solution: per-sync alerting, last_synced_at columns, freshness
     dashboards.

3. Governance now spans warehouse + SaaS tools.
   - PII synced to Marketo is now subject to Marketo's retention.
   - GDPR delete must propagate to every destination.
   - Solution: PII tags on every column, per-destination policy,
     destination-side deletes.

Step-by-step trace.

Responsibility	Before reverse ETL	After reverse ETL
Model PK stability	nice-to-have	hard contract
Column rename	dashboard fix	downstream sync break
Freshness	warehouse-wide	per-destination
PII	warehouse policy	propagated to N SaaS tools
Lineage	dbt + BI	dbt + BI + syncs

The data team learns to think of every model with at least one sync as an operational data product. The discipline is closer to backend engineering than to "writing SQL" — versioned, monitored, alerted, paged.

Output:

Practice	New requirement once reverse ETL is in the stack
dbt contracts	required on every sync model
dbt exposures	every sync surfaced in lineage
PII tagging	per-column tags propagated to destination policy
Alerting	per-sync row-error rate and freshness SLA
On-call	one person owns sync health

Why this works — concept by concept:

Models become contracts — the (primary_key, columns, types) tuple is now a stable API. Any change is a versioned migration with downstream blast-radius assessment.
Freshness becomes per-destination — the warehouse SLA is the upper bound; each sync has its own, often tighter, freshness contract because downstream SaaS automation acts on it.
PII becomes propagated — a column tagged "email PII" in the warehouse must inherit the same handling everywhere it lands. GDPR delete is the canonical stress test.
Lineage becomes end-to-end — dbt exposures are the standard way to surface "this model is consumed by this Hightouch sync" inside the dbt docs and the data catalog.
On-call gets a new pager — the day a sync fails silently is the day the data team learns operational analytics needs operational ownership. One person owns sync health, full stop.
Cost — the new responsibilities are mostly process; the dbt features (contracts, exposures, tags) ship out of the box. Marginal infrastructure cost is the reverse ETL vendor subscription itself.

ETL
Topic — etl
ETL pipeline problems (data engineering)

Practice →

2. The reverse ETL data model — models, audiences, syncs

Every reverse ETL platform organises around four nouns: model, audience, sync, destination — learn them once and every vendor feels the same

The mental model in one line: a model is a warehouse query that produces one row per entity; an audience is a filtered subset of a model; a sync is a mapping of model rows into a destination; a destination is the SaaS tool. Once you learn this four-noun vocabulary, every vendor UI collapses to the same shape and the differences become mostly cosmetic.

The four-noun glossary.

Model. A SQL query (or dbt model reference) that returns rows of a single entity — one_row_per_user, one_row_per_account, one_row_per_subscription. The model has a primary key column and a set of attribute columns.
Audience. A filter expression layered on top of a model — WHERE plan = 'pro' AND last_seen_at < CURRENT_DATE - INTERVAL '30 days'. Audiences are reusable across syncs and across destinations.
Sync. The full specification: which model (or audience), which destination, which field mappings, which sync mode, which schedule. A sync is the deployable unit.
Destination. The SaaS tool credentials + the destination object (Salesforce Contact, HubSpot Company, Intercom User, Marketo Lead, Facebook Custom Audience).

Sync modes you will encounter.

Insert. New rows are inserted into the destination; existing rows are untouched. Used for append-only destinations like logging or analytics events.
Update. Existing rows are updated; new rows are not inserted. Used when the destination owns identity creation (e.g. only update Salesforce contacts that already exist via lead capture).
Upsert. Insert new rows, update existing rows. The most common mode for customer attribute syncs.
Mirror. Make the destination match the model exactly — insert new, update changed, delete rows no longer in the model. The most powerful and the most dangerous; usually scoped to audiences (e.g. "the at-risk audience").
Delete only. Remove rows from the destination based on a "tombstone" model. Often used for GDPR delete propagation.

Identity resolution at the sync boundary.

External ID matching. The most common pattern: the warehouse primary key (salesforce_contact_id, hubspot_vid) is the same as the destination's primary key. The sync upserts on that key.
Email / phone matching. When the warehouse and the destination both store contact PII, syncs can match on email or phone. Brittle to changes (a user's email change creates a "new" record) but works for greenfield setups.
Custom external_id field. Hightouch and Census both support designating a custom external ID field in the destination (e.g. Marketo's external_id_c). The sync writes the warehouse PK there once, then matches on it forever.
Composite key matching. Some destinations (Salesforce, Marketo) support compound external IDs (e.g. account_id + region). Rarely used; useful when the same person lives in multiple tenants.

Idempotency — the contract that saves the team.

Stable primary key on every model. If the warehouse PK can change, the sync will double-write or fail to dedupe — every reverse ETL platform assumes the model PK is stable across runs.
Idempotent upserts. A retry on the same row must produce the same destination state. Most SaaS APIs support id based upsert; some require a "create-or-update" two-step.
Diff-only by default. Sync only the rows that changed since the last successful run. Saves API quota, reduces destination clutter, simplifies observability ("zero diffs is a healthy sync, not a broken one").

Change detection — three strategies.

Full refresh. Read the entire model every run, ship every row. Simple, expensive, almost never the right answer above 100k rows.
Diff-only (snapshot). Store a hash of every (PK, attribute) tuple on each successful run. On the next run, compare hashes and only ship the diffs.
CDC mirror. Subscribe to the warehouse's change-data-capture stream (Snowflake streams, BigQuery change streams, Databricks CDC) and apply diffs incrementally. The lowest-latency option; vendor support varies.

Worked example — defining a model with a stable PK and clean attributes

Detailed explanation. A reverse ETL model is not a fact table. It is a one-row-per-entity row set with attributes the destination cares about. The biggest mistake newcomers make is reusing an analytics fact table as the model — fact tables have multiple rows per entity, and the sync will explode or drop most of them.

Question. Given a fact_orders table and a dim_customers table, write the right dbt model for a "current customer state" reverse ETL sync into Salesforce.

Input — fact_orders.

order_id	customer_id	amount	order_date
1	C1	50	2026-06-01
2	C1	30	2026-06-10
3	C2	200	2026-05-20

Input — dim_customers.

customer_id	name	salesforce_contact_id
C1	Alice	003A1
C2	Bob	003A2

Code.

-- WRONG — multiple rows per customer; will fail upsert.
SELECT
    c.salesforce_contact_id,
    o.amount,
    o.order_date
FROM dim_customers c
JOIN fact_orders o ON o.customer_id = c.customer_id;

-- RIGHT — one row per customer with aggregated attributes.
-- File: models/marts/reverse_etl_customer_state.sql
SELECT
    c.salesforce_contact_id,
    c.name,
    COUNT(o.order_id)                              AS lifetime_orders,
    COALESCE(SUM(o.amount), 0)                     AS lifetime_revenue,
    MAX(o.order_date)                              AS last_order_at,
    COUNT(o.order_id) FILTER
        (WHERE o.order_date >= CURRENT_DATE - 30)  AS orders_last_30d
FROM dim_customers c
LEFT JOIN fact_orders o ON o.customer_id = c.customer_id
GROUP BY c.salesforce_contact_id, c.name;

Step-by-step explanation.

The wrong model emits two rows for C1 (one per order). The Hightouch sync sees two rows with the same salesforce_contact_id, fails the "unique PK" assertion, and either rejects the sync or upserts the last row arbitrarily.
The right model wraps fact_orders in a GROUP BY on customer_id, collapsing every customer to one row. Attributes are aggregated: COUNT for orders, SUM for revenue, MAX for last order date.
LEFT JOIN preserves customers with zero orders. COALESCE(SUM(...), 0) turns the NULL sum into a clean 0 for downstream Salesforce automations.
COUNT(...) FILTER (WHERE ...) produces the "last 30 days" attribute without a separate subquery. Postgres / Snowflake / BigQuery support FILTER; SQL Server uses COUNT(CASE WHEN ... THEN 1 END).

Output (the reverse ETL model).

salesforce_contact_id	name	lifetime_orders	lifetime_revenue	last_order_at	orders_last_30d
003A1	Alice	2	80	2026-06-10	2
003A2	Bob	1	200	2026-05-20	0

Rule of thumb. A reverse ETL model is SELECT ... FROM ... GROUP BY entity_id plus joins. If the model emits more than one row per entity, the sync is wrong. Add a dbt-unique test on the PK column so the next CI run catches it.

Worked example — defining an audience from a model

Detailed explanation. Audiences are reusable filtered subsets of a model. A typical pattern: one underlying marts.reverse_etl_customer_state model, multiple audiences ("at-risk", "high-value", "trial-expiring"), each subscribed to a different destination.

Question. Define three audiences on top of the customer state model: at-risk (churn_risk > 0.7), high-value (lifetime_revenue > 5000), and active-trial (plan = 'trial' AND days_remaining < 7). Show how each maps to a different destination.

Input — marts.reverse_etl_customer_state.

salesforce_contact_id	name	plan	lifetime_revenue	churn_risk	trial_ends_at
003A1	Alice	pro	8000	0.05	NULL
003A2	Bob	trial	0	NULL	2026-06-18
003A3	Cara	pro	1200	0.82	NULL
003A4	Dan	trial	0	NULL	2026-06-30

Code.

-- Audience: at_risk
SELECT * FROM {{ ref('marts.reverse_etl_customer_state') }}
WHERE churn_risk > 0.7;

-- Audience: high_value
SELECT * FROM {{ ref('marts.reverse_etl_customer_state') }}
WHERE lifetime_revenue > 5000;

-- Audience: active_trial
SELECT * FROM {{ ref('marts.reverse_etl_customer_state') }}
WHERE plan = 'trial'
  AND trial_ends_at IS NOT NULL
  AND trial_ends_at - CURRENT_DATE BETWEEN 0 AND 7;

# Three syncs, one model, three destinations.
- name: at_risk_to_intercom
  audience: at_risk
  destination: intercom
  sync_mode: mirror
  mappings:
    - source: churn_risk         -> Company.churn_risk_attr
    - source: lifetime_revenue   -> Company.ltv_attr

- name: high_value_to_facebook_ads
  audience: high_value
  destination: facebook_ads
  sync_mode: mirror
  mappings:
    - source: email              -> custom_audience.email_hash

- name: active_trial_to_iterable
  audience: active_trial
  destination: iterable
  sync_mode: mirror
  mappings:
    - source: trial_ends_at      -> User.trial_end_date
    - source: name               -> User.first_name

Step-by-step explanation.

The single underlying model marts.reverse_etl_customer_state is the source of truth. Every audience is a filter on top of it.
Audience at_risk mirrors to Intercom for CS alerting. The sync ships only the matching subset and removes the tag when a customer drops out of the audience (mirror mode).
Audience high_value mirrors hashed emails to a Facebook custom audience. Add/remove behaviour follows audience membership automatically.
Audience active_trial syncs to Iterable for an automated email sequence. The mirror mode adds users when they enter the trial window and removes them when the trial ends.
Each sync inherits the same model contract — change the column, every audience and sync notices on the next run.

Output.

Audience	Members	Destination	Effect
at_risk	003A3 (Cara)	Intercom	tagged as at_risk
high_value	003A1 (Alice)	Facebook	added to custom audience
active_trial	003A2 (Bob)	Iterable	trial-end sequence triggered

Rule of thumb. Build one model per entity, many audiences per model, one or more syncs per audience. The fan-out pattern (1 model → N audiences → M syncs) keeps the definition of an entity DRY and lets each downstream team pick the slice they care about.

Worked example — change detection: snapshot diff vs full refresh

Detailed explanation. Diff-only syncs are the default in every modern reverse ETL platform. They store a hash (or row checksum) per primary key after each successful run; on the next run they compare the new model output against the stored snapshot and emit only the changed rows. Full refresh is sometimes correct but very expensive.

Question. Given a 1M-row customer state model where 0.2% of rows change between runs, compare full-refresh API cost (every run ships every row) with diff-only (only changed rows shipped). Use a destination with a 200-row-per-API-call batch limit.

Input — assumptions.

Variable	Value
Total model rows	1,000,000
Rows changed per run	2,000 (0.2%)
Destination batch size	200 rows
Syncs per day	24
Destination API call cost	$0.001

Code.

Full refresh per run:
    api_calls    = ceil(1_000_000 / 200) = 5_000
    runs_per_day = 24
    daily_cost   = 5_000 * 24 * $0.001 = $120

Diff-only per run:
    api_calls    = ceil(2_000 / 200) = 10
    runs_per_day = 24
    daily_cost   = 10 * 24 * $0.001 = $0.24

Cost ratio: 500x cheaper with diff-only.
Time ratio: same — typical API latency dominated by call count, not payload size.

Step-by-step explanation.

Full refresh ships every row on every run. With 1M rows and 200/batch, the platform issues 5,000 API calls per run. 24 runs/day is 120,000 calls/day.
Diff-only ships only the 0.2% changed rows. 2,000 rows / 200 per batch = 10 API calls per run. 24 runs/day is 240 calls/day.
The math is independent of vendor — every reverse ETL platform that supports diff-only will produce this savings on a typical attribute-update workload.
Diff-only does require the platform to maintain the previous snapshot. The snapshot is typically stored in the reverse ETL platform's own metadata DB (Hightouch) or as a hidden audit table in the source warehouse (Census's "tracking table" pattern).

Output.

Strategy	API calls / day	Cost / day	Quota risk
Full refresh	120,000	$120	high (Salesforce 15k cap)
Diff-only	240	$0.24	very low

Rule of thumb. Default to diff-only on every sync. Use full refresh only for "catch-up after a destination outage" or for small reference tables under ~10k rows. The 100–500× API quota savings are not optional at scale — Salesforce will hard-stop you at 15k API calls per 24h on the standard plan.

Reverse ETL interview question on idempotency

A senior interviewer often probes: "Your nightly sync runs, fails halfway through with a network blip, and reruns automatically. How do you guarantee the destination ends up in the same state it would have been if the sync had succeeded the first time?"

Solution Using the idempotent upsert contract

Idempotency is guaranteed if and only if:

1. Every model row has a stable primary key.
   - The PK is the natural identity (salesforce_contact_id),
     not a row number or a hash that changes between runs.
   - dbt test: unique + not_null on the PK column.

2. The sync mode is upsert (not insert) on a destination-side
   external ID field.
   - Salesforce: Upsert /sobjects/Contact/extId/{externalId}
   - HubSpot: Upsert /contacts/v1/contact/createOrUpdate/email/{email}
   - Marketo: leads/createOrUpdate with lookupField=externalId

3. The destination accepts a duplicate row as a no-op when
   nothing has actually changed.
   - Hightouch: built-in "skip unchanged rows" toggle.
   - Census: built-in idempotency cache.
   - RudderStack: ETag / If-Match conditional updates.

4. Retries on transient errors (5xx, network timeout) are
   safe because step 2 guarantees the second call lands the
   same destination state as the first.

5. Permanent errors (4xx) go to a dead-letter queue for
   manual inspection, NOT into the auto-retry loop.

Step-by-step trace.

Run	Outcome	Destination state	Idempotent?
Run 1 (initial)	success	1000 rows in Salesforce	n/a
Run 2 (no diff)	success	1000 rows unchanged	yes — zero API calls
Run 3 (1 row change)	success	1000 rows, 1 updated	yes — 1 API call
Run 4 (mid-run network blip)	partial fail at row 500	500 of 999 deltas applied	next run resumes
Run 4 retry	success	all deltas applied	yes — final state matches success-on-first-try

The fourth row shows the key behaviour: a half-applied sync is safe because each row's upsert is idempotent. The retry picks up the unfinished deltas without re-applying the already-applied ones.

Output:

Property	Behaviour with idempotency contract
Network blip	safe — retry resumes
Same model, two runs back-to-back	second run is a no-op
Schema change downstream	sync fails loudly, no half-update
Concurrent runs	platform locks the sync to one instance
Duplicate row in model	dbt test fails before sync starts

Why this works — concept by concept:

Stable PK — the primary key is the bridge between warehouse identity and destination identity. The whole upsert mechanism depends on it being stable across runs.
External ID upsert — every modern SaaS API offers an upsert primitive keyed on a custom external ID. Use it. Two-step "search-then-create-or-update" patterns are error-prone and not idempotent under concurrency.
Diff-only + skip-unchanged — short-circuits the destination call entirely when nothing has changed. A healthy sync run can legitimately make zero API calls.
Dead-letter queue — permanent errors (validation failure, missing required field) are not retried in a tight loop; they go to an inspect-and-fix queue. The retry loop is only for transient errors.
Concurrent-run lock — every reverse ETL platform single-instances each sync. Two parallel runs of the same sync would race on the diff snapshot and corrupt the next-run baseline.
Cost — idempotency is essentially free once the contract is in place. The cost is the up-front discipline of designing models with stable PKs and configuring destination external IDs.

Data
Topic — dimensional-modeling
Dimensional modelling problems (data engineering)

Practice →

3. Hightouch vs Census vs RudderStack — vendor comparison

Each vendor optimises for a different team shape — pick by who owns syncs and how dbt-native your stack is

The mental model in one line: Hightouch is the audience-builder-first managed platform, Census is the dbt-native data-team-first managed platform, RudderStack is the open-source CDP + reverse ETL combined platform with a self-hostable option. Once you map team shape and stack constraints to vendor identity, the choice becomes obvious — and obvious choices are easier to defend in a procurement meeting.

The vendor matrix in one table.

Capability	Hightouch	Census	RudderStack
Destinations (2026)	200+	180+	200+ (events + reverse ETL)
dbt integration	strong (model picker, exposures)	strongest (dbt exposures native, "data-team first")	adequate
Audience builder	first-class visual UI	SQL-first, basic UI builder	basic
Sequences / journeys	yes (Hightouch sequences)	yes (Census audiences with priority)	partial
Identity resolution	strong (configurable matching)	strong (entity model)	event-stream-first
Self-hosted option	no (managed only)	no (managed only)	yes (RudderStack OSS + BYOC)
Combined CDP + reverse ETL	no	no	yes (event stream + reverse ETL)
Observability	strong (per-row, per-sync)	strong (sync alerts)	adequate
Pricing model	per-destination + MTU	per-row synced	per-MTU + events

Hightouch — audience-builder first, GTM-team first.

Strengths. Best-in-class audience builder UI (drag-and-drop filters, custom calculations); broadest destination catalogue; "Hightouch sequences" let marketing build journeys without leaving the tool; deep observability with row-level error inspection.
Best fit. Teams where the audience definitions live half in SQL and half in marketing's head; companies with 5+ destinations across CRM + marketing + ads.
Trade-offs. Managed-only (no self-host); MTU-based pricing surprises mid-market companies as their user count grows; Hightouch's UI-first audience editor can drift from the dbt definition of an entity if not policed.

Census — data-team first, dbt-native.

Strengths. Tightest dbt integration of the three — Census reads dbt_project.yml, recognises exposures, and surfaces sync metadata back into the dbt docs; "entity" model is a first-class concept; sync alerting is mature.
Best fit. Data teams that already live in dbt and want the warehouse-to-SaaS contract owned by analytics engineers, not marketing ops.
Trade-offs. Audience-builder UI is intentionally minimal (SQL is the way); fewer "GTM goodies" like multi-channel journeys; managed-only; per-row pricing means batch refreshes can sting.

RudderStack — open-source CDP + reverse ETL combined.

Strengths. Open-source under AGPLv3 with a managed plan; combines event streaming (Segment-style) with reverse ETL in one tool; self-hostable for BYOC / on-prem / compliance-driven shops; the only one of the three that can serve sub-30-second event reverse ETL.
Best fit. Companies that need both event collection and reverse ETL but want to avoid SaaS sprawl; compliance / BYOC use cases; engineering-heavy teams comfortable running infra.
Trade-offs. UI is less polished than Hightouch / Census; destination catalogue runs slightly behind on long-tail SaaS tools; the self-hosted operational cost is real (operate Postgres, Kubernetes, observability).

Pricing dimensions to model before procurement.

MTU (Monthly Tracked Users). Most platforms charge per unique entity synced per month. The metric grows roughly with total customer base.
Per-row synced. Census's primary metric. Drives a "diff-only is required" discipline because full refresh becomes ruinously expensive.
Per-destination. Hightouch's standard plans cap the number of destinations on lower tiers. Multi-channel companies feel this fast.
Per-seat. Both Hightouch and Census charge per audience-builder seat above a baseline.
Events (RudderStack). Event-stream pricing is per event, not per unique user. Plan for both axes.

Self-hosted (RudderStack OSS) vs managed trade-off.

Self-hosted wins for. BYOC compliance, data-residency, "all data must stay in our VPC," low cost at very large MTU counts (>1M).
Managed wins for. Speed (live in a day vs a quarter), no infra ops burden, faster destination roll-outs, no upgrade cycles.
Hybrid pattern. Many shops run RudderStack OSS for event collection (zero per-event vendor cost) and Hightouch managed for reverse ETL (fastest catalogue + audience UI).

Worked example — picking Hightouch when GTM owns audiences

Detailed explanation. A B2B SaaS company has a 6-person revenue ops team that owns Salesforce, HubSpot, Marketo, Outreach, and a half-dozen ad accounts. They want to build "buying-committee" audiences without filing a Jira to data each time. The data team owns the underlying dbt model; revenue ops owns the audience layer on top.

Question. Given the company profile (GTM-heavy, 5+ destinations, audience-builder UI matters), justify Hightouch as the right pick. List the decisive feature differences vs Census and RudderStack.

Input — the company profile.

Property	Value
Audience owners	Revenue ops (non-SQL)
Destinations	Salesforce, HubSpot, Marketo, Outreach, FB Ads, LinkedIn Ads
Warehouse	Snowflake + dbt
Sync latency	30 minutes acceptable
Self-host requirement	none

Code.

Decision matrix:

| Need                       | Hightouch | Census | RudderStack |
|----------------------------|-----------|--------|-------------|
| Drag-drop audience builder | strong    | basic  | basic       |
| 6+ destinations            | yes       | yes    | yes         |
| dbt exposure surfacing     | yes       | best   | adequate    |
| Multi-channel sequences    | yes       | partial| partial     |
| No-SQL revenue ops users   | strong    | weak   | weak        |

Decision: Hightouch wins on (1) audience builder, (4) sequences,
(5) non-SQL audience editors. Census's dbt-first stance is a real
strength but the GTM team owns audiences in this org.

Step-by-step explanation.

The team's bottleneck is "GTM ops cannot self-serve audiences." Hightouch's audience builder is the only one of the three optimised for that exact persona.
Census's strength (dbt-native) does not help when the audience layer is owned outside the data team. The model is still in dbt; the audience-on-top-of-model is what's UI-driven.
RudderStack's event-stream story is not relevant — this team is not building real-time personalisation, just attribute syncs at 30-minute cadence.
The decisive feature is the audience builder UI, with Hightouch sequences as a bonus for multi-step marketing journeys.

Output.

Decision	Hightouch
Why	audience builder + sequences + destination catalogue
Estimated MTU cost	$$ (mid-market plan)
Implementation timeline	4 weeks to first sync

Rule of thumb. Hightouch wins when GTM owns the audience layer and non-SQL editors need to ship audiences without filing tickets. Census wins when the data team owns the audience layer and dbt is the single source of truth. RudderStack wins when you need both CDP event collection and reverse ETL or you must self-host.

Worked example — picking Census when dbt is the source of truth

Detailed explanation. A fintech with strict change-management has a small analytics engineering team that defines every metric, every entity, and every audience in dbt. Marketing ops "subscribes" to dbt models via tickets. The team wants the sync layer to inherit dbt's contract testing, exposures, and lineage natively.

Question. Given the company profile (dbt-first, analytics engineering owns audiences, strict change management), justify Census over Hightouch. List the decisive dbt integration features.

Input — the company profile.

Property	Value
Audience owners	Analytics engineering (SQL-fluent)
Source of truth	dbt models, branch-protected
Destinations	Salesforce, Iterable, Customer.io
Sync latency	1 hour acceptable
Compliance	strict — every change reviewed

Code.

Census dbt-native features that decided it:

1. dbt project sync — Census reads dbt_project.yml directly.
   Models appear in Census with the same name as in dbt.

2. dbt exposures — every Census sync is automatically surfaced
   as a dbt exposure. Lineage in dbt docs shows the destination.

3. Git-backed sync definitions — sync YAML lives in the dbt
   repo, change-managed via PR.

4. dbt tests propagate — failing dbt tests block the sync.
   Census never ships a failing-test row to a destination.

5. Entity model — Census's "entity" concept is the equivalent
   of a dbt model with documented PK + columns. Discoverable
   across the team.

Step-by-step explanation.

The data team's discipline is "everything ships via PR." Census's git-backed sync definitions extend that discipline to the reverse ETL layer.
Hightouch supports a Terraform provider for sync-as-code, but the UI-first culture pulls non-engineers off the git workflow. Census's SQL-first culture matches the team.
dbt exposures inside Census are decisive — every destination becomes a known consumer in the lineage graph. Census surfaces "this sync depends on this model" automatically.
Failing dbt tests blocking the sync is the killer feature for compliance — it means a regression in the model never silently corrupts a downstream SaaS field.

Output.

Decision	Census
Why	dbt-native + git-backed syncs + exposures + test-gating
Estimated cost	$$ (per-row pricing acceptable at this volume)
Implementation timeline	6 weeks to production sync

Rule of thumb. Census wins when the analytics engineering team owns the audience layer and "everything ships via PR" is a non-negotiable. The dbt integration is real, not cosmetic — it changes how the team operates day to day.

Worked example — picking RudderStack OSS for BYOC compliance

Detailed explanation. A healthcare SaaS must keep PII inside its own VPC. Sending raw email addresses through a multi-tenant SaaS reverse ETL platform is a compliance blocker. RudderStack OSS runs inside the customer VPC, never touches the vendor's infrastructure, and combines event collection (replacing Segment) with reverse ETL in one tool.

Question. Given the company profile (PII must stay in VPC, single tool preferred for events + syncs), justify RudderStack OSS over the managed options.

Input — the company profile.

Property	Value
Compliance	PII must stay in customer VPC
Existing event tool	considering Segment replacement
Destinations	Salesforce Health Cloud, HubSpot, internal API
Sync latency	5 minutes for high-priority
Team	engineering-heavy, comfortable running infra

Code.

Why RudderStack OSS wins on this profile:

1. Self-hosted in customer VPC.
   - No PII leaves the customer's cloud account.
   - Audit trail end-to-end within customer-owned storage.

2. Combined event stream + reverse ETL.
   - Single tool covers Segment-like event collection AND
     Hightouch-like warehouse reverse ETL.
   - One destinations catalogue, one UI, one set of credentials.

3. Event-stream reverse ETL.
   - Sub-30-second latency on high-priority warehouse changes
     via the event-stream path (not the batch path).

4. AGPLv3 source-available.
   - Customer can patch, audit, and extend.
   - No vendor lock-in for compliance-critical features.

Trade-offs accepted:
- Operate Postgres, Redis, K8s yourself.
- Destination catalogue runs slightly behind Hightouch on
  long-tail tools.
- UI is less polished — engineers, not marketers, configure syncs.

Step-by-step explanation.

The PII-in-VPC requirement removes Hightouch and Census from contention immediately — both are managed-only.
The combined event-stream + reverse ETL story removes Segment from the picture and consolidates spend.
RudderStack OSS's event-stream reverse ETL path is the only sub-30-second option in this comparison — relevant for the "high-priority sync" use case.
The trade-off is operational burden. The team must own the Postgres metadata DB, Redis broker, and Kubernetes orchestration. An engineering-heavy org accepts this.

Output.

Decision	RudderStack OSS
Why	self-hosted compliance + combined CDP + sub-30s reverse ETL
Estimated cost	infrastructure + 0.5 SRE FTE
Implementation timeline	8 weeks to production

Rule of thumb. RudderStack wins on three triggers: BYOC compliance, single-tool consolidation of CDP + reverse ETL, or sub-30-second latency requirements via event-stream reverse ETL. If none of those triggers fires, prefer Hightouch or Census for the operational simplicity of managed.

Reverse ETL interview question on the buy-vs-build decision

A senior interviewer often frames it as: "Your CTO is asking whether we can just build reverse ETL in-house with Airflow + Python + the destination SDKs. Walk me through the buy-vs-build decision."

Solution Using the operational-burden lens

The build-it-yourself stack:

1. Airflow / Dagster orchestration.
2. Custom Python writers for each destination API.
3. Snapshot diff engine (you build it).
4. Queue + worker pool with retry semantics (you build it).
5. Dead-letter queue + inspection UI (you build it).
6. Per-row error logging (you build it).
7. Schema-change detection (you build it).
8. Audit log + lineage (you build it).
9. Audience builder UI for non-engineers (... you build it).
10. PII tagging + governance UI (you build it).

The buy stack:

1. Hightouch / Census / RudderStack subscription.
2. Sync configuration (a week of work).

The break-even calculation:

- Year 1 build cost: 2 senior engineers × 6 months = ~$300k.
- Year 1 buy cost:   ~$30k–$80k subscription, depending on MTU.
- Year 2 build cost: 1 engineer × full year maintenance = ~$200k.
- Year 2 buy cost:   ~$50k–$120k subscription.

Buy wins decisively unless:
- You have a destination not on the vendor catalogue (rare).
- You have a sub-second latency requirement (use a feature store).
- You have a compliance constraint requiring on-prem (use OSS).

Step-by-step trace.

Component	Buy time	Build time
Destination connectors	1 day per destination (config)	2 weeks per destination (code + tests)
Diff engine	included	4 weeks
Queue + retry	included	6 weeks
Dead-letter inspection	included	2 weeks
Audience builder UI	included	12+ weeks (and your data team has to maintain it)
Schema-change detection	included	4 weeks

The "build everything" path lands at 6–9 months for a v1 covering 5 destinations with no UI. The "buy" path lands at 4–6 weeks for the same scope plus an audience UI.

Output:

Year	Build cost	Buy cost
1	~$300k	~$50k
2	~$200k	~$80k
3	~$200k	~$100k

Why this works — concept by concept:

Connector breadth — vendors maintain hundreds of destination integrations as their full-time job. A 2-engineer team building from scratch will cover 5–10 destinations at best in year 1.
Diff engine is the moat — every reverse ETL platform's secret sauce is the diff/snapshot/incremental detection logic. Building a reliable one is a 6-month research project, not a weekend hack.
Audience UI — the moment a non-engineer needs to ship an audience, you need a UI. Building that internally is a years-long product investment that has nothing to do with your company's actual product.
Observability — per-row error tracking, dead-letter queues, sync success ring charts — all included in the vendor stack. Building them stalls your data team for months.
Compliance escape hatch — RudderStack OSS exists precisely for the rare cases where vendor managed cannot work. Use OSS, not in-house build.
Cost — over a 3-year window the buy path is 3–5× cheaper and ships in 1/10 the time. The only counter-arguments are scale (>10M MTU and you renegotiate hard) or compliance (and OSS solves that).

ETL
Topic — api-integration
API integration problems (data engineering)

Practice →

4. Sync architecture — incremental detection, queues, rate limits

A sync is a diff engine plus a queue plus a worker pool plus a rate-limited destination API — every reverse ETL platform implements the same four-stage pipeline

The mental model in one line: the warehouse query produces rows, the diff engine classifies each row as insert/update/delete vs the previous snapshot, the queue absorbs back-pressure, and the worker pool drains the queue into the destination API while respecting per-destination rate limits. Once you can draw the four stages on a whiteboard, every "why is my sync slow / failing / partial?" question becomes a probe of which stage is the bottleneck.

Stage 1 — warehouse query and snapshot detection.

Query. The model SQL (or audience-filtered model SQL) runs against the warehouse. Result is materialised either into a temp table or streamed.
Snapshot store. The previous run's (pk, hash(attributes)) set lives somewhere — a hidden table in the warehouse, a Postgres metadata DB in the vendor's infra, or a CDC stream offset.
Diff classification. For each current row: if PK absent in snapshot → INSERT; if PK present and hash differs → UPDATE; for each snapshot PK absent in current → DELETE (or "tombstone").

Stage 2 — staging / queue.

Per-sync queue. Each sync gets its own queue, single-instanced. No parallel runs of the same sync.
Back-pressure absorption. When the destination's API is slow, the queue grows; workers pull at the destination's pace, not the warehouse's pace.
Persistence. Queues persist to disk so a vendor restart does not lose in-flight rows.

Stage 3 — worker pool.

Worker concurrency. Configured per destination; usually 1–8 parallel workers per sync.
Batch packing. Workers pack queue rows into destination-specific batches (Salesforce: 200/batch, HubSpot: 100/batch, Marketo: 300/batch).
Token-bucket rate limiter. Each worker checks the destination's quota before issuing the call.

Stage 4 — destination API.

Auth. OAuth, API key, service account — refreshed automatically by the platform.
Rate limit response. 429 (Too Many Requests) triggers exponential backoff and a slowdown of the worker pool.
Per-row error response. 4xx errors on specific rows are recorded as row-level failures, surfaced in the sync log, and either retried (transient) or dead-lettered (permanent).

Destination rate limits in the wild (2026 baselines).

Destination	Limit	Notes
Salesforce	15,000 / 24h (standard)	per-org, all APIs share
HubSpot	100 / 10s + 250k / day	per-portal
Marketo	100 / 20s + 50k / day	per-instance
Intercom	1,000 / minute	per-app
Iterable	4 / second list endpoints	varies by endpoint
Facebook Custom Audience	200,000 users / API call	batched mode
Slack	1 / second per webhook	basic tier

Retry semantics.

Transient (5xx, 429, network timeout) — retry with exponential backoff. Typical: 1s → 2s → 4s → 8s → 16s → 32s, then dead-letter.
Permanent (4xx with validation error) — log and dead-letter immediately. Retrying will not help.
Auth (401, token expired) — refresh the token and retry once, then alert.
Quota exhausted (429 with daily-cap header) — pause the sync until the quota window resets; alert if the window is >12 hours.

Latency tiers.

Hourly batches. Default for most syncs. 5–60 minutes end-to-end.
Sub-minute batches. Census + small models. 30 seconds–5 minutes.
CDC mirror. Continuous; reflects warehouse changes in seconds.
Event-stream reverse ETL. RudderStack's path; reflects in 1–30 seconds.

Worked example — the diff engine in pseudo-code

Detailed explanation. The diff engine is the heart of every reverse ETL platform. It compares the current model row set against the previous snapshot and emits a stream of insert/update/delete events. Knowing the shape of this code helps debug "why did my sync ship row X?" questions.

Question. Write a pseudo-code sketch of a diff engine that takes (current_rows, previous_snapshot) and emits classified events. Explain how it handles deletes.

Input.

previous_snapshot:
  C1 -> hash("Alice|pro|0.05")
  C2 -> hash("Bob|trial|null")
  C3 -> hash("Cara|pro|0.40")

current_rows:
  C1 -> ("Alice", "pro", 0.05)         # unchanged
  C2 -> ("Bob",   "pro", 0.10)         # changed (trial -> pro)
  C4 -> ("Dan",   "trial", null)       # new
  # C3 missing -> deleted

Code.

def diff_engine(current_rows, previous_snapshot):
    """Yield classified change events."""
    current_keys  = set(current_rows.keys())
    previous_keys = set(previous_snapshot.keys())

    # INSERTs — PKs in current but not previous.
    for pk in current_keys - previous_keys:
        yield ("INSERT", pk, current_rows[pk])

    # UPDATEs — PKs in both, hash differs.
    for pk in current_keys & previous_keys:
        new_hash = row_hash(current_rows[pk])
        if new_hash != previous_snapshot[pk]:
            yield ("UPDATE", pk, current_rows[pk])
        # else: unchanged, emit nothing (this is the big saving).

    # DELETEs — PKs in previous but not current.
    # Only if sync_mode == "mirror"; otherwise skip deletes.
    for pk in previous_keys - current_keys:
        yield ("DELETE", pk, None)

    # Persist new snapshot for next run.
    new_snapshot = {pk: row_hash(row) for pk, row in current_rows.items()}
    save_snapshot(new_snapshot)

Step-by-step explanation.

The set difference current - previous yields rows present this run but not last run — INSERTs.
The set intersection plus hash comparison yields rows present in both runs whose attributes changed — UPDATEs. Unchanged rows are skipped silently (zero API calls).
The set difference previous - current yields rows present last run but absent this run — DELETEs. Only emitted in mirror sync mode; upsert mode ignores them.
The new snapshot is persisted at the end. If the run crashes before this point, the next run sees the same previous snapshot and re-classifies the same diffs (idempotent recovery).
The row hash function is typically MD5 / xxHash over the JSON serialisation of attributes in a canonical column order. Hash collisions are theoretically possible; in practice the rate is negligible at billion-row scale.

Output.

Event	PK	Attributes
INSERT	C4	(Dan, trial, NULL)
UPDATE	C2	(Bob, pro, 0.10)
DELETE	C3	—

Rule of thumb. Always store the previous snapshot durably (warehouse table, Postgres, or S3). A lost snapshot triggers a "full diff against empty," which classifies every row as INSERT and floods the destination — the canonical "first run after vendor restart was a disaster" outage.

Worked example — the rate limiter and the 429 backoff loop

Detailed explanation. Every destination has rate limits. The worker pool must respect them or risk getting the entire integration locked. The token-bucket + exponential-backoff pattern is the universal solution.

Question. Sketch a worker loop that drains a queue of upsert events into a Salesforce-like API with a 15k/24h limit, handles 429 responses, and emits to dead-letter on permanent errors.

Input.

Queue items:
  - upsert C1 with payload P1
  - upsert C2 with payload P2
  - upsert C3 with payload P3 (will return 400 — invalid email)
  - upsert C4 with payload P4

Destination state:
  - quota_remaining = 14_998
  - quota_resets_at = 24h from now

Code.

def worker(queue, destination, rate_limiter, dead_letter):
    while True:
        event = queue.pop()
        if event is None:
            break

        # 1. Respect the destination's rate limit.
        rate_limiter.acquire(destination)

        # 2. Make the API call.
        backoff = 1
        for attempt in range(7):
            try:
                response = destination.upsert(event.pk, event.payload)
                if response.status == 200:
                    break
                if response.status == 429:
                    # Rate limited — exponential backoff.
                    sleep(backoff)
                    backoff = min(backoff * 2, 60)
                    continue
                if 400 <= response.status < 500:
                    # Permanent error — dead letter.
                    dead_letter.push(event, response.body)
                    break
                if 500 <= response.status:
                    # Transient server error — retry.
                    sleep(backoff)
                    backoff = min(backoff * 2, 60)
                    continue
            except NetworkTimeout:
                sleep(backoff)
                backoff = min(backoff * 2, 60)
                continue
        else:
            # Out of attempts — dead letter.
            dead_letter.push(event, "max_retries_exceeded")

Step-by-step explanation.

rate_limiter.acquire blocks the worker until the token bucket has a slot. Implementation is typically a Redis script that decrements a per-destination counter and refills it at the destination's rate.
The retry loop runs up to 7 attempts. On 429, the worker sleeps and retries (backoff 1s → 2s → 4s → ... capped at 60s).
On 5xx transient server errors, the worker also retries — server-side issues are usually self-healing within seconds.
On 4xx permanent errors (validation failure, malformed payload, missing required field), the worker stops retrying and pushes the event to the dead-letter queue for human inspection.
Network timeouts (no response) are treated as transient — the worker retries with backoff.
If all 7 attempts fail, the event is dead-lettered with max_retries_exceeded so on-call has visibility.

Output (events that reach the destination vs dead-letter).

Event	Destination state	Dead-letter?
C1	upserted	no
C2	upserted	no
C3	rejected (400 invalid email)	yes
C4	upserted	no

Rule of thumb. The retry loop should always distinguish transient (4 categories: 429, 5xx, timeout, auth-refresh) from permanent (4xx). Mixing them either burns rate limits on hopeless retries or silently drops fixable failures.

Worked example — back-pressure from a slow destination

Detailed explanation. When the destination API is slow (or rate-limit-restricted), the queue grows. A well-designed reverse ETL platform absorbs the growth and only fails when the queue passes a configured high-water mark — not every time the destination has a slow minute.

Question. Given a warehouse producing 10k rows/minute and a destination accepting 100 rows/minute, model the queue growth over an hour. Show why a "queue depth" alert is the right SLI and how to use it for early warning.

Input.

Variable	Value
Warehouse output rate	10,000 rows/min
Destination accept rate	100 rows/min
Initial queue depth	0
Alert threshold	50,000 rows

Code.

def queue_growth(warehouse_rate, destination_rate, minutes):
    depth = 0
    growth_per_min = warehouse_rate - destination_rate
    log = []
    for minute in range(1, minutes + 1):
        depth += growth_per_min
        log.append((minute, depth))
    return log

# Compute for one hour:
growth = queue_growth(10_000, 100, minutes=60)
# Alert fires when depth crosses 50_000.
alert_minute = next(m for m, d in growth if d > 50_000)
print(f"Queue depth alert at minute {alert_minute}")
# -> Queue depth alert at minute 6

Step-by-step explanation.

Net growth per minute = warehouse output - destination accept = 10000 - 100 = 9900 rows/min.
After 1 min: 9,900 rows queued. After 5 min: 49,500 queued. After 6 min: 59,400 — crosses the 50k alert threshold.
The alert at minute 6 gives on-call 50 minutes of headroom before the queue passes a typical "platform refuses to enqueue" limit of ~500k rows.
The right remediation depends on the cause: (a) destination is rate-limited — wait for the quota to reset and accept the lag; (b) destination is genuinely broken — pause the sync until the destination is healthy; (c) warehouse is producing duplicates — fix the model.
Without the queue-depth alert the team only learns about the problem when the platform errors out at 500k+ — too late, downstream is already stale by hours.

Output.

Minute	Queue depth	Alert?
1	9,900	no
3	29,700	no
5	49,500	no
6	59,400	yes
60	594,000	platform errors

Rule of thumb. Alert on queue depth, not on sync errors. A sync error is the symptom; queue depth is the leading indicator. Set the alert threshold at 30–50% of the platform's enqueue ceiling to buy on-call time.

Reverse ETL interview question on rate-limit-aware design

A senior interviewer often asks: "Salesforce has a 15k API calls per day quota and our customer state model has 200k rows. How do you design a sync that fits inside the quota?"

Solution Using batching + diff-only + audience filtering

The math first:

  raw rows                            = 200_000
  Salesforce upsert batch size        = 200 rows / call
  full refresh calls                  = 1_000 calls / run
  diff-only on 0.5% changed rows      = 1_000 changed rows
  diff-only batch calls               = ceil(1_000 / 200) = 5 calls / run
  hourly cadence                      = 24 runs / day
  daily API calls                     = 5 * 24 = 120 calls / day

  Headroom under the 15k quota: 124x.

The design:

1. Composite Tooling API batching.
   - Use Salesforce's Composite/sObject Collections API:
     200 records per call vs 1 record per Standard upsert.

2. Diff-only sync mode (no full refresh).
   - Reverse ETL platform stores last-run snapshot.
   - Ship only rows whose attribute hash changed.

3. Audience scoping.
   - Many syncs only need the "active" subset of customers.
   - Filter at the audience layer (plan != 'churned')
     so the diff engine compares smaller sets.

4. Cadence sized to business need.
   - Sales routing: every 30 minutes.
   - Account health: every 6 hours.
   - LTV refresh: every 24 hours.
   - Do not over-spec freshness; quota is finite.

5. Per-sync quota guard.
   - Configure the reverse ETL platform's "max API calls per
     window" knob to a sub-quota share per sync.
   - Hightouch and Census both expose this; RudderStack via config.

Step-by-step trace.

Design choice	Effect on quota
Full refresh	1,000 calls/run × 24 = 24,000/day — over quota
Diff-only	5 calls/run × 24 = 120/day
Audience scoping	reduces diff size further
Per-sync quota guard	prevents any one sync from monopolising quota
Hourly vs 30-min cadence	doubles or halves daily API calls

The combination of (2) and (4) is decisive. Diff-only converts the metric from "rows in the model" to "rows that changed," which on most attribute syncs is 0.1–2% of the model.

Output:

Strategy	Calls / day	Inside 15k quota?
Full refresh hourly	24,000	no
Diff-only hourly	120	yes
Diff-only every 30m	240	yes
Diff-only every 5m	1,440	yes
Full refresh every 5m	288,000	catastrophic

Why this works — concept by concept:

Batched upsert — Salesforce's composite endpoint is the single biggest lever. Going from 1 row per call to 200 rows per call drops the call count by 200×.
Diff-only sync — the second biggest lever. Only ship rows that actually changed. Drops the call count by 50–500× on typical attribute workloads.
Audience filtering — shrinks the model to the rows that matter. Skipping churned customers saves both diff computation and quota.
Cadence sizing — the third lever. Match the sync frequency to the actual business cadence; "fresh every 5 minutes" is rarely needed for a CRM attribute.
Per-sync quota guard — defensive design. Even if one sync misbehaves (e.g. a model bug emits 200k diffs), the guard prevents it from burning the org-wide quota and breaking unrelated syncs.
Cost — the design is essentially free. All the levers are configuration, not code. The cost is the discipline to model the math up-front for each new sync.

ETL
Topic — etl
ETL design problems (data engineering)

Practice →

5. Governance, observability, and failure modes

A sync that has no governance, no observability, and no defined failure modes is not a data product — it is a time bomb

The mental model in one line: governance answers "who can sync what to where"; observability answers "is the sync healthy right now"; failure modes answer "what breaks and how do we know". The discipline that separates a hobbyist sync from a production data product is treating these three pillars as first-class — versioned, owned, and on-call paged.

Governance — five non-negotiables.

Field-level PII tagging. Every column tagged pii=email | phone | address | name | ssn. Tags propagate to the sync layer so destinations can enforce per-tag policy.
Per-destination policy. "Email PII can sync to Marketo; SSN PII cannot sync to anything." Hightouch and Census both support sync-level allow/deny rules.
Audience approval. New audiences > 10k members require analytics-engineering sign-off. Catches "I just synced 200k users to Facebook by accident."
GDPR delete propagation. A user's right-to-delete must reach every destination. The platform must support a "delete pipeline" sync (model = users_to_delete, mode = delete-only, fanned out to every destination).
Audit log. Every sync edit, schedule change, and credential rotation is logged with actor + timestamp.

Observability — six SLIs to track.

Sync success rate. Percent of runs that finished without a top-level error. Target: >99.5%.
Row-error rate. Percent of rows in a successful run that failed (typically destination 4xx validation). Target: <1%.
Freshness lag. Time since last successful run vs the scheduled cadence. Target: <2× cadence.
Queue depth. Pending rows waiting for the worker pool. Leading indicator of destination slowness.
Rejected payload sample. Stratified sample of dead-letter events for human inspection.
Latency p50 / p99. Wall-clock time from model row produced to destination row accepted.

Failure modes — the four most common.

Mapping drift. Warehouse column renamed; destination field still expects the old name; sync silently writes NULL or fails.
Schema drift. Column type changed (INT → BIGINT, VARCHAR(50) → VARCHAR(500)); destination rejects with type-mismatch error.
Row-cap breach. Audience suddenly grows from 5k to 200k members because a filter became overly permissive; destination quota burns out.
Credential expiry. OAuth refresh token expires; sync fails with 401; team finds out hours later when freshness lag alert fires.

Catalog + lineage — surfacing syncs as dbt exposures.

Every sync is a known consumer of one or more dbt models. The standard surface is a dbt exposure:

# models/marts/exposures.yml
exposures:
  - name: salesforce_lead_score_sync
    type: application
    owner:
      name: Analytics Engineering
      email: ae@example.com
    depends_on:
      - ref('reverse_etl_customer_state')
    description: |
      Hightouch sync into Salesforce.Contact.lead_score__c.
      Cadence: every 30 minutes.
      On-call: data-team rotation.

Cost guardrails.

Per-sync row caps. "This sync will never ship more than 50k rows per run; abort if it tries."
Audience size caps. "This audience will never include more than 100k members; alert if it does."
Quota share caps. "This sync will use no more than 30% of the destination's daily API quota."
Frequency caps. "Even if scheduled hourly, no more than 24 runs per day."

Worked example — propagating PII tags from dbt to the sync layer

Detailed explanation. Field-level PII tagging is the foundation of governance. When a column is tagged in dbt, the tag must propagate to every downstream sync so per-destination policy can enforce "this PII can/cannot land here." Census and Hightouch both read dbt meta tags directly.

Question. Tag dim_users.email as pii=email in dbt, configure Census to read the tag, and define a per-destination policy that allows email to sync to Marketo but blocks it from a marketing experimentation tool.

Input.

# dbt model schema.yml
version: 2

models:
  - name: dim_users
    columns:
      - name: user_id
        tests: [unique, not_null]
      - name: email
        meta:
          pii: email
          contains_pii: true
      - name: ssn
        meta:
          pii: ssn
          contains_pii: true

Code.

# Census destination policy.
destinations:
  marketo:
    allowed_pii_tags: [email, name]
    blocked_pii_tags: [ssn, phone, address]

  experimentation_tool:
    allowed_pii_tags: [name]
    blocked_pii_tags: [email, ssn, phone, address]
    # Note: email is blocked here.

# Census sync definition.
syncs:
  - name: users_to_marketo
    source: dim_users
    destination: marketo
    mappings:
      - source: email   -> Lead.Email          # OK — email allowed in Marketo
      - source: name    -> Lead.Name           # OK

  - name: users_to_experimentation
    source: dim_users
    destination: experimentation_tool
    mappings:
      - source: name    -> User.display_name   # OK
      - source: email   -> User.identifier     # BLOCKED — sync refuses to compile

Step-by-step explanation.

The dbt meta block tags the column with structured PII metadata. Census's dbt project reader picks up the tag automatically — no second source of truth.
The destination policy lists allowed and blocked PII categories per destination. Marketo accepts email + name; the experimentation tool accepts only name.
When the sync to Marketo compiles, every mapping is checked against the policy. Email → Lead.Email is allowed; the sync ships.
When the sync to the experimentation tool compiles, the email mapping triggers a policy violation. Census refuses to compile the sync; the engineer sees a clear error and either removes the mapping or escalates for an exception approval.
The policy is enforced at compile time, before any row hits a network. A misconfigured sync never reaches the destination.

Output.

Sync	Policy decision
users_to_marketo	compiles + ships
users_to_experimentation	refused to compile (email blocked)

Rule of thumb. Tag PII at the dbt column level; let the reverse ETL platform read tags and enforce per-destination policy at compile time. Never enforce PII policy at the row level at runtime — at runtime the data has already left the warehouse.

Worked example — the freshness SLA alert

Detailed explanation. Every sync has a freshness contract — "fresh within 2 hours" — set by the consuming team. The platform tracks the actual freshness and alerts when the contract is breached. The alert wakes on-call before the marketing team complains.

Question. Configure a freshness alert for the salesforce_lead_score_sync (cadence 30 min, SLA 2h) and walk through the on-call response when it fires.

Input.

Sync	Cadence	SLA	Freshness now
salesforce_lead_score_sync	30 min	2h	3h 15m ago

Code.

# Census alert definition (illustrative).
alerts:
  - name: lead_score_sync_freshness
    sync: salesforce_lead_score_sync
    condition: minutes_since_last_success > 120  # 2h SLA
    severity: page
    notify:
      - pagerduty: data-team-oncall
      - slack: "#data-alerts"
    runbook: |
      Sync has not succeeded in over 2 hours.
      Steps:
        1. Check Census dashboard for recent error.
        2. If 401 — refresh OAuth credential.
        3. If 429 — wait for quota reset; backfill afterwards.
        4. If model SQL error — open dbt repo, fix, redeploy.
        5. If destination outage — pause sync, monitor status page.

Step-by-step explanation.

The alert condition minutes_since_last_success > 120 measures actual freshness against the 2h SLA. The 30-minute cadence is the target; the SLA is the deadline.
When the alert fires, PagerDuty pages the on-call data engineer and posts to the Slack channel. The runbook is in the alert body, not in a separate wiki.
The on-call reads the Census dashboard, identifies the failure category (auth, quota, model error, destination outage), and applies the matching runbook step.
The runbook covers the four most-common failure modes. Steps 1–3 are operational; step 4 escalates to the model owner; step 5 escalates to the destination vendor.

Output (timeline of the on-call response).

Time	Event
03:00	Last successful run.
03:30	Scheduled run fails — 401 (token expired).
04:00	Second scheduled run fails — 401.
04:30	Third scheduled run fails — 401.
05:00	Freshness alert fires (2h SLA breached). PagerDuty pages on-call.
05:05	On-call reads runbook, refreshes OAuth credential.
05:10	Sync retries successfully. Freshness lag drops to 10 minutes.

Rule of thumb. Freshness lag is the right top-line SLI for a sync — not "did the last run succeed." A sync that runs and succeeds every hour is fine. A sync that runs every 30 minutes but has failed for the last 4 runs is broken, and only the freshness lag catches it.

Worked example — schema drift catches before deploy

Detailed explanation. Schema drift happens when a model's column type or name changes in a way the downstream sync cannot accept. The right place to catch it is in dbt CI, before merge — not in production after the sync starts failing.

Question. Configure dbt contracts on the reverse_etl_customer_state model and walk through what happens when a developer tries to rename lifetime_revenue to lifetime_value without coordinating with the sync.

Input.

# dbt contract on the model.
models:
  - name: reverse_etl_customer_state
    config:
      contract:
        enforced: true
    columns:
      - name: salesforce_contact_id
        data_type: varchar
        tests: [unique, not_null]
      - name: name
        data_type: varchar
      - name: lifetime_orders
        data_type: integer
      - name: lifetime_revenue
        data_type: numeric
      - name: last_order_at
        data_type: timestamp

Code.

-- Developer's PR — renames lifetime_revenue.
-- File: models/marts/reverse_etl_customer_state.sql
SELECT
    c.salesforce_contact_id,
    c.name,
    COUNT(o.order_id)                AS lifetime_orders,
    COALESCE(SUM(o.amount), 0)       AS lifetime_value,   -- renamed!
    MAX(o.order_date)                AS last_order_at
FROM dim_customers c
LEFT JOIN fact_orders o ON o.customer_id = c.customer_id
GROUP BY c.salesforce_contact_id, c.name;

Step-by-step explanation.

The developer renames lifetime_revenue to lifetime_value in the SELECT clause.
dbt CI runs dbt build. The contract check inspects the actual output schema against the declared columns: list.
The output column lifetime_value does not match the declared lifetime_revenue. dbt fails the build with a clear error: "column lifetime_revenue not produced; column lifetime_value produced unexpectedly."
The CI failure blocks the merge. The developer either reverts the rename or files a coordinated migration (rename in dbt + rename mapping in sync + cutover plan).
Without the contract, the rename would merge, the next sync run would silently ship NULL for lifetime_revenue (Salesforce field overwritten with NULL), and the marketing team would discover the bug three days later when their nurture sequence fires for everyone.

Output.

Stage	Outcome
Pre-contract	rename merges, sync silently writes NULL, downstream stale
With contract	rename blocked in CI, coordinated migration required

Rule of thumb. Every dbt model with at least one reverse ETL sync should have an enforced contract. The contract is the bridge between "data team owns the model" and "operational team owns the destination" — it makes drift loud instead of silent.

Reverse ETL interview question on the sync as a data product

A senior interviewer often asks: "How do you turn a one-off sync from a side-project into a production data product? What does the full lifecycle look like?"

Solution Using the data-product lifecycle

The data-product lifecycle for a reverse ETL sync:

1. INTAKE
   - Consumer team files a sync request.
   - Required fields: model, destination, fields, cadence,
     SLA, on-call owner.

2. DESIGN
   - Analytics engineer reviews the model PK + idempotency.
   - PII tags audited; destination policy verified.
   - Audience defined if filtering required.
   - dbt contract on the source model.
   - Cost estimate (quota + MTU).

3. BUILD
   - Sync YAML / config committed to git.
   - CI runs dbt build + sync linting.
   - PR review by analytics engineering.

4. DEPLOY
   - Sync deployed to staging destination first.
   - Manual QA on 10 sample rows.
   - Cut over to production destination.

5. MONITOR
   - dbt exposure surfaced in catalog.
   - Freshness alert + row-error alert configured.
   - Queue-depth alert configured.
   - On-call runbook attached.

6. ITERATE
   - Quarterly review of sync health metrics.
   - Audience drift review (size still in expected range?).
   - Destination policy review (PII still compliant?).
   - Cost review (still inside quota envelope?).

7. RETIRE
   - When the consumer no longer needs it: archive the sync,
     drop the dbt exposure, document the deprecation.

Step-by-step trace.

Phase	Owner	Output
Intake	Consumer team + AE	sync request ticket
Design	Analytics engineering	sync design doc
Build	Analytics engineering	sync YAML + PR
Deploy	Analytics engineering	staging then prod
Monitor	Data on-call	dashboards + alerts
Iterate	Analytics engineering	quarterly review notes
Retire	Analytics engineering	deprecation note

The discipline is the same as any backend service. The vocabulary borrows from product management (intake, MVP, monitoring, deprecation) more than from data engineering (model, refresh, materialise).

Output:

Artifact	Where it lives
Sync config	dbt repo / sync YAML
dbt contract	model schema.yml
dbt exposure	exposures.yml
Alerts	observability platform
Runbook	alert body + wiki
Cost budget	per-sync row cap + quota share
On-call rota	PagerDuty schedule

Why this works — concept by concept:

Intake gates entry — not every "we want a sync" idea becomes a sync. The intake form forces the consumer to articulate model, destination, SLA, and ownership before any engineering time is spent.
dbt contracts gate change — every sync model has an enforced contract. Drift is caught at PR time, not at production-failure time.
Exposures surface lineage — the data catalog knows every sync. When a model changes, the catalog shows every downstream sync that will be affected.
Alerts surface failure — freshness lag, row-error rate, and queue depth are the three SLIs. Every sync has them; on-call wakes up to them.
Quarterly review surfaces drift — audiences grow, costs shift, PII policy evolves. Quarterly review catches slow drift before it becomes an incident.
Retirement is explicit — syncs are retired explicitly, not abandoned. A retired sync is archived in git and removed from exposures so the catalog stays accurate.
Cost — the discipline is overhead. For a low-stakes internal sync, the full lifecycle is overkill. For any sync touching customer-facing automation, the lifecycle is the floor.

Data
Topic — data-transformation
Data transformation problems (data engineering)

Practice →

Cheat sheet — reverse ETL recipes

Lead score → Salesforce. Model fct_lead_score (one row per Salesforce contact) → audience "lead_score >= 80" → upsert into Contact.lead_score__c. Cadence: 30 minutes. Use composite API batching for 200 rows/call.
Account churn risk → Intercom. Model dim_accounts with churn_risk → audience "churn_risk > 0.7" → mirror sync sets Company.churn_risk_tag = at_risk and clears the tag when the account drops out of the audience.
High-value users → Facebook custom audience. Model dim_users joined to fct_user_revenue → audience "ltv_usd > 5000" → mirror sync hashes emails and pushes to a Meta custom audience. Reflects add/remove automatically on each run.
Slack high-value signup alert. Model fct_signups filtered to "plan = pro AND first_seen_at >= today" → RudderStack event sync → Slack webhook posts to #sales-alerts with the new account name + plan + region.
Marketing suppression list. Model dim_users filtered to "opted_out = true OR gdpr_deleted = true" → mirror sync to every marketing destination's suppression list (Marketo, Iterable, Customer.io, Mailchimp).
Reverse ETL → product analytics. Model marts.user_cohorts with (user_id, cohort_label) → upsert into Amplitude's cohorts API, mirrored to Mixpanel's cohort endpoint. Lets PMs filter funnels by warehouse-defined cohorts.
GDPR delete pipeline. Model users_to_delete (one row per requested deletion) → delete-only sync fanned out to Salesforce, HubSpot, Marketo, Intercom, Iterable, Facebook. Idempotent: a row deleted twice is a no-op.
Trial-ending sequence trigger. Model dim_users filtered to "plan = trial AND trial_ends_at BETWEEN today AND today + 7" → mirror sync to Iterable user property trial_end_date. Iterable workflow fires the in-app + email sequence.
Customer attribute fan-out. Single model marts.customer_attributes (one row per customer) → multiple syncs to Salesforce, HubSpot, Intercom, Iterable each picking the columns they need. One source, many destinations.
Sales territory routing. Model dim_accounts with territory_code → upsert into Salesforce Account.RoutingTerritory__c. Pairs with a Salesforce assignment rule that reads the field at lead creation.
NPS score sync. Model marts.nps (one row per account with rolling NPS) → upsert into Salesforce Account.nps_rolling__c. Customer success team filters Salesforce dashboards by NPS bucket.
Webhook fan-out. Model fct_account_events (one row per significant account event) → RudderStack event sync → internal API webhook, Slack channel, and Salesforce task creation in parallel.

Frequently asked questions

Is reverse ETL the same as a CDP?

Not quite — they overlap but solve different starting problems. A CDP (Customer Data Platform like Segment or RudderStack Event) collects events from your sources and forwards them to destinations; the warehouse is optional. Reverse ETL starts from the warehouse — it assumes you already have a single source of truth for customer attributes and ships that to destinations. The modern stack often uses both: a CDP collects events into the warehouse (forward path), and a reverse ETL tool ships warehouse-aggregated state back to operational tools (reverse path). RudderStack is unusual in offering both in one product; Hightouch and Census focus on the reverse ETL half only.

Do I need a customer data warehouse before reverse ETL?

Yes — you need a warehouse and a single canonical definition of the entity you want to sync. The warehouse can be Snowflake, BigQuery, Databricks, Redshift, or Postgres; it does not have to be branded a "customer data warehouse." What matters is that one SQL query produces one row per entity with the attributes you need to ship. If your data is still scattered across SaaS tools with no aggregation layer, you have a forward ETL problem first, and reverse ETL has nothing to sync.

How is Hightouch different from Census?

Hightouch optimises for the GTM / revenue ops persona — drag-and-drop audience builder, multi-channel journeys (Hightouch Sequences), broad destination catalogue (200+), strong observability with row-level error inspection. Census optimises for the analytics engineering / data team persona — tightest dbt integration of any vendor (reads dbt_project.yml, surfaces exposures, git-backed sync configs), SQL-first audience model, sync-test gating tied to dbt tests. Pick Hightouch when non-SQL users own the audience layer; pick Census when the data team owns it end-to-end and dbt is the source of truth.

Can I build reverse ETL myself with Airflow + APIs?

Yes, technically — and you should not, in practice. A v1 covering 5 destinations takes two senior engineers about 6 months to build: connectors, diff engine, queue + retry, dead-letter inspection, audience builder UI, schema-change detection, audit logging, PII governance. The three production vendors (Hightouch, Census, RudderStack) ship all of that for the price of about one engineer-year per year. The only cases where in-house build wins are (a) you have an extremely narrow scope (one destination, never more), (b) you are at a scale where MTU pricing genuinely hurts (>10M MTU and you can renegotiate hard), or (c) you have a hard BYOC compliance constraint and even RudderStack OSS does not fit.

What latency can reverse ETL realistically deliver?

Batch reverse ETL typically delivers 5–60 minute end-to-end latency, dominated by the warehouse query time plus the destination API throughput. Census claims sub-minute sync on small models with their fastest tier; Hightouch's shared infrastructure typically lands around 5–15 minutes. RudderStack's event-stream reverse ETL path closes the loop in seconds to a minute for individual event triggers but is not magic for batch attribute updates. If your use case requires sub-second response (in-session personalisation, fraud blocking, real-time bidding), reverse ETL is the wrong tool — you want an online feature store or an event-stream architecture that does not round-trip through a warehouse query.

How do I handle GDPR deletes through reverse ETL?

Build a dedicated delete pipeline. The pattern: one warehouse model users_to_delete with one row per requested deletion (user_id, email, requested_at), fanned out as a delete-only sync to every destination that received that user's PII. Each destination has a delete or "right-to-be-forgotten" API; Hightouch and Census both expose delete-only sync modes that wire into them. Idempotency matters — a user deleted twice should be a no-op. Audit-log every delete sync run for compliance evidence. Crucially, the platform itself must be able to delete its sync history for the deleted user; verify your vendor's GDPR posture before committing to PII-heavy syncs.

Practice on PipeCode

Drill the ETL practice library → for the warehouse-to-destination data movement patterns that reverse ETL formalises.
Layer in API integration drills → for the rate-limit + retry + idempotency primitives every sync depends on.
Stack the dimensional modelling library → so your reverse ETL models are one-row-per-entity by default.
Sharpen the data transformation library → for the aggregation patterns that turn fact tables into reverse ETL models.
Practise streaming problems → for the event-stream reverse ETL path RudderStack and modern Hightouch / Census tiers ship.
For the broader interview surface, read top data engineering interview questions →.
Stack the prerequisites with the only 5 skills you need to become a data engineer →.
Sharpen the system-design axis with the ETL system design course →.
For long-form data modelling craft, work through data modelling for DE interviews →.

Pipecode.ai is Leetcode for Data Engineering — every reverse ETL recipe above ships with hands-on practice rooms where you design the model, write the idempotent upsert, and reason about rate limits against real graded inputs. PipeCode pairs every reading with 450+ DE-focused problems and a real-time scoring engine, so you never have to wonder whether your sync design will hold up at scale.

Practice ETL now → API integration drills →

RAG Data Pipelines: Chunking, Embeddings, Vector Stores & Freshness

Gowtham Potureddi — Tue, 16 Jun 2026 14:15:39 +0000

rag pipeline looks like a model problem to a newcomer — interviewers know it is really a four-stage data pipeline problem dressed up in transformer vocabulary. The model is the cheap part; the expensive, error-prone, on-call-rotation part is the ingest, chunk, embed, index, and refresh loop that decides what context the model ever sees. When a retrieval-augmented generation system gives wrong answers in production, the fix almost never lives in the prompt — it lives in the pipeline.

This guide is the cheat sheet you wished existed the first time a stakeholder asked "why does the bot still cite the old policy?" It walks the end-to-end architecture, the four families of chunking strategies (fixed, recursive, semantic, hierarchical), embedding model selection and the metadata sidecar that travels with every chunk, hybrid dense + BM25 retrieval with cross-encoder reranking, and the freshness SLO + reindex playbook that keeps a rag data pipeline from drifting. Each section pairs a teaching block with a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the streaming practice library →, rehearse on ETL pipeline problems →, and stack the data-modelling muscles with dimensional modeling drills →.

On this page

RAG as a data pipeline problem, not a prompting problem
End-to-end RAG pipeline architecture
Chunking strategies — fixed, semantic, recursive, hierarchical
Embeddings + storage — choosing models and shaping the index
Freshness, reindex, and ACLs
Cheat sheet — RAG pipeline recipes
Frequently asked questions
Practice on PipeCode

1. RAG as a data pipeline problem, not a prompting problem

The model is the cheap part — the pipeline is where 80% of `rag pipeline` quality issues live

The one-sentence invariant: a RAG system is only as good as the chunks the retriever can return, and the chunks are only as good as the ingest, normalize, split, and embed pipeline that produced them. Once you internalise that "retrieval is the bottleneck," every late-night "why did it hallucinate that fact?" ticket resolves into one of four pipeline questions — was the source ingested? was it chunked at a sensible boundary? was the embedding model right for the content? was the index fresh?

The four stages every rag data pipeline shares.

Ingest. Pull documents from the source — Confluence, S3, Postgres, Notion, support tickets, code repos — into a staging zone. Strip boilerplate, expand attachments, dedupe. The output is a clean text corpus tagged with source metadata.
Chunk + embed. Split each document into retrieval-sized units (paragraphs, sections, sliding windows) and feed each chunk through an embedding model to produce a dense vector. Persist the chunk text, the vector, and a metadata sidecar.
Index. Push every vector into a vector store (Pinecone, Weaviate, pgvector, Qdrant, Milvus) and keep the chunk text in a sidecar store (Postgres, S3) so the retrieval path can hand the model the text, not just the vector ID.
Retrieve + rerank. At query time, embed the user question with the same model, do an approximate-nearest-neighbour search, fuse with a keyword score (BM25), rerank the top-N with a cross-encoder, and assemble the top-k chunks into the prompt.

Three places quality silently dies.

At the chunk boundary. A fixed-size chunker that splits mid-sentence loses the conceptual unit. The retriever returns the right vicinity but the model gets half a sentence — and confidently completes it the wrong way.
At the metadata boundary. No tenant_id on a chunk means the multi-tenant retrieval filter has nothing to filter on, and tenant A starts seeing tenant B's documents in the answer. This is the most expensive bug in retrieval augmented generation shipping today.
At the freshness boundary. A nightly batch reindex means "the new policy" is invisible to retrieval until 3am tomorrow. By then the stakeholder has already gone to Slack and screenshotted the wrong answer.

Where data engineers own the work.

DE owns ingest, chunking, embedding orchestration, vector store schema, metadata sidecar, freshness SLO, ACL pushdown, eval harness ingestion. Everything between source-of-truth and the moment a chunk arrives in the prompt.
ML / applied scientists own the embedding model choice, the reranker model choice, prompt template tuning, and the eval scoring rubric. Everything that decides "given a chunk, how is it scored / consumed."
Boundary. The eval harness is shared: DE supplies the golden Q&A inputs and the evaluation runs against the deployed pipeline; ML defines the metrics (recall@k, MRR, faithfulness).

The 2026 reality.

Vector stores are commoditising fast. pgvector inside Postgres handles low-tens-of-millions of vectors with reasonable latency; Pinecone, Qdrant, Weaviate, and Milvus take over above that. The choice is now a TCO decision, not a feature one.
Hybrid search is the default. Pure dense retrieval lost ~5-15 points of recall on out-of-distribution keywords vs hybrid; teams now ship dense + BM25 score fusion (RRF or weighted) as table stakes.
Reranking is mandatory above ~10 users. Cross-encoders are 50-100x slower than ANN but lift top-k precision dramatically. The standard shape is top-50 ANN → reranker → top-5 prompt.
Freshness is a contract. The mature pattern is CDC-from-source → embed worker → vector upsert, with a P95 source-to-retrieval lag SLO measured in minutes, not hours.

Worked example — the four ways RAG silently returns the wrong answer

Detailed explanation. Most "the bot lies" tickets fall into four pipeline buckets, and the fastest way to fix them is to know which bucket before you touch the prompt template. The four are: chunk boundary, missing metadata filter, stale index, embedding-model mismatch. Each has a code-shaped fix.

Question. A support bot keeps citing the wrong refund policy for tenant acme. The right policy is in Confluence, but the bot pulls the old one. List the four failure modes that could explain this and show the smallest pipeline-side test for each.

Input. A retrieved-context audit log row per query.

query_id	tenant_id	retrieved_chunk_id	retrieved_chunk_source	retrieved_chunk_text	last_modified
q1	acme	c123	confluence/acme-refunds-v1	"Refunds are 14 days..."	2025-08-01

Code.

# Four pipeline-side tests — run each before touching the prompt

# 1) Chunk boundary test: does the right chunk even exist?
chunks = vector_store.search(query="acme refund policy", filter={"tenant_id": "acme"})
for c in chunks[:5]:
    print(c.source, c.text[:120])
# expected: at least one chunk from confluence/acme-refunds-v2

# 2) Metadata filter test: is tenant_id present on the new doc's chunks?
new_chunks = vector_store.scan(filter={"source": "confluence/acme-refunds-v2"})
assert all(c.metadata.get("tenant_id") == "acme" for c in new_chunks)

# 3) Freshness test: how stale is the most-recent chunk for this source?
latest = max(c.metadata["last_modified"] for c in new_chunks)
print("source-to-retrieval lag:", now() - latest)

# 4) Embedding model test: same model on both write and read paths?
assert embed_model_id == vector_store.collection.embed_model_id

Step-by-step explanation.

Test 1 confirms whether the new chunk is retrievable at all. If it is not in the top results for an obvious query, the chunking strategy or the embedding step failed. If it is in the top 50 but not top 5, you have a reranker / score fusion problem, not a retrieval problem.
Test 2 confirms the metadata sidecar is intact. A common shape: the ingest job upserted text+vector but skipped the metadata payload. Without tenant_id, the multi-tenant filter pulls nothing for acme and the system falls back to a cross-tenant default — which still contains the old v1 doc.
Test 3 quantifies the freshness lag. If "now - last_modified" exceeds the SLO, the CDC stream is stalled or the embed worker is backed up. The fix is operational (drain the queue), not algorithmic.
Test 4 catches the silent killer: the team upgraded the embedding model on the write path but the retrieval service still embeds queries with the old one. Vectors are in different spaces; cosine similarity is meaningless. The fix is a blue/green collection swap, not a hot upgrade.

Output.

Failure mode	Test signal	Owner
Chunk missing / mis-split	search returns no v2 source	DE — chunker
Missing metadata	no tenant_id on v2 chunks	DE — ingest job
Stale index	source-to-retrieval lag > SLO	DE — CDC + worker
Embedding model mismatch	embed model IDs differ	ML + DE

Rule of thumb. Before touching the prompt template, run the four pipeline tests. If all four pass and the answer is still wrong, then the problem is in the model or the prompt — and you can hand it to the ML team with high confidence the data layer is clean.

Worked example — why "just use a bigger context window" does not fix bad chunks

Detailed explanation. A common temptation when a RAG system gives wrong answers is to widen the context window and stuff more chunks into every prompt. This reliably makes the bill explode without lifting accuracy, because the recall problem (the right chunk was not retrieved at all) is not solved by giving the model more wrong chunks. The fix is upstream: better chunking, better embeddings, hybrid + rerank — not a bigger prompt.

Question. Your support bot retrieves the top-3 chunks and returns wrong answers ~12% of the time. A colleague proposes lifting top-k from 3 to 30 (10x bigger context). Quantify why this is unlikely to fix the bug.

Input. Retrieval log breakdown over 1000 wrong-answer queries.

failure cause	share of wrong answers
right chunk not in top-50 at all	62%
right chunk in top-50 but ranked outside top-3	25%
right chunk in top-3 but model still hallucinated	13%

Code.

# Failure mode breakdown — a tiny helper to bucket wrong answers
def diagnose(query, gold_chunk_id, top_n=50):
    candidates = vector_store.search(query, k=top_n)
    ranks = [i for i, c in enumerate(candidates) if c.id == gold_chunk_id]
    if not ranks:
        return "not_in_top_n"          # recall miss — bigger k will not help
    rank = ranks[0]
    if rank >= 3:
        return "rank_too_low"          # rerank fix
    return "model_failed"              # prompt / model fix

buckets = collections.Counter(diagnose(q.text, q.gold_id) for q in failed_queries)
print(buckets)
# Counter({'not_in_top_n': 620, 'rank_too_low': 250, 'model_failed': 130})

Step-by-step explanation.

The "lift top-k from 3 to 30" idea only helps the 25% of wrong answers where the right chunk was in top-50 but ranked below 3 — and only partially, because the model now also has 27 extra noisy chunks to confuse it.
The dominant failure mode (62%) is recall miss — the right chunk is nowhere in the top-50. Increasing k from 3 to 30 changes none of those queries: the chunk is still missing. The fix is upstream — better chunking strategy, hybrid + BM25 score fusion, or re-embedding with a stronger model.
The "model failed" bucket (13%) is the only one a prompt or model upgrade can fix, and it is the smallest bucket.
Net: lifting top-k to 30 fixes at most 5-8% of the failures (the easier rerank misses), at 10x the LLM token cost. The right ROI is improving recall and adding a reranker.

Output.

Action	Failures fixed	Cost change
Lift top-k from 3 to 30	~5-8%	10x prompt tokens
Add hybrid + BM25 fusion	~40-50%	2x ingest, ~0 query
Add cross-encoder reranker	~20% (on top of hybrid)	+50ms latency
Rechunk with semantic splitter	~15%	one-time reindex

Rule of thumb. Diagnose the failure bucket before spending money. "Lift top-k" is the single most-tempting and least-effective RAG fix — every dollar in extra prompt tokens is roughly five dollars not spent on the actual recall problem upstream.

Data engineering interview question on `rag pipeline` ownership

A senior interviewer often opens with: "Walk me through the four stages of a production RAG pipeline, who owns each stage, and the single most common failure mode in each. Where does a data engineer add the most value?" It blends pipeline architecture, retrieval, and freshness into one ownership map.

Solution Using the stage-ownership matrix

# A minimal pipeline-ownership map — runnable as a doc-test
STAGES = [
    {
        "stage": "ingest",
        "owner": "DE",
        "common_failure": "missing source connector or stale CDC stream",
        "telemetry": "source_lag_seconds",
    },
    {
        "stage": "chunk_embed",
        "owner": "DE (chunker) + ML (model choice)",
        "common_failure": "chunk size too small / boundary mid-sentence",
        "telemetry": "chunks_per_doc, embed_qps",
    },
    {
        "stage": "index",
        "owner": "DE",
        "common_failure": "missing metadata sidecar (tenant_id, ACL)",
        "telemetry": "metadata_coverage_pct",
    },
    {
        "stage": "retrieve_rerank",
        "owner": "DE (infra) + ML (reranker)",
        "common_failure": "embedding model mismatch between write and read",
        "telemetry": "recall_at_5, mrr_at_10",
    },
]

for s in STAGES:
    print(f"{s['stage']:>16}  owned by {s['owner']:<30}  watch: {s['telemetry']}")

Step-by-step trace.

Stage	Owner	Common failure	Telemetry
ingest	DE	stale CDC stream	source_lag_seconds
chunk_embed	DE + ML	chunk size too small	chunks_per_doc
index	DE	missing metadata	metadata_coverage_pct
retrieve_rerank	DE + ML	embed model mismatch	recall_at_5

The trace highlights that three of four stages are DE-owned outright, and the fourth is shared. The senior signal in this answer is naming the metadata sidecar coverage as the biggest preventable failure — most candidates focus on the embedding model and miss that the index schema is where the multi-tenant safety lives.

Output:

Stage	DE value-add	Single biggest win
ingest	source connectors + CDC	sub-minute source lag
chunk_embed	strategy per content type	semantic + hierarchical
index	metadata schema + ACL	100% tenant_id coverage
retrieve_rerank	hybrid + filter pushdown	dense + BM25 fusion

Why this works — concept by concept:

Stage ownership matrix — naming the owner per stage frames RAG as a data product, not a model deployment. Interviewers reward this framing because it predicts how the candidate will run on-call rotations.
Telemetry per stage — the senior move is to attach an observable metric to each stage so the on-call rotation can localise failures in seconds. Source lag, metadata coverage, recall@5 are the canonical three.
Metadata sidecar is the safety story — every chunk carries tenant_id, source, ACL, last_modified — these are not optional, they are the only thing keeping retrieval augmented generation from cross-tenant leakage and stale answers.
Hybrid + rerank is the precision story — dense retrieval alone misses on rare keywords; BM25 alone misses on synonyms; fusion + rerank is the production default for hybrid search.
Cost — ingest is O(docs) one-time + O(changes) per CDC tick; embed is O(chunks) at write time; retrieve is O(log N) ANN + O(top-N) rerank per query. Reranker dominates query latency; embed dominates ingest latency.

DE
Topic — ETL
ETL pipeline problems (DE)

Practice →

2. End-to-end RAG pipeline architecture

The four-stage `rag data pipeline` — ingest, chunk + embed, index, retrieve + rerank — and the metadata sidecar that travels with every chunk

The mental model in one line: a production rag pipeline is two pipelines joined at a vector store — an offline batch+CDC pipeline that ingests, chunks, embeds, and upserts; and an online request pipeline that embeds the query, searches, reranks, and assembles the prompt. Once you draw those two flows on a whiteboard, every RAG architecture question collapses into "where does this responsibility sit?"

The offline ingest pipeline.

Source connectors. One per source system. Confluence, Notion, Google Drive, S3, Postgres CDC, Slack export, GitHub repo, support ticket export. Each emits raw documents into a staging bucket with provenance metadata.
Normalize + strip. Strip HTML / Markdown formatting, remove boilerplate (navigation, footers, code-of-conduct banners), expand attachments, OCR images if needed, dedupe near-identical pages.
Split into logical units. Section-aware split: headings define sections, paragraphs define chunks within sections. Tables and code blocks are split as atomic units (do not chunk a table).
Chunker. Apply the per-content-type chunking strategy (covered in detail in section 3). Output is (doc_id, chunk_idx, text, metadata).
Embedder. Batch-embed chunks through the embedding model. Cache by content hash so unchanged chunks are not re-embedded on every run.
Vector store upsert. Upsert (vector, chunk_id, metadata) into the vector store. The chunk text goes to a sidecar store (Postgres, DynamoDB, S3) keyed by chunk_id.

The online retrieval pipeline.

Query embed. The same embedding model encodes the user query into a vector. Critical: write and read paths must use the same model.
ANN search with metadata filter. Approximate-nearest-neighbour search returns top-N candidates restricted by the metadata filter (tenant_id, source, ACL, recency window).
Score fusion (hybrid). Dense ANN scores are fused with a sparse BM25 score from a parallel inverted index. Weighted sum or Reciprocal Rank Fusion (RRF) are the two common shapes.
Reranker. A cross-encoder model takes the query + each of the top-N candidates and produces a precision-tuned score. Top-k after rerank are the chunks that go into the prompt.
Prompt assembly. Top-k chunks are concatenated with provenance headers ("Source: confluence/page-123, modified 2026-06-10") so the LLM can cite sources.

The metadata sidecar — what every chunk carries.

tenant_id — for multi-tenant SaaS, every retrieval is filtered by tenant_id = ?. No exceptions.
source — where the chunk came from (confluence/page-123, s3://bucket/key, postgres://schema.table#row). Powers attribution and trust signals.
document_id — group chunks back to their parent document for "show me the full doc" follow-ups.
last_modified — for freshness SLO measurement and time-window filters.
acl_ids — list of permission tags that the retrieval filter intersects with the user's permissions.
content_type — prose, code, table, transcript — used to pick the right reranker or trigger content-specific rules.
embed_model_id — the embedding model version that produced this vector. Critical for blue/green model swaps.

Observability — what to graph.

Source lag. Per source, the P95 source-to-retrieval lag. The single most important SLO for rag freshness.
Index hit rate. What fraction of queries return any chunks at all (low = empty store or over-filtered metadata).
Fallback ratio. Fraction of queries that fall through to the "no context" prompt path. A sharp uptick is the first signal of a stalled embed worker.
Recall@5 on golden set. Offline metric scored nightly against a curated Q&A set with known-correct chunk IDs.
Reranker latency P95. Cross-encoders dominate query latency; alert on regressions.

The eval harness.

Golden Q&A set. 200-2000 curated (question, expected_chunk_id, expected_answer) triples maintained by the product team plus SMEs.
Offline eval. Nightly run scores recall@k, MRR, faithfulness against the golden set. Drops trigger PagerDuty.
Online eval (LLM-as-judge). Sample of live queries scored by a stronger LLM against rubric. Drift detection.
Shadow traffic. New embedding model or chunker runs in shadow mode against live queries before the swap — compare top-5 overlap and recall before promoting.

Worked example — building the offline ingest stage in 50 lines

Detailed explanation. The minimal happy path: pull a document, normalize, chunk with overlap, embed in a batch, upsert with metadata. Everything in production is a hardening of these five steps — retries, dedup, content-hash caching, sidecar persistence.

Question. Sketch the offline ingest stage that takes a Confluence page, splits it into 500-token chunks with 75-token overlap, embeds each chunk, and upserts into a vector store with a tenant-aware metadata sidecar.

Input. A single Confluence page object.

field	value
page_id	"PAGE-42"
tenant_id	"acme"
body_markdown	"## Refund policy\n\nAt Acme..." (~3500 tokens)
last_modified	"2026-06-10T09:14:00Z"

Code.

import hashlib
from typing import Iterable

CHUNK_TOKENS = 500
OVERLAP_TOKENS = 75


def chunk_text(text: str, size: int, overlap: int) -> Iterable[str]:
    """Sliding-window chunker — tokens approximated as whitespace splits."""
    tokens = text.split()
    step = size - overlap
    for i in range(0, len(tokens), step):
        yield " ".join(tokens[i : i + size])
        if i + size >= len(tokens):
            break


def ingest_page(page: dict) -> None:
    chunks = list(chunk_text(page["body_markdown"], CHUNK_TOKENS, OVERLAP_TOKENS))
    # Batch-embed all chunks in one call — 10-50x cheaper than per-chunk
    vectors = embed_model.encode(chunks)
    rows = []
    for idx, (text, vec) in enumerate(zip(chunks, vectors)):
        chunk_id = f"{page['page_id']}#chunk-{idx}"
        rows.append({
            "id": chunk_id,
            "vector": vec,
            "metadata": {
                "tenant_id": page["tenant_id"],
                "source": f"confluence/{page['page_id']}",
                "document_id": page["page_id"],
                "last_modified": page["last_modified"],
                "chunk_idx": idx,
                "content_hash": hashlib.sha256(text.encode()).hexdigest(),
                "embed_model_id": embed_model.id,
            },
        })
        text_store.put(chunk_id, text)            # sidecar — chunk text in Postgres / S3
    vector_store.upsert(rows)                     # vectors + metadata

Step-by-step explanation.

The chunker is a sliding-window splitter — 500 tokens forward, 75 tokens back, so each chunk shares ~15% with its neighbours. Overlap is what saves you when the answer straddles a chunk boundary.
embed_model.encode(chunks) is called once with the full batch. Per-chunk calls are 10-50x more expensive due to per-request overhead and HTTP round-trips.
Every chunk gets a deterministic chunk_id derived from the document id and chunk index, so re-ingesting the same page produces the same IDs — upsert overwrites instead of duplicating.
The metadata sidecar carries every filter the retrieval path will ever need: tenant_id (multi-tenancy), source and document_id (attribution), last_modified (freshness), chunk_idx (sibling lookup), content_hash (dedup / skip-unchanged), and embed_model_id (blue/green safety).
The chunk text goes into a sidecar text store keyed by chunk_id. The vector store only stores the vector — sidecar lookup happens at the prompt-assembly stage.
Idempotency: re-ingesting the same page is a no-op for unchanged chunks (same hash, same vector, same metadata) and a one-row update for changed chunks.

Output.

chunk_id	tenant_id	text snippet	vector dims
PAGE-42#chunk-0	acme	"## Refund policy At Acme..."	1536
PAGE-42#chunk-1	acme	"...within 14 days of purchase..."	1536
PAGE-42#chunk-2	acme	"...exceptions for digital goods..."	1536

Rule of thumb. Every chunk needs three things from day one: a deterministic ID, a content hash, and a metadata sidecar that includes tenant_id, source, and last_modified. Bolt them on later and you are rewriting the ingest job, not patching it.

Worked example — the online retrieval stage end-to-end

Detailed explanation. Once the ingest stage has populated the index, the online path is a tight five-step pipeline: embed query, ANN search with metadata filter, BM25 fusion, rerank, assemble prompt. The whole loop should run in <300ms P95 to feel responsive.

Question. Sketch the online retrieval stage that takes a user query, applies the tenant filter, fuses dense + BM25 scores, reranks the top-50 with a cross-encoder, and returns the top-5 chunks plus assembled prompt.

Input. A single user query plus the calling user's tenant and ACL list.

field	value
query	"What is Acme's refund window for digital goods?"
tenant_id	"acme"
acl_ids	["public", "support"]

Code.

def retrieve_and_rerank(query: str, tenant_id: str, acl_ids: list[str]) -> dict:
    # 1) Embed the query with the SAME model used at ingest
    qvec = embed_model.encode([query])[0]

    # 2) ANN search with metadata filter — tenant + ACL pushdown
    candidates = vector_store.search(
        vector=qvec,
        top_k=50,
        filter={
            "tenant_id": tenant_id,
            "acl_ids": {"$in": acl_ids},
        },
    )

    # 3) Hybrid score fusion — Reciprocal Rank Fusion (RRF) with k=60
    bm25_hits = bm25_index.search(query, top_k=50, filter={"tenant_id": tenant_id})
    fused = rrf_fuse(candidates, bm25_hits, k=60)

    # 4) Cross-encoder rerank on the top 50 fused candidates
    pairs = [(query, text_store.get(c.id)) for c in fused[:50]]
    rerank_scores = reranker.score(pairs)
    top5 = sorted(zip(fused[:50], rerank_scores), key=lambda x: -x[1])[:5]

    # 5) Assemble the prompt with provenance headers
    blocks = []
    for cand, _ in top5:
        m = cand.metadata
        text = text_store.get(cand.id)
        blocks.append(
            f"[source: {m['source']} · modified {m['last_modified']}]\n{text}"
        )
    return {
        "prompt_context": "\n\n---\n\n".join(blocks),
        "citations": [c.metadata["source"] for c, _ in top5],
    }


def rrf_fuse(dense, sparse, k=60):
    """Reciprocal Rank Fusion — combine two ranked lists by 1/(k+rank)."""
    scores: dict[str, float] = {}
    for rank, hit in enumerate(dense):
        scores[hit.id] = scores.get(hit.id, 0) + 1 / (k + rank + 1)
    for rank, hit in enumerate(sparse):
        scores[hit.id] = scores.get(hit.id, 0) + 1 / (k + rank + 1)
    return sorted(
        {c.id: c for c in dense + sparse}.values(),
        key=lambda c: -scores[c.id],
    )

Step-by-step explanation.

The query is embedded with the same model that produced the index vectors. If the model IDs disagree, vectors are in different spaces and cosine similarity is noise — this is the silent killer covered in section 1.
The ANN search pushes the tenant filter down into the index — it is not a post-filter. Pinecone, Qdrant, Weaviate, and pgvector all support metadata predicate pushdown, and using it is the difference between a 10ms query and a 500ms query at scale.
The BM25 search runs in parallel against a sparse inverted index (Elasticsearch, OpenSearch, Lucene) with the same tenant filter. Dense + sparse together is the hybrid search shape.
RRF fuses the two ranked lists by summing 1/(k+rank) — no need to normalize scores, no per-system weight tuning. k=60 is the standard starting value from the IR literature.
The cross-encoder reranker takes (query, chunk_text) pairs and produces a precision-tuned score. It is 50-100x slower per pair than ANN, so it only runs on the top 50 — never the full corpus.
Top-5 reranked chunks become the prompt context. Each block carries a source and last_modified header so the LLM can quote the source and the consumer can audit freshness.

Output.

step	result
ANN top-50 (dense)	50 candidates
BM25 top-50 (sparse)	50 candidates
RRF fused	~70 unique candidates
Reranked top-5	5 chunks for prompt
Prompt context	~2500 tokens with citations

Rule of thumb. The online stage is a fixed five-step pipeline: embed → ANN+filter → BM25+fusion → rerank → assemble. Every step has a 100ms budget; every step has its own telemetry. If P95 latency drifts, the slow step is almost always the reranker — cap the rerank candidate set and tune k.

Worked example — the eval harness golden-set scoring loop

Detailed explanation. A golden Q&A set is the only reliable way to know whether a pipeline change improved or hurt retrieval. The shape: a CSV / Parquet of (question, expected_chunk_id, expected_answer) triples maintained by SMEs. The harness runs every nightly job and scores recall@k and MRR. A sustained drop pages the on-call.

Question. Sketch a nightly offline eval that scores the current pipeline against a golden set and reports recall@5 and mean reciprocal rank (MRR) at 10.

Input. A 500-row golden set.

question_id	question	expected_chunk_id
g001	"Refund window for digital goods?"	PAGE-42#chunk-1
g002	"How long is the trial period?"	PAGE-09#chunk-3

Code.

def score_pipeline(golden: list[dict], k: int = 5) -> dict:
    recall_hits = 0
    rr_sum = 0.0
    for q in golden:
        hits = retrieve_and_rerank(
            query=q["question"],
            tenant_id=q.get("tenant_id", "default"),
            acl_ids=q.get("acl_ids", ["public"]),
        )
        ids = [h.id for h in hits["chunks"][:10]]
        if q["expected_chunk_id"] in ids[:k]:
            recall_hits += 1
        try:
            rank = ids.index(q["expected_chunk_id"]) + 1
            rr_sum += 1 / rank
        except ValueError:
            pass  # not in top-10 → reciprocal rank contributes 0
    return {
        "recall_at_k": recall_hits / len(golden),
        "mrr_at_10": rr_sum / len(golden),
        "n": len(golden),
    }


# Nightly job
result = score_pipeline(load_golden_set(), k=5)
publish_metric("rag.recall_at_5", result["recall_at_k"])
publish_metric("rag.mrr_at_10", result["mrr_at_10"])
if result["recall_at_k"] < 0.85:
    page_oncall("recall@5 regression", result)

Step-by-step explanation.

For each golden question, the harness runs the real online retrieval pipeline — same embed, same filter, same rerank — against the live index. No mocking.
recall_at_k is "did the gold chunk appear in the top-k?" averaged across the golden set. The standard production threshold is 0.85-0.95 depending on stakes.
MRR at 10 is "the average of 1/rank over the golden set, with rank=∞ contributing 0." MRR rewards getting the right chunk high in the ranking — it is a precision-tilted recall metric.
Both metrics are published to the metric store and trended over time. A sustained drop ≥5 points triggers PagerDuty and a rollback investigation.
The harness is the single source of truth on whether a chunking change, an embedding model upgrade, or a reranker swap helped. Without it, every change is a vibe-based decision.

Output.

metric	value	threshold
recall@5	0.91	≥ 0.85
MRR@10	0.74	≥ 0.65
n	500	—

Rule of thumb. No golden set, no production RAG. Build a 200-row golden set on day one; grow it to 1000-2000 as edge cases emerge. The harness is what turns RAG from "vibes shipping" into a real data product with an SLO.

Data engineering interview question on online vs offline pipelines

A senior interviewer might frame this as: "Draw the offline ingest and online retrieval pipelines on a whiteboard, then mark where each one can fail at 3am and how you would alert on it." It probes both system design and on-call instincts.

Solution Using a two-pipeline diagram + failure-mode catalogue

# Two pipelines joined at the vector store
OFFLINE = [
    "source_connector",
    "normalize",
    "chunk",
    "embed",
    "vector_upsert",
]
ONLINE = [
    "query_embed",
    "ann_search_with_filter",
    "bm25_search",
    "rrf_fuse",
    "rerank",
    "assemble_prompt",
]

FAILURES = {
    "source_connector": "auth expired / source down → source_lag_seconds spike",
    "chunk": "boundary mid-sentence → recall@5 drop on golden set",
    "embed": "model API down → embed_queue_depth grows unbounded",
    "vector_upsert": "schema mismatch → upsert_error_rate spike",
    "ann_search_with_filter": "metadata not indexed → P95 latency spike",
    "rerank": "model timeout → fallback_ratio spike",
}

Step-by-step trace.

Pipeline	Stage	Failure signal	Alert
offline	source_connector	source_lag_seconds > 600	page
offline	embed	embed_queue_depth growing	page
offline	vector_upsert	upsert_error_rate > 1%	page
online	ann_search	P95 latency > 200ms	page
online	rerank	fallback_ratio > 5%	page
online	assemble_prompt	empty_context_ratio > 2%	warn

The trace highlights that offline and online have orthogonal failure modes — an offline stall produces a staleness failure; an online stall produces a latency or quality failure. Each needs its own SLO.

Output:

Pipeline	Top SLO	Alert threshold
offline ingest	P95 source-to-retrieval lag	< 5 min
online retrieve	P95 end-to-end latency	< 300 ms
online quality	recall@5 (nightly)	≥ 0.85

Why this works — concept by concept:

Two pipelines, one vector store — the offline path is throughput-bound (catch up on backlog); the online path is latency-bound (respond in <300ms). Splitting them lets you scale the embed worker independently from the retrieval service.
SLO per pipeline — the offline SLO is a freshness number (source_lag_seconds); the online SLO is a latency number (P95_response_time); the quality SLO is an offline metric (recall@5). All three matter.
Filter pushdown — the metadata filter (tenant_id, acl_ids) is pushed into the ANN search, not applied after. This is the single biggest performance lever in the online path.
Reranker is the precision lever — without it, you get hybrid-search recall but consumer-noisy ranking. With it, top-5 precision climbs sharply at the cost of ~50-100ms.
Cost — offline: O(docs) one-time + O(changes) per CDC tick + O(chunks) embedding cost. Online: one embed call + one ANN query + one BM25 query + one rerank batch per request. Reranker dominates online cost; embedding dominates offline cost.

DE
Topic — streaming
Streaming pipeline problems (DE)

Practice →

3. Chunking strategies — fixed, semantic, recursive, hierarchical

`chunking strategies` decide what the retriever can ever return — pick the strategy per content type, not per project

The mental model in one line: a chunk is the smallest unit your retriever can return, so the chunk shape is the upper bound on retrieval precision — split mid-sentence and the model gets half an idea; split mid-paragraph and you lose the connective tissue between claims. Once you say "the chunk is the retrieval unit," you stop optimising token counts and start optimising meaningful boundaries.

The four families.

Fixed token windows. Pick a target size (e.g. 500 tokens) and a stride. Cheap, deterministic, dumb at boundaries. Default starting point — beat it before adding complexity.
Recursive character / token splitter. Try to split by paragraph; if too big, fall back to sentence; if still too big, fall back to a fixed token window. LangChain's RecursiveCharacterTextSplitter popularised this shape.
Semantic chunking. Embed each sentence and split where the cosine similarity between adjacent sentences drops below a threshold — i.e. where the topic visibly shifts. Higher quality, ~2-3x more expensive at ingest.
Hierarchical / parent-child. Index small child chunks (for high-recall retrieval) but at prompt time return the parent chunk (a paragraph or section) that contains the matched child. The model gets context; the retriever gets precision.

Overlap windows.

What overlap is. Each chunk shares its first N tokens with the previous chunk's last N tokens. 10-20% is the standard range (~50-100 tokens for a 500-token chunk).
Why it matters. A claim that straddles a chunk boundary appears in both chunks instead of being orphaned. Overlap is the cheapest insurance against boundary loss.
When to skip it. Atomic content (a code block, a table, a paragraph) does not need overlap — it is already its own unit.

Per-content-type strategy table.

Content type	Recommended strategy	Notes
Prose / docs	recursive or semantic, 400-600 tokens, 15% overlap	semantic if budget allows
Code	one chunk per function / class (AST split)	never split mid-function
Tables / structured	one chunk per table or per row group	preserve column headers
Transcripts / chat	one chunk per turn or per N seconds	speaker label as metadata
FAQs	one chunk per Q+A pair	the Q is the retrieval signal
Long PDFs (manuals)	hierarchical: child = paragraph, parent = section	retrieve child, serve parent

Common chunking interview probes.

"What is the right chunk size?" — there is no universal right; start at 500 tokens with 75 overlap, then tune against the golden set. Senior signal: name "embedding model context window" as the hard upper bound (most embedding models are 512 tokens) and "downstream LLM context budget" as the soft constraint.
"When do you use semantic chunking?" — when prose has shifting topics within a single document (long-form blogs, transcripts, multi-topic reports). Skip it for tightly-scoped reference docs where fixed windows match natural paragraph length.
"What is parent-child chunking?" — index small chunks for high-recall ANN, but at prompt time return the parent chunk (a paragraph, section, or sliding window) so the LLM gets enough context. Standard shape on long-form RAG.
"How do you chunk code?" — never mid-function. Use the language's AST to split at function and class boundaries; index the function signature + docstring + body as one chunk.

Worked example — fixed-window vs recursive splitter on the same document

Detailed explanation. A fixed-window splitter is the simplest possible chunker — slide a window of N tokens with stride S. It is fast, but it cheerfully cuts sentences and paragraphs in half. A recursive splitter tries semantic boundaries first (paragraphs, then sentences) before falling back to a fixed window, so most splits land at natural boundaries.

Question. Implement a fixed-window splitter and a recursive splitter for a 1200-token document. Compare the splits and show why the recursive version preserves more semantic units.

Input. A document with three paragraphs (400 + 350 + 450 tokens, separated by \n\n).

paragraph	tokens
P1 — refund policy	400
P2 — exceptions	350
P3 — appeals	450

Code.

def fixed_window(text: str, size: int = 500, overlap: int = 75) -> list[str]:
    tokens = text.split()
    out, step = [], size - overlap
    for i in range(0, len(tokens), step):
        chunk = tokens[i : i + size]
        if not chunk:
            break
        out.append(" ".join(chunk))
        if i + size >= len(tokens):
            break
    return out


def recursive_split(text: str, size: int = 500, overlap: int = 75) -> list[str]:
    """Try paragraph → sentence → fixed-window fallback."""
    # 1) Try paragraph split first
    paras = text.split("\n\n")
    out: list[str] = []
    for p in paras:
        ptokens = p.split()
        if len(ptokens) <= size:
            out.append(p)
            continue
        # 2) Paragraph too big → fall back to sentence split
        sents = re.split(r"(?<=[.!?])\s+", p)
        buf: list[str] = []
        buf_tok = 0
        for s in sents:
            st = s.split()
            if buf_tok + len(st) > size and buf:
                out.append(" ".join(buf))
                buf, buf_tok = [], 0
            buf.append(s)
            buf_tok += len(st)
        if buf:
            out.append(" ".join(buf))
        # 3) Sentence-level chunks still too big? fall back to fixed window
        out = [c if len(c.split()) <= size else fixed_window(c, size, overlap)
               for c in out]
    # Flatten any nested lists from step 3
    flat: list[str] = []
    for c in out:
        flat.extend(c) if isinstance(c, list) else flat.append(c)
    return flat

Step-by-step explanation.

Fixed-window on the 1200-token document with size=500, overlap=75 produces three chunks: tokens 0-500, 425-925, 850-1200. The first chunk ends mid-paragraph 2 (token 500 is inside P2). The second chunk starts mid-paragraph 2. Boundary loss.
Recursive split tries paragraph boundaries first. P1 (400 tokens) is ≤500, so it becomes one chunk. P2 (350 tokens) is ≤500, so it becomes one chunk. P3 (450 tokens) is ≤500, so it becomes one chunk. Three clean paragraph-aligned chunks, zero mid-sentence splits.
If a paragraph exceeds 500 tokens, the recursive splitter falls back to sentence split — still semantic, just one level finer. Only paragraphs and sentences that exceed the size fall back to the dumb fixed-window splitter.
The recursive version produces more chunks on average (one per paragraph instead of one per 500-token window), but every chunk respects a natural boundary. Retrieval precision improves at the cost of slightly more chunks to index.

Output.

Strategy	# chunks	mid-sentence splits
fixed_window(500, 75)	3	2 (inside P2)
recursive_split(500, 75)	3	0

Rule of thumb. Default to recursive splitting for prose — the cost over a fixed window is negligible (one extra pass over the text) and the recall lift is consistent. Reserve plain fixed-window for tightly-controlled content types where the natural unit is already the window size (timeseries, log lines, code lines).

Worked example — semantic chunking by similarity drop

Detailed explanation. Semantic chunking embeds each sentence and walks the document looking for drops in cosine similarity between adjacent sentences — those drops mark topic shifts. A chunk is the run of sentences between two drops. The cost is one extra embedding call per sentence at ingest, but the chunks line up with topical boundaries that no syntactic splitter can find.

Question. Implement a semantic chunker that splits a document at sentence boundaries where the cosine similarity between consecutive sentences drops below a threshold.

Input. A 6-sentence document that shifts topic twice.

sentence_idx	text
0	"Refund policy: we accept returns within 14 days."
1	"Returns must be in original packaging."
2	"Shipping is free on orders over $50."
3	"We use FedEx and UPS for delivery."
4	"Our support hours are 9 to 5 EST."
5	"Reach us via email or chat."

Code.

def semantic_chunk(text: str, threshold: float = 0.55) -> list[str]:
    sents = re.split(r"(?<=[.!?])\s+", text)
    if len(sents) < 2:
        return sents

    vecs = embed_model.encode(sents)
    boundaries = [0]  # split *before* these indices

    for i in range(1, len(vecs)):
        sim = cosine(vecs[i - 1], vecs[i])
        if sim < threshold:
            boundaries.append(i)
    boundaries.append(len(sents))

    chunks: list[str] = []
    for a, b in zip(boundaries[:-1], boundaries[1:]):
        chunks.append(" ".join(sents[a:b]))
    return chunks


def cosine(a, b):
    import math
    dot = sum(x * y for x, y in zip(a, b))
    na = math.sqrt(sum(x * x for x in a))
    nb = math.sqrt(sum(y * y for y in b))
    return dot / (na * nb + 1e-9)

Step-by-step explanation.

Each sentence is embedded once at ingest time — this is the cost driver. For a 100-sentence document, semantic chunking issues 100 embed calls instead of zero (fixed-window).
Adjacent sentences are compared pairwise. The cosine similarity between sentence 1 ("packaging") and sentence 2 ("shipping") is below 0.55 → boundary inserted before sentence 2.
The walk continues: sentence 3 vs 4 similar (both about shipping), no split. Sentence 4 vs 5 below 0.55 (shipping → support hours), boundary inserted.
Three chunks emerge: [0,1] refund/returns, [2,3] shipping, [4,5] support. Each chunk is a topical unit, even though they all sit in one source document.
Threshold is tuned on the golden set. Too high → too many chunks (every sentence is its own chunk); too low → no splits at all. 0.45-0.65 is the typical range with OpenAI / Cohere embeddings.

Output.

chunk_idx	sentences	topic
0	0, 1	refund / returns
1	2, 3	shipping
2	4, 5	support

Rule of thumb. Use semantic chunking when documents mix topics (long-form blogs, all-hands transcripts, multi-section policy docs). Skip it when each source document is already a single tight topic — the marginal recall gain does not pay for the ingest cost.

Worked example — hierarchical (parent-child) chunking

Detailed explanation. Index small chunks (sentences or short paragraphs) for high-recall ANN matching, but at prompt assembly time return the parent chunk (a section or full paragraph) that contains the matched child. The retriever gets to match on tight semantic units; the LLM gets enough surrounding context to answer.

Question. Implement a parent-child chunker that indexes sentence-level child chunks but returns the paragraph-level parent at retrieval time.

Input. A 3-paragraph document where each paragraph has 2-3 sentences.

paragraph_idx	sentence_idx	text
0	0	"Refunds are accepted within 14 days."
0	1	"Returns must be in original packaging."
1	0	"Digital goods are non-refundable."
1	1	"Subscriptions cancel at period end."
2	0	"Contact support at help@acme.com."

Code.

def parent_child_chunk(doc_id: str, text: str) -> tuple[list[dict], dict]:
    parents: dict[str, str] = {}
    children: list[dict] = []

    paras = text.split("\n\n")
    for p_idx, para in enumerate(paras):
        parent_id = f"{doc_id}#para-{p_idx}"
        parents[parent_id] = para

        sents = re.split(r"(?<=[.!?])\s+", para)
        for s_idx, sent in enumerate(sents):
            children.append({
                "id": f"{doc_id}#sent-{p_idx}-{s_idx}",
                "text": sent,
                "parent_id": parent_id,
            })
    return children, parents


def retrieve_with_parent(query: str, top_k: int = 5) -> list[str]:
    # 1) ANN search over the child (sentence) index
    child_hits = vector_store.search(query=query, top_k=top_k * 3)

    # 2) Resolve to parent chunks, dedupe — same parent counted once
    seen, out = set(), []
    for c in child_hits:
        pid = c.metadata["parent_id"]
        if pid in seen:
            continue
        seen.add(pid)
        out.append(parent_store.get(pid))
        if len(out) == top_k:
            break
    return out

Step-by-step explanation.

At ingest, every sentence becomes a child chunk with a vector, and every paragraph becomes a parent record (text only — no vector needed because parents are never directly searched).
Each child carries a parent_id in its metadata sidecar so the retrieval path can resolve from match back to context.
At query time, the ANN search runs over the child index — short, semantically tight units that match queries crisply.
The result list of child matches is collapsed by parent_id (dedupe) and the parent paragraph text is returned. If sentences 0 and 1 of paragraph 0 both match, the paragraph is returned once.
top_k * 3 over-retrieves on the child side because multiple children from the same parent may match — over-fetch lets you still emit top_k unique parents.

Output (for query "How long for refunds?").

match	child_id	parent returned
1st hit	doc#sent-0-0	"Refunds are accepted within 14 days. Returns must be in original packaging."

Rule of thumb. Use parent-child when chunks need to be small enough for tight retrieval (sentence-level) but the LLM needs surrounding context to answer (paragraph or section level). It is the default shape for technical docs, legal docs, and long-form policy.

Worked example — overlap window math and why 15% is the default

Detailed explanation. Overlap is the cheapest insurance against a fact landing across a chunk boundary. The math is simple: with overlap O on a chunk of size S, every claim within the first O tokens of a chunk also appears in the previous chunk; every claim within the last O tokens also appears in the next chunk. A claim has two shots at being returned.

Question. Given chunk size 500 tokens and overlap 75 tokens, what fraction of the document appears in two chunks? When is overlap worth it and when is it waste?

Input. A document of 5000 tokens.

Code.

def overlap_stats(doc_tokens: int, size: int, overlap: int) -> dict:
    step = size - overlap
    n_chunks = (doc_tokens - overlap) // step + (1 if (doc_tokens - overlap) % step else 0)
    total_tokens_stored = n_chunks * size
    redundant = total_tokens_stored - doc_tokens
    return {
        "n_chunks": n_chunks,
        "total_tokens_stored": total_tokens_stored,
        "redundant_tokens": redundant,
        "overlap_pct": overlap / size,
        "storage_overhead_pct": redundant / doc_tokens,
    }


for overlap in (0, 50, 75, 100, 200):
    print(overlap, overlap_stats(5000, 500, overlap))

Step-by-step explanation.

Step = size - overlap = 500 - 75 = 425. To cover 5000 tokens, you need ceil((5000 - 75) / 425) ≈ 12 chunks (vs 10 chunks with zero overlap).
Total tokens stored = 12 × 500 = 6000. Doc has 5000 unique tokens. Redundant tokens = 1000, which is exactly 2 × (chunks - 1) × overlap = 2 × 11 × ... approximated by (n_chunks - 1) * overlap overhead.
Storage overhead at 15% overlap is ~20% extra tokens stored (and embedded, and indexed). That is the cost.
The benefit: every fact within 75 tokens of a chunk boundary now appears in two chunks, doubling its chance of being retrieved.
15% (~75 tokens on a 500-token chunk) is the empirical sweet spot — below 10% leaves obvious boundary gaps; above 25% pays cost without much extra recall.

Output.

overlap	n_chunks	redundant tokens	storage overhead
0	10	0	0%
50	11	500	10%
75	12	1000	20%
100	13	1500	30%
200	17	3500	70%

Rule of thumb. Start at 15% overlap (75 tokens on a 500-token chunk) — it is the empirically calibrated default. Drop to 0 only for atomic content (a row, a function, a Q+A pair) where there is no boundary to worry about. Past 25% you pay storage and ingest cost without commensurate recall gain.

Data engineering interview question on picking a chunking strategy

A senior interviewer might frame this as: "You inherit a RAG system with 100k mixed documents — long-form policy PDFs, support tickets, code samples, transcripts. The current chunker is a fixed 1000-token window with no overlap. Recall@5 is 0.62 on the golden set. Where do you start?"

Solution Using a per-content-type chunking strategy

def chunk_dispatch(doc: dict) -> list[dict]:
    """Pick the right chunker per content type — strategy is not one-size-fits-all."""
    content_type = doc["content_type"]
    if content_type == "policy_pdf":
        # Long-form, multi-topic → hierarchical (sentence child, paragraph parent)
        return parent_child_chunker(doc, child_size=120, parent_size=600, overlap=15)
    if content_type == "support_ticket":
        # Short, conversational → one chunk per ticket, no overlap
        return [{"text": doc["body"], "metadata": doc["metadata"]}]
    if content_type == "code":
        # AST-aware split at function / class boundaries
        return ast_chunker(doc, language=doc["language"])
    if content_type == "transcript":
        # Topic-shift aware → semantic chunking
        return semantic_chunker(doc, threshold=0.55)
    # Default for plain prose
    return recursive_chunker(doc, size=500, overlap=75)

Step-by-step trace.

content_type	strategy	starting params	expected recall@5 lift
policy_pdf	hierarchical	child 120 / parent 600	+0.15-0.20
support_ticket	atomic	one chunk per ticket	+0.05-0.10
code	AST split	per function / class	+0.10-0.15
transcript	semantic	threshold 0.55	+0.10-0.15
plain prose	recursive	500 / 75 overlap	+0.05 baseline

The trace highlights that a single strategy across all content types is the bug, not a missing feature. Switching from fixed-1000 to per-type strategies typically lifts recall@5 by 0.15-0.25 in aggregate — the single largest one-shot win in any RAG project.

Output:

Metric	Before (fixed 1000)	After (per-type)
recall@5	0.62	~0.83
MRR@10	0.41	~0.63
avg chunks per doc	5.2	11.7

Why this works — concept by concept:

Per-content-type dispatch — the chunk shape that maximises recall depends entirely on the content shape. Code wants AST boundaries; policy wants section boundaries; tickets are already atomic; transcripts shift topic. One chunker for all four is a contradiction.
Hierarchical for long-form — small children give tight retrieval; large parents give the LLM enough context. The standard winning shape for technical docs.
Semantic for shifting topics — costs more at ingest but pays back on multi-topic documents (transcripts, blogs). Skip on tight-scope docs.
AST for code — splitting mid-function loses the function signature or the body. AST-aware chunking pairs them — interviewers love this answer because it shows content-aware thinking.
Cost — fixed-window is O(doc_tokens); recursive adds ~1 pass; semantic adds one embed call per sentence; hierarchical doubles the storage (children + parents). The recall lift typically pays back within 1 week of token cost.

DE
Topic — data transformation
Data transformation problems (DE)

Practice →

4. Embeddings + storage — choosing models and shaping the index

`embeddings` decide what "similar" means in your index — and the metadata sidecar decides whether the right user is allowed to see it

The mental model in one line: an embedding model is a learned similarity function — it determines which chunks the retriever calls "close" — and a vector stores schema is the index plus the metadata sidecar that keeps that similarity safe to ship. Once you say "the embedding decides recall, the metadata decides safety," every RAG storage question fits into one of those two columns.

Embedding model selection — the four axes.

Quality (recall@k on MTEB). OpenAI text-embedding-3-large, Cohere embed-v3, and OSS models like BGE-large and e5-mistral lead the public benchmarks. The gap between top-tier hosted and top OSS narrowed sharply in 2025-2026.
Dimensions. 256-dim models are 6x cheaper to store and search than 1536-dim models, with single-digit-point recall trade-off. Most production teams ship 384-768 dim now.
Cost. Per-million-token embedding cost. Hosted models charge ~$0.02-0.13 per million tokens; self-hosted OSS is GPU-amortised.
Latency. Per-batch latency at ingest. Critical for the freshness path — a stalled embedder is the most common cause of rag freshness SLO misses.

The "same model on read and write" rule.

The vectors in the index were produced by model M; a query must be embedded by model M to be comparable. Different models live in different vector spaces — cosine similarity between them is noise.
The fix when you want to upgrade: do a full re-embed into a new collection (covered in section 5 under blue/green).
The pre-flight check: store embed_model_id in the metadata sidecar of every chunk, and assert it matches the query embedder at retrieval time. Cheap insurance against a silent bug.

Batch vs streaming embedding.

Batch. A nightly job re-embeds all changed chunks. Fine for low-freshness use cases (knowledge base of reference docs). Cheaper per-token.
Streaming. CDC → Kafka → embed worker → vector upsert. Sub-minute lag. Required for high-freshness use cases (ticket triage, live policy lookup).
Hybrid. Batch for bulk re-embeds; streaming for incremental change. Most production systems converge to this shape.

Metadata schema — the columns every chunk carries.

tenant_id — multi-tenant filter pushdown. Required.
source — origin URI for attribution and trust signals.
document_id — group children back to parent doc.
last_modified — for freshness SLO and recency filters.
acl_ids — list of permission tags intersected with the user's permissions.
content_type — prose, code, table, transcript — drives content-specific behaviour.
embed_model_id — embedding model version. The blue/green safety pin.
content_hash — SHA-256 of chunk text. Powers skip-unchanged and dedupe.

Hybrid retrieval — dense + BM25.

Dense alone. Great on semantic synonyms ("revenue" matches "income"). Misses on rare keywords ("RT-2718 error code") because rare tokens have weak embeddings.
BM25 alone. Great on rare keywords. Misses on synonyms. Sensitive to vocabulary mismatch.
Hybrid (RRF or weighted). Recovers both regimes. The 2026 default for production RAG. The typical weighted formula: 0.6 * dense_score + 0.4 * bm25_score after both are min-max normalised; the typical rank-based formula is Reciprocal Rank Fusion with k=60.
When weighted vs RRF. RRF when scores from the two systems are not directly comparable (different score scales); weighted when you have tuned weights from a held-out set.

Reranking — the precision multiplier.

Bi-encoder (ANN). The embedding model is a bi-encoder: query and doc are embedded separately and compared by cosine. Fast (millions of vectors per second) but loses precision because the comparison is just a dot product.
Cross-encoder (reranker). A second model that takes (query, doc_text) as a pair and outputs a relevance score. Far more expressive (full attention across query + doc) but slow.
The standard shape. ANN top-N → reranker → top-k. N is typically 50; k is typically 5. The reranker only fires on 50 candidates per query, so the wall-clock cost is bounded.
Models. Cohere rerank-v3, BGE-reranker, mxbai-rerank. Hosted models add ~50-100ms; self-hosted are 30-200ms depending on GPU and batch size.

Worked example — picking embedding dimensions: 256 vs 768 vs 1536

Detailed explanation. Higher-dimension embeddings can express more nuance but pay for it in storage (4 bytes per dim × N chunks), index size, and query latency. Most teams over-pay for dimensions; a calibrated dim choice is one of the highest-leverage decisions in a RAG project.

Question. A team has 10M chunks. Compare storage and query cost across 256, 768, and 1536 dimensions. Show the ANN recall trade-off.

Input. 10M chunks, all float32 vectors.

Code.

def storage_estimate(n_chunks: int, dims: int) -> dict:
    bytes_per_chunk = dims * 4   # float32
    raw_bytes = n_chunks * bytes_per_chunk
    return {
        "dims": dims,
        "bytes_per_chunk": bytes_per_chunk,
        "raw_storage_gb": raw_bytes / 1e9,
        "approx_search_relative_cost": dims,
    }


for dims in (256, 384, 768, 1024, 1536):
    print(storage_estimate(10_000_000, dims))

Step-by-step explanation.

10M chunks × 1536 dims × 4 bytes = 61.4 GB raw vectors. At 768 dims → 30.7 GB. At 256 dims → 10.2 GB. Six-fold storage difference between the extremes.
ANN search cost is roughly proportional to dims for the distance computation (HNSW edge comparisons scale linearly). A 1536-dim query is ~6x more compute per node visited than a 256-dim query.
Recall trade-off (from MTEB benchmarks): going from 1536 → 768 typically costs 0-3 points of recall@10. Going to 384 costs 3-6 points. Going to 256 costs 5-10 points but pays back 6x on storage and search.
Matryoshka embeddings (variable-dim truncation, where lower-dim prefixes are themselves usable embeddings) let you store at 1536 and search at 384 — a popular 2026 pattern.

Output.

dims	raw storage (10M)	rel. search cost	typical recall lost
256	10.2 GB	1.0×	5-10 pts
384	15.4 GB	1.5×	3-6 pts
768	30.7 GB	3.0×	0-3 pts
1024	40.9 GB	4.0×	0-2 pts
1536	61.4 GB	6.0×	reference

Rule of thumb. Default to 384-768 dims for production RAG; reserve 1536 for cases where the golden-set recall difference justifies the 6x storage and search cost. Matryoshka embeddings (store 1536, search 384) are the best of both worlds when the embedding model supports them.

Worked example — hybrid retrieval with Reciprocal Rank Fusion

Detailed explanation. Dense and sparse retrieval miss on orthogonal vocabulary regimes. Reciprocal Rank Fusion combines them by rank rather than score — no need to normalise scales, no per-system weight tuning. The 2026 default for hybrid search.

Question. Implement RRF that fuses a dense ANN result list and a BM25 result list into a single ranked output. Compare with a naive weighted-score fusion on a small example.

Input. Dense and sparse top-5 for query "refund window digital".

rank	dense_id	dense_score	sparse_id	sparse_score
1	A	0.92	C	18.4
2	B	0.88	A	17.1
3	C	0.71	D	14.0
4	D	0.65	B	12.5
5	E	0.62	F	9.8

Code.

def rrf(dense: list[str], sparse: list[str], k: int = 60) -> list[str]:
    scores: dict[str, float] = {}
    for rank, doc_id in enumerate(dense):
        scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank + 1)
    for rank, doc_id in enumerate(sparse):
        scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank + 1)
    return sorted(scores, key=lambda d: -scores[d])


def weighted(dense, sparse, w_dense=0.6, w_sparse=0.4):
    # Both scores must be min-max normalised first
    def norm(items, key):
        vals = [v for _, v in items]
        lo, hi = min(vals), max(vals)
        return {k_: (v - lo) / (hi - lo + 1e-9) for k_, v in items}

    d = norm(dense, "dense")
    s = norm(sparse, "sparse")
    ids = set(d) | set(s)
    return sorted(ids, key=lambda i: -(w_dense * d.get(i, 0) + w_sparse * s.get(i, 0)))


dense_ids = ["A", "B", "C", "D", "E"]
sparse_ids = ["C", "A", "D", "B", "F"]
print("RRF:", rrf(dense_ids, sparse_ids, k=60))

Step-by-step explanation.

RRF assigns each candidate a score of 1/(k + rank) for each list it appears in. Lower ranks get more weight. The score for a doc that appears in both lists is the sum of the two contributions.
For doc A: rank 1 in dense (score 1/61 = 0.0164), rank 2 in sparse (score 1/62 = 0.0161). Total = 0.0325.
For doc C: rank 3 in dense (1/63 = 0.0159), rank 1 in sparse (1/61 = 0.0164). Total = 0.0323.
A wins narrowly over C because it ranked higher in dense; both win over docs that appear in only one list because they get only one contribution.
RRF needs no score normalisation. Weighted score fusion requires normalising the dense cosine similarities and the BM25 scores to a common range and tuning the weights on a held-out set. RRF is robust to both.
k=60 is the value from the original RRF paper. Smaller k weights the top of each list more aggressively; larger k flattens the contribution. The default works well in practice.

Output (RRF top-5).

rank	doc	rrf_score
1	A	0.0325
2	C	0.0323
3	B	0.0318
4	D	0.0314
5	E	0.0164

Rule of thumb. Default to RRF for hybrid retrieval — it is robust, requires no tuning, and works whenever you can produce two ranked lists. Switch to weighted score fusion only if you have a held-out tuning set and the RRF result is leaving recall on the table.

Worked example — metadata pushdown vs post-filter

Detailed explanation. Multi-tenant RAG must restrict every retrieval to the calling tenant. Where the filter is applied matters enormously — pushdown into the ANN index is cheap; post-filter after ANN is catastrophic at scale.

Question. Show two implementations of a tenant-aware retrieve: one with metadata pushdown into the vector store, one with a post-filter on the application side. Quantify the cost difference.

Input. A 50M-chunk index where each tenant has ~50k chunks (1000 tenants).

Code.

# BROKEN — post-filter on the application side
def retrieve_postfilter(query: str, tenant_id: str, k: int = 5) -> list:
    qvec = embed_model.encode([query])[0]
    candidates = vector_store.search(vector=qvec, top_k=2000)  # huge over-fetch
    return [c for c in candidates if c.metadata["tenant_id"] == tenant_id][:k]

# CORRECT — push the filter down into the ANN search
def retrieve_pushdown(query: str, tenant_id: str, k: int = 5) -> list:
    qvec = embed_model.encode([query])[0]
    return vector_store.search(
        vector=qvec,
        top_k=k,
        filter={"tenant_id": tenant_id},
    )

Step-by-step explanation.

With 50M chunks and 1000 tenants, a tenant's chunks are 0.1% of the index. To get a top-5 after filtering, the post-filter approach must over-fetch by a huge factor — empirically 2000-10000 candidates to reliably surface 5 from the target tenant.
ANN search cost is O(log N) per query, but at large over-fetch the constants matter — fetching 10000 vs 5 candidates is roughly 100-1000x the wall-clock at scale.
Pushdown lets the ANN index restrict the search graph traversal to nodes that match the filter — modern vector stores (Pinecone, Qdrant, Weaviate, pgvector with vchord / pgvecto.rs) support this natively.
Worst case for post-filter: the user has a "needle in haystack" tenant where the top 2000 ANN candidates contain zero of the tenant's chunks → user gets empty results despite having matching chunks in the index.
Pushdown is also the only safe form — post-filter is a defence-in-depth layer at best, not the primary tenant boundary.

Output.

Approach	Avg fetched	P95 latency	tenant safety
post-filter	2000-10000	400-2000 ms	weak
pushdown	k (5)	15-50 ms	strong

Rule of thumb. Always push tenant and ACL filters down into the vector store; never rely on application-side post-filter. The performance difference is 1-2 orders of magnitude, and the safety boundary lives in exactly one place — the vector store query itself.

Worked example — cross-encoder reranker on top of hybrid

Detailed explanation. A cross-encoder reranker is the standard precision multiplier on top of hybrid retrieval. The shape is fixed: hybrid produces 50 candidates → reranker scores all 50 in one batch → top-5 are the prompt context. Adds ~50-100ms; lifts top-5 precision dramatically.

Question. Add a Cohere-style reranker to the hybrid pipeline. Show the batch call shape, the latency budget, and what happens when the reranker times out.

Input. A query plus 50 hybrid-fused candidate chunks.

Code.

def hybrid_then_rerank(query: str, tenant_id: str, k: int = 5) -> list:
    # 1) Dense + sparse → fuse → 50 candidates
    candidates = hybrid_search(query, tenant_id, top_k=50)
    texts = [text_store.get(c.id) for c in candidates]

    # 2) Cross-encoder batch — single call with all 50 pairs
    try:
        scores = reranker.score_batch(
            query=query, documents=texts, timeout_ms=120
        )
    except TimeoutError:
        # 3) Graceful degradation — return top-k from hybrid without rerank
        log.warn("reranker_timeout", query_hash=hash(query))
        metric("rag.rerank_timeout", 1)
        return candidates[:k]

    # 4) Sort by reranker score; return top-k
    ranked = sorted(zip(candidates, scores), key=lambda x: -x[1])
    return [c for c, _ in ranked[:k]]

Step-by-step explanation.

The reranker call is batched — all 50 (query, doc) pairs go in one HTTP request. Per-pair calls would multiply latency by ~50.
Latency budget: 120ms timeout. Cohere rerank-v3 typically lands at 60-100ms for 50 pairs; budget the timeout above the P99 to avoid spurious timeouts.
Graceful degradation: if the reranker times out, fall back to the hybrid top-k. Never fail the whole query — degrade to "less precise but still answered."
Metric rag.rerank_timeout is graphed and alerted on. A sustained timeout rate (≥5%) is the first sign the reranker is overloaded or the model API is unhealthy.
The reranker is the only online step that can be hot-swapped. Promote a new reranker behind a feature flag, run it shadow-style for a week against the golden set, then flip.

Output.

step	latency budget	typical actual
query embed	30 ms	10-25 ms
ANN search (filtered)	40 ms	15-30 ms
BM25 search	30 ms	10-20 ms
RRF fuse	5 ms	<2 ms
reranker batch (50)	120 ms	60-100 ms
prompt assemble	20 ms	5-15 ms
total	< 300 ms	100-200 ms

Rule of thumb. Budget the reranker on its own line. Cap candidate count at 50 (sometimes 100 for very high-stakes queries). Always wrap the call with a timeout and a graceful-degradation fallback to "hybrid without rerank" — never fail the user-facing query because the reranker hiccupped.

Data engineering interview question on storage schema and hybrid retrieval

A senior interviewer might frame this as: "You are designing the vector store schema for a multi-tenant retrieval augmented generation service. Walk me through the schema, why each column exists, and how the online retrieval query uses every column."

Solution Using a metadata-rich schema and pushdown filters

# Vector store row schema — every column has a job
SCHEMA = {
    "id": "text PRIMARY KEY",         # deterministic chunk_id
    "vector": "vector(768)",          # embedding
    "tenant_id": "text NOT NULL",     # multi-tenant pushdown
    "source": "text NOT NULL",        # attribution
    "document_id": "text NOT NULL",   # parent doc grouping
    "chunk_idx": "int NOT NULL",      # sibling lookup
    "content_type": "text",           # prose / code / table / transcript
    "last_modified": "timestamptz",   # freshness SLO + recency filter
    "acl_ids": "text[]",              # permission tags
    "embed_model_id": "text",         # blue/green safety pin
    "content_hash": "text",           # skip-unchanged + dedupe
}

# The retrieval query — every metadata column is exercised
def retrieve(query, tenant_id, user_acl, max_age_days):
    qvec = embed_model.encode([query])[0]
    return vector_store.search(
        vector=qvec,
        top_k=50,
        filter={
            "tenant_id": tenant_id,
            "acl_ids": {"$overlap": user_acl},
            "last_modified": {"$gte": days_ago(max_age_days)},
            "embed_model_id": embed_model.id,
        },
    )

Step-by-step trace.

Column	Used by
id	upsert idempotency, sidecar text lookup
vector	ANN cosine similarity
tenant_id	filter pushdown — multi-tenant safety
source	prompt attribution, trust signal
document_id	parent-chunk resolution, "show full doc"
chunk_idx	sibling lookup ("read the next chunk too")
content_type	content-aware rerank or rules
last_modified	freshness SLO, recency filter
acl_ids	permission pushdown (overlap match)
embed_model_id	blue/green safety — match read model
content_hash	skip-unchanged on reingest, dedupe

The trace highlights that every metadata column earns its keep — none are decorative. Drop tenant_id and you have a cross-tenant leakage bug; drop embed_model_id and you have a silent model-mismatch bug; drop acl_ids and you have an authz failure.

Output:

Decision	Rationale
768 dims	balance recall vs storage
pgvector or Qdrant	pushdown filter support
ACL as array	overlap-match against user's permissions
`content_hash` indexed	re-ingest skips unchanged chunks
`last_modified` indexed	freshness queries are common

Why this works — concept by concept:

Metadata pushdown is the single biggest perf lever — restricting the ANN search to the tenant's slice changes the constants by 1-2 orders of magnitude at scale.
embed_model_id is the silent-bug pin — the most subtle RAG failure mode is read and write paths using different embedding models; the metadata column makes the mismatch detectable in seconds.
content_hash is the skip-unchanged pin — letting the ingest job re-embed only changed chunks turns nightly reindex from a multi-hour batch into a sub-minute incremental.
ACL overlap match — modern vector stores support array-overlap as a filter primitive. Permission pushdown then runs at the same speed as tenant pushdown.
Cost — schema overhead is ~200 bytes per chunk for metadata; the index on tenant_id and last_modified is the difference between a 200ms and a 20ms query at 50M chunks. Every byte earns its keep.

DE
Topic — indexing
Indexing problems (DE)

Practice →

5. Freshness, reindex, and ACLs

`rag freshness` is an SLO, not a feature — and the reindex playbook is what keeps it honest

The mental model in one line: freshness is the P95 lag between a source document being updated and the corresponding chunk being retrievable — and a production RAG pipeline either ships an explicit freshness SLO with telemetry or silently drifts into "the bot still cites the old policy." Once you state the SLO, every reindex strategy maps to "how do I keep this lag under X."

The freshness SLO.

Definition. P95 source-to-retrieval lag — the time between a source-of-truth update (Confluence save, Postgres commit, S3 put) and the moment the new chunk is retrievable in the vector store.
Typical thresholds. Knowledge base reference docs: 30-60 minutes. Live policy / pricing lookup: 1-5 minutes. Real-time support context: <30 seconds.
Why SLO not feature. A feature is "we reindex nightly." An SLO is "we promise P95 < 5 minutes and we will page if it breaks." Only the SLO survives contact with stakeholders.
Telemetry. now - max(last_modified) per source for the embed worker; query-side now - retrieved_chunk.last_modified. Graph both; alert on either.

Incremental reindex via CDC.

Source CDC. Postgres logical replication (Debezium), Confluence webhooks, Notion webhooks, S3 event notifications. The source emits "what changed" events into a topic (Kafka, Kinesis).
Embed worker. Consumes the topic, fetches the changed document, re-chunks, re-embeds only changed chunks (skip-unchanged via content_hash), upserts vectors and metadata.
Upsert semantics. Deterministic chunk_ids mean upsert is idempotent — the same change replayed produces the same final state.
Backfill mode. A new source connector starts in "full backfill" (read every doc once) and graduates to "CDC tail" (read changes only).

Tombstoning deletes.

The problem. A document deleted in the source must vanish from retrieval — but most teams only mark the source as deleted and forget the vector store.
The fix. When CDC emits a "delete" event, the embed worker either hard-deletes the chunk_ids for that document, or soft-deletes them by setting is_active=false in the metadata sidecar (and filters is_active=true on every retrieve).
Why soft-delete + nightly purge. Soft-delete is reversible (whoops, didn't mean to nuke that doc); nightly purge sweeps hard-deletes after a grace window.
Test. Every nightly eval golden set includes a "deleted doc must not appear" canary. If it appears, page.

Embedding model upgrade = full re-embed.

The rule. The vectors in the index were produced by model M. Switching to model M' invalidates every vector; queries embedded by M' do not match vectors from M.
The pattern. Blue/green collections. Create collection_v2 with the new model, re-embed everything, dual-write during transition, cut over reads atomically, then drop collection_v1.
Cost. A full re-embed of 100M chunks at $0.02/M tokens × 500 tokens/chunk ≈ $1000 in API spend, plus the worker compute. Budget for it before promising the upgrade.
Validation. Run the golden-set eval against v2 before cutover. If recall@5 is not meaningfully better, the upgrade is not worth it.

Per-tenant ACL pushdown.

The shape. Each chunk carries acl_ids: ["public", "support", "engineering"]. Each user has user_acl: ["public", "support"]. The retrieval filter is acl_ids OVERLAP user_acl pushed down into the vector store.
Why pushdown not post-filter. Same reason as tenant_id — post-filter forces huge over-fetch; pushdown is one extra predicate in the ANN traversal.
Sourcing the ACL list. Materialised at ingest time from the source's ACL system (Confluence space permissions, Notion page permissions, S3 bucket policy tags). Refreshed via CDC when permissions change.
Permission change is itself a CDC event. A user joins a new team → their user_acl updates → next query sees the wider chunk set. No reindex needed.

Quality regression detection.

Golden set drift. Recall@5 / MRR@10 trended over time. A sudden drop after an ingest job means a recent change broke retrieval.
Top-cause attribution. Diff the recent ingest config (chunker change, embedder change, schema change) against the last "green" deploy.
Auto-rollback. Some teams configure auto-rollback on a sustained ≥5pt recall drop — return the index to its previous state until a human investigates.

Common interview probes on freshness and reindex.

"What is your freshness SLO and how do you measure it?" — name a P95 number, name a telemetry source, name the alert.
"How do you handle deletes?" — soft-delete in metadata + filter on retrieve + nightly hard-purge sweep.
"How do you upgrade an embedding model in production?" — blue/green collections, dual-write, golden-set validation, atomic cutover.
"How do you enforce per-user permissions?" — acl_ids array column on every chunk; overlap filter pushed down into the vector store; permission changes flow through the same CDC stream.

Worked example — CDC-driven incremental reindex worker

Detailed explanation. A streaming embed worker consumes a CDC topic, fetches the changed document, re-chunks, re-embeds only changed chunks (skip-unchanged via content_hash), and upserts. Sub-minute lag at steady state.

Question. Sketch the embed worker that consumes a source.changes Kafka topic and upserts vectors with sub-5-minute P95 lag.

Input. A stream of CDC events.

event	doc_id	op	last_modified
1	PAGE-42	upsert	2026-06-15T10:00:00Z
2	PAGE-09	delete	2026-06-15T10:00:05Z
3	PAGE-42	upsert	2026-06-15T10:00:30Z

Code.

def embed_worker_loop():
    consumer = kafka.Consumer("source.changes", group_id="rag-embed-worker")
    for event in consumer:
        try:
            if event["op"] == "delete":
                handle_delete(event["doc_id"])
            else:
                handle_upsert(event["doc_id"])
            consumer.commit(event)
            metric("rag.source_lag_seconds", now() - event["source_ts"])
        except Exception as e:
            log.error("embed_worker_error", error=str(e), event=event)
            consumer.send_to_dlq(event)


def handle_upsert(doc_id: str) -> None:
    doc = source.fetch(doc_id)                  # idempotent fetch by id
    new_chunks = chunk_dispatch(doc)            # per-content-type chunker
    existing = vector_store.scan(filter={"document_id": doc_id})
    existing_hashes = {c.metadata["content_hash"] for c in existing}

    to_upsert = []
    for new in new_chunks:
        h = hashlib.sha256(new["text"].encode()).hexdigest()
        if h in existing_hashes:
            continue                            # skip unchanged
        new["metadata"]["content_hash"] = h
        to_upsert.append(new)

    if to_upsert:
        vectors = embed_model.encode([c["text"] for c in to_upsert])
        for c, v in zip(to_upsert, vectors):
            c["vector"] = v
        vector_store.upsert(to_upsert)


def handle_delete(doc_id: str) -> None:
    # Soft-delete — mark as inactive; nightly purge sweeps hard deletes
    vector_store.update_metadata(
        filter={"document_id": doc_id},
        patch={"is_active": False, "deleted_at": now()},
    )

Step-by-step explanation.

The worker consumes one event at a time, processes it, and commits the offset only after the upsert succeeds. At-least-once delivery semantics are fine because chunk_ids are deterministic and upsert is idempotent.
On an upsert event, the worker fetches the full document from the source (CDC events typically carry only IDs, not bodies), re-chunks, and computes the content hash of each chunk.
The skip-unchanged optimisation: chunks whose hash matches a previously stored one are not re-embedded. For a single-paragraph edit on a 50-paragraph doc, only 1-2 chunks change — the savings compound.
On a delete event, the worker soft-deletes by setting is_active=false on every chunk for that doc. The retrieval path filters is_active=true, so deleted chunks are invisible immediately.
A nightly purge job hard-deletes rows where deleted_at < now() - 7 days, freeing index space.
metric("rag.source_lag_seconds", now() - event.source_ts) is the per-event freshness measurement. Aggregated as P95 across an hour, this is the freshness SLO.
Errors go to a DLQ (dead-letter queue) so a single bad doc cannot block the whole stream. The DLQ has its own alert.

Output.

event	action	chunks re-embedded	lag
1 (PAGE-42 upsert)	fetch + chunk + embed	3 of 12 changed	~45 s
2 (PAGE-09 delete)	soft-delete	0 (metadata only)	~5 s
3 (PAGE-42 upsert)	fetch + chunk + embed	1 of 12 changed	~30 s

Rule of thumb. The skip-unchanged + soft-delete + DLQ trio is the production shape of a CDC embed worker. Without them you re-embed everything on every change (cost explodes), you hard-delete on every event (no recovery), or you crash on the first malformed doc (whole stream stalls).

Worked example — blue/green embedding model upgrade

Detailed explanation. A model upgrade invalidates every vector in the index. The safe pattern is blue/green: build a parallel v2 collection with the new model, dual-write incoming CDC events into both, validate against the golden set, cut over reads atomically, drop the old collection.

Question. Sketch the blue/green upgrade flow from embedding model e5-large-v1 to e5-large-v2 for a 50M-chunk index.

Input. Existing collection_v1 with all chunks embedded by e5-large-v1.

Code.

# Phase 1 — Create v2 collection and full re-embed (background)
def backfill_v2():
    create_collection("collection_v2", dims=v2_model.dims)
    for batch in scan_chunks("collection_v1", batch_size=1000):
        texts = [text_store.get(c.id) for c in batch]
        vectors = v2_model.encode(texts)
        rows = []
        for c, v in zip(batch, vectors):
            rows.append({
                **c.as_dict(),
                "vector": v,
                "metadata": {**c.metadata, "embed_model_id": "e5-large-v2"},
            })
        upsert("collection_v2", rows)


# Phase 2 — Dual-write incoming CDC into both collections
def handle_upsert_dualwrite(doc_id: str):
    doc = source.fetch(doc_id)
    new_chunks = chunk_dispatch(doc)
    v1_vecs = v1_model.encode([c["text"] for c in new_chunks])
    v2_vecs = v2_model.encode([c["text"] for c in new_chunks])
    for c, v1, v2 in zip(new_chunks, v1_vecs, v2_vecs):
        upsert("collection_v1", [{**c, "vector": v1}])
        upsert("collection_v2", [{**c, "vector": v2}])


# Phase 3 — Validate v2 against the golden set
def validate_v2() -> bool:
    v1_metrics = score_pipeline(golden, collection="collection_v1")
    v2_metrics = score_pipeline(golden, collection="collection_v2")
    return v2_metrics["recall_at_k"] >= v1_metrics["recall_at_k"] - 0.005


# Phase 4 — Atomic read cutover via feature flag
def retrieve(query, tenant_id):
    collection = "collection_v2" if FLAGS["read_v2"] else "collection_v1"
    model = v2_model if FLAGS["read_v2"] else v1_model
    qvec = model.encode([query])[0]
    return search(collection, qvec, filter={"tenant_id": tenant_id})


# Phase 5 — Drop v1 after a 7-day soak
def cleanup_v1():
    if FLAGS["read_v2"] and days_since_cutover() >= 7:
        drop_collection("collection_v1")

Step-by-step explanation.

Phase 1 (backfill): a background job iterates every chunk in v1, re-embeds with the new model, writes into v2. Heavy compute and API spend; runs over hours or days for a large index.
Phase 2 (dual-write): from the moment the backfill starts, every CDC event is written into both collections. This keeps v2 in sync with new edits while the backfill is still running.
Phase 3 (validate): run the golden-set eval against both collections. v2 must match or beat v1 on recall@5 / MRR@10. A regression is a stop sign.
Phase 4 (cutover): flip a feature flag. The read path switches from v1 to v2 and from v1_model to v2_model. The flip is instant and reversible.
Phase 5 (cleanup): after a 7-day soak (during which you can flip back if something breaks), drop v1 and reclaim storage.
Throughout: dual-write costs 2x the embed compute. Plan the rollout so the cost window is bounded — typically a few days, not weeks.

Output.

phase	duration	risk
backfill	hours-days	high cost, no user impact
dual-write	duration of rollout	2x embed cost
validate	1-3 days	catches regressions
cutover	instant	reversible via flag
cleanup	after 7d soak	irreversible

Rule of thumb. Never hot-swap an embedding model. Always go blue/green: backfill → dual-write → validate → atomic cutover → soak → drop old. The 2x embed cost during rollout is the price you pay for zero downtime and instant rollback.

Worked example — per-tenant ACL pushdown end-to-end

Detailed explanation. Multi-tenant RAG with per-document permissions means every chunk carries an acl_ids array and every query carries the user's user_acl. The retrieval predicate is acl_ids OVERLAP user_acl, pushed down into the vector store.

Question. Show the ingest-side ACL materialisation and the retrieval-side overlap filter for a Confluence-backed RAG with space-level permissions.

Input. A Confluence page with space ACLs.

field	value
page_id	"PAGE-42"
space_id	"ENG"
space_acl	["eng-team", "leadership"]

Code.

# Ingest — materialise ACL on every chunk
def ingest_with_acl(page):
    acl_ids = space_acl_resolver.resolve(page["space_id"])  # ["eng-team", "leadership"]
    chunks = chunk_dispatch(page)
    for c in chunks:
        c["metadata"]["acl_ids"] = acl_ids
        c["metadata"]["space_id"] = page["space_id"]
    vectors = embed_model.encode([c["text"] for c in chunks])
    for c, v in zip(chunks, vectors):
        c["vector"] = v
    vector_store.upsert(chunks)


# Retrieve — overlap filter pushed down
def retrieve_for_user(query, user_id, tenant_id):
    user_acl = user_directory.get_acl(user_id)  # e.g. ["eng-team", "all-hands"]
    qvec = embed_model.encode([query])[0]
    return vector_store.search(
        vector=qvec,
        top_k=50,
        filter={
            "tenant_id": tenant_id,
            "acl_ids": {"$overlap": user_acl},   # array overlap
            "is_active": True,
        },
    )


# Permission change — same CDC stream
def on_user_acl_change(user_id):
    # No reindex needed — next query picks up the new ACL
    user_directory.invalidate(user_id)

def on_space_acl_change(space_id):
    # Re-materialise ACL on every chunk in this space
    new_acl = space_acl_resolver.resolve(space_id)
    vector_store.update_metadata(
        filter={"space_id": space_id},
        patch={"acl_ids": new_acl},
    )

Step-by-step explanation.

At ingest, the chunker gets the space's ACL list and writes it into every chunk's metadata. The space ID is also stored so the chunks can be updated atomically if the space ACL changes.
The retrieval query pushes acl_ids OVERLAP user_acl down into the vector store. The user sees only chunks whose ACL intersects their permission set.
User permission change (joins a new team) does not require reindex — the user's user_acl updates in the directory, next query reflects it.
Space permission change (a previously-private space becomes shared with another team) does require updating every chunk in that space — but the update is a metadata patch, not a re-embed. Sub-second on most vector stores.
The overlap filter is one extra predicate in the ANN search, so the perf cost is negligible compared to the tenant filter.
Combined with tenant_id and is_active, the retrieval contract is "tenant-isolated, ACL-filtered, soft-delete-aware." Three predicates, all pushed down, all enforced in the vector store itself.

Output.

user	user_acl	accessible chunks
alice (eng)	["eng-team", "all-hands"]	PAGE-42 (eng-team) + public
bob (sales)	["sales", "all-hands"]	not PAGE-42
ceo	["leadership", "all-hands"]	PAGE-42 (leadership) + public

Rule of thumb. ACL pushdown belongs in the same query as the tenant filter — both filters live or die together in the vector store. Application-side post-filter for permissions is an authz time bomb; never ship it as the primary boundary.

Data engineering interview question on freshness and tombstoning

A senior interviewer might frame this as: "Stakeholder reports the bot is still citing a deleted policy doc. How do you design the pipeline so this cannot happen, and what is the SLO you commit to?"

Solution Using soft-delete + freshness SLO + golden-set canary

# 1) Soft-delete on CDC delete events
def handle_delete(doc_id):
    vector_store.update_metadata(
        filter={"document_id": doc_id},
        patch={"is_active": False, "deleted_at": now()},
    )

# 2) Retrieve filters out inactive chunks
def retrieve(query, tenant_id):
    return vector_store.search(
        vector=embed_model.encode([query])[0],
        top_k=50,
        filter={"tenant_id": tenant_id, "is_active": True},
    )

# 3) Nightly purge sweeps hard deletes
def nightly_purge():
    vector_store.delete(filter={
        "is_active": False,
        "deleted_at": {"$lt": days_ago(7)},
    })

# 4) Golden-set canary catches regressions
DELETED_DOC_CANARY = {
    "question": "What was the old refund window?",
    "must_not_match_source": "confluence/PAGE-deleted-99",
}

def canary_check():
    hits = retrieve(DELETED_DOC_CANARY["question"], tenant_id="default")
    leaked = any(h.metadata["source"] == DELETED_DOC_CANARY["must_not_match_source"]
                 for h in hits)
    if leaked:
        page_oncall("deleted_doc_leak", source=DELETED_DOC_CANARY)

Step-by-step trace.

Step	What it does	SLO
1 soft-delete	metadata `is_active=false` on CDC delete	P95 < 30 s
2 retrieve filter	only `is_active=true` chunks returned	always
3 nightly purge	hard-delete after 7-day soak	nightly
4 canary	golden-set probe for known-deleted docs	every nightly eval

The trace highlights that deletion safety is the conjunction of four steps, not any one of them — the soft-delete handles the instant invisibility, the retrieve filter enforces it, the nightly purge reclaims storage, and the canary catches the day someone breaks step 2.

Output:

Failure mode	Catch step
CDC delete event missed	step 4 (canary in next eval)
Soft-delete not yet propagated	step 1 (30s SLO)
Retrieve filter dropped	step 4 (canary fires)
Hard purge premature	7-day soak window

Why this works — concept by concept:

Soft-delete first, hard-delete later — gives a reversible window. Whoops-deletes recover in seconds; intentional deletes flush after 7 days.
Retrieve-side filter on is_active — the safety boundary lives in the query, not in the deletion handler. Even if the delete event is replayed or lost, the next query still respects is_active.
Golden-set canary — a known-deleted doc in the golden set is a continuous integration test for deletion. If the bot ever cites it, the eval fails before the user sees it.
Freshness SLO ties the loop — the P95 source-to-retrieval lag SLO covers both adds and deletes; one number, alerted on, owned by DE.
Cost — soft-delete is one metadata write per deleted chunk; canary is one extra row in the golden set; purge is a nightly bulk delete. All bounded; none cost more than a few minutes a day.

DE
Topic — data validation
Data validation problems (DE)

Practice →

Cheat sheet — RAG pipeline recipes

Chunk size starter. 500 tokens with 75-token overlap (15%). Tune against the golden set before changing.
Per-content-type strategy. Prose → recursive; code → AST; tables → atomic; transcripts → semantic; long-form policy → hierarchical (sentence child, paragraph parent).
Hybrid score fusion. Reciprocal Rank Fusion (RRF) with k=60 — robust, no tuning. Weighted 0.6 * dense + 0.4 * BM25 only if you have a held-out tuning set.
Metadata filter on retrieve. WHERE tenant_id = $1 AND acl_ids OVERLAP $2 AND is_active = true — pushed down into the vector store, never post-filter.
Reranker shape. Top-50 ANN candidates → cross-encoder batch → top-5 prompt context. Wrap with 120ms timeout and graceful degradation.
Embed model rule. Store embed_model_id in every chunk's metadata; assert it matches the query embedder at retrieval time. Catches the silent model-mismatch bug.
Freshness pipeline. Source CDC → Kafka → embed worker → vector upsert. P95 source-to-retrieval lag SLO < 5 minutes (tune to use case).
Skip-unchanged. Hash chunk text with SHA-256; store in metadata; re-embed only when hash changes. Turns nightly reindex into sub-minute incremental.
Tombstone delete. Soft-delete via is_active=false in metadata; filter on retrieve; nightly purge sweep hard-deletes after 7 days.
Embedding model upgrade. Blue/green collections — backfill → dual-write → golden-set validate → atomic cutover → 7-day soak → drop old. Never hot-swap.
ACL pushdown. acl_ids array on every chunk; acl_ids OVERLAP user_acl filter pushed down. Permission changes update the user directory, not the chunks (except space-level changes).
Golden set. 200-2000 (question, expected_chunk_id) triples maintained by SMEs. Nightly recall@5 and MRR@10 with alerts on a sustained 5pt drop.
Deletion canary. A known-deleted doc in the golden set whose source must not appear in retrieval. Continuous integration for deletion.
DLQ on the embed worker. Errors do not block the stream; a malformed doc goes to a DLQ with its own alert.
Telemetry. Source lag, embed queue depth, ANN P95, reranker P95, fallback ratio, recall@5. Six numbers, six graphs.

Frequently asked questions

What chunk size should I start with for a new RAG pipeline?

Start at 500 tokens with 75-token overlap (15%). It is the empirically calibrated default that lands inside most embedding model context windows (most are 512 tokens) and leaves enough room for the LLM to receive 5-10 chunks per prompt. Tune from there against your golden set — long-form policy docs often benefit from hierarchical chunking (120-token children, 600-token parents); transcripts benefit from semantic chunking with similarity threshold 0.55; code should be split by AST function/class boundaries with no overlap. The cheat-sheet recipe to memorise: "500/75 default, per-content-type adapters, golden-set tuning."

Do I need hybrid search or is dense retrieval enough?

You almost always need hybrid search in production. Pure dense retrieval misses on rare keywords (error codes, product SKUs, proper nouns the embedder has never seen), which is where 10-15 points of recall@5 hide. Pure BM25 misses on semantic synonyms ("revenue" vs "income"). Fusing them with Reciprocal Rank Fusion (RRF, k=60) recovers both regimes without per-system weight tuning. Skip hybrid only if (a) your content is all natural-language prose with no rare-vocab terms and (b) your golden-set recall is already at the threshold you need. In every other case, hybrid is the 2026 default for hybrid search.

How do I keep my RAG index fresh?

State a freshness SLO (e.g. P95 source-to-retrieval lag < 5 minutes), implement a CDC pipeline (Postgres logical replication, Confluence webhooks, Notion webhooks, S3 events) that emits change events into Kafka, and run an embed worker that consumes the stream, re-chunks the changed document, and upserts the new vectors with deterministic chunk_ids. Use content_hash to skip-unchanged chunks so a one-paragraph edit only re-embeds one chunk. For deletes, soft-delete via is_active=false in metadata and filter on retrieve; nightly purge sweeps hard-deletes after a 7-day soak. Graph now - max(last_modified) per source as the SLO telemetry; alert on breaches.

When do I need a reranker?

You need a reranker as soon as top-5 precision matters — which in practice is "as soon as your bot ships to real users." The standard shape is hybrid retrieves 50 candidates, the cross-encoder reranks all 50 in one batch, top-5 go into the prompt. Reranker adds 50-100ms; lifts recall@1 and MRR significantly because cross-encoders apply full attention across the query and the chunk text, not just a dot product. Cohere rerank-v3 and BGE-reranker are the common hosted/OSS picks. Always wrap the call with a timeout and a graceful-degradation fallback to "hybrid top-k without rerank" — never fail the user-facing query because the reranker hiccupped.

How do I enforce per-user permissions in RAG?

Store an acl_ids array on every chunk at ingest time (materialised from the source's permission system — Confluence space ACL, Notion page permissions, S3 bucket tags), and store each user's user_acl in a directory service. At retrieval time, push a acl_ids OVERLAP user_acl predicate down into the vector store alongside the tenant_id filter. Never apply ACL as an application-side post-filter — the perf cost is huge (massive over-fetch needed) and the safety story is weaker because the boundary lives in two places. User permission changes update the directory only (no reindex needed); space-level permission changes patch the acl_ids metadata on every chunk in that space (sub-second on most vector stores).

How do I evaluate RAG quality before shipping?

Build a golden set of 200-2000 (question, expected_chunk_id, expected_answer) triples with subject-matter experts. Run a nightly eval that fires every question through the live retrieval pipeline (same embed model, same filter, same rerank) and scores recall@5 (was the gold chunk in the top-5?) and MRR@10 (1/rank, averaged). Threshold recall@5 ≥ 0.85 as a typical production gate; alert on a sustained 5-point drop. Add a "deleted doc canary" to the golden set so deletion regressions surface. For online quality, sample live queries and score them with an LLM-as-judge against a rubric — useful for drift detection but not as ground truth. Without a golden set, every RAG change is a vibe-based decision.

Practice on PipeCode

Drill the ETL pipeline practice library → for the ingest, chunk, and embed stages of a RAG pipeline.
Rehearse on streaming problems → when the interviewer wants CDC-driven freshness pipelines.
Sharpen data transformation drills → for the chunker and metadata sidecar shaping work.
Layer the indexing library → for the metadata pushdown and ANN filter patterns.
Stack the data validation library → for the golden-set eval harness and deletion canary patterns.
Cover the real-time analytics library → for the freshness SLO and end-to-end latency patterns.
For the broader DE surface, read top data engineering interview questions →.
Stack the prerequisites with the only 5 skills you need to become a data engineer →.
Sharpen the foundations with the ETL system design course →.
For pipeline orchestration craft, work through Apache Spark internals for DE interviews →.

Pipecode.ai is Leetcode for Data Engineering — every RAG pipeline recipe above ships with hands-on practice rooms where you write the CDC consumer, the chunker, the hybrid retrieval query, and the golden-set eval harness against real graded inputs. PipeCode pairs every reading with 450+ DE-focused problems and a real-time scoring engine, so you never have to wonder whether your `rag data pipeline` will behave the same in production as it did on the whiteboard.

Practice ETL pipelines now →
Streaming drills →

Vector Databases for Data Engineers: Pinecone vs Weaviate vs Qdrant vs pgvector

Gowtham Potureddi — Tue, 16 Jun 2026 13:07:48 +0000

vector database is the phrase data engineers hear in every roadmap meeting now that retrieval-augmented generation has moved from a hackathon demo to a production line item — and it is also the phrase that hides the most expensive design decisions of the last five years. The choice of store, index, sharding scheme, and embedding-model upgrade path determines whether a retrieval-heavy product responds in 40 milliseconds at 100 million vectors or chokes a single replica into a one-second tail at 10 million.

This guide is the comparison you wished existed the first time a product manager asked you "should we just use pgvector?" and the answer was longer than a Slack reply. It walks the three real choices — what a vector database actually does, where it sits in the platform, and the four-vendor matrix (pinecone, weaviate, qdrant, pgvector) — then drops into the index-type ladder (hnsw, ivfflat, scalar / product quantization, DiskANN) and the ops surface (memory sizing, multi-tenancy, drift, reindex). Each section pairs a teaching block with a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the database design practice library →, rehearse on indexing problems →, and stack the storage-design muscles with system design drills →.

On this page

What a vector database actually is (and what it isn't)
The vector DB role in a data platform
Pinecone vs Weaviate vs Qdrant vs pgvector — vendor comparison
Index types — HNSW, IVFFlat, quantization, DiskANN
Ops, cost, and failure modes
Cheat sheet — vector DB recipes
Frequently asked questions
Practice on PipeCode

1. What a vector database actually is (and what it isn't)

A vector database stores high-dimensional embeddings and supports approximate nearest neighbour search — and that is its entire job

The one-sentence invariant: a vector database is a specialised store whose primary access pattern is "find the K rows whose embedding vectors are closest to this query vector," using an approximate-nearest-neighbour index that trades a few percent of recall for orders-of-magnitude speedup over exact KNN. Once you internalise that one sentence, the four vendor matrices, the index-type alphabet soup, and the "do I need this?" decision collapse into a sequence of clean engineering trade-offs.

The two core workloads.

Pure similarity search. Given a query vector, return the top-K nearest vectors by cosine, dot product, or L2 distance. The classic retrieval-augmented generation lookup: "embed the user question, find the K closest document chunks, send them to the LLM."
Metadata-filtered retrieval. Return the top-K nearest vectors that also satisfy a structured filter — tenant_id = 7 AND lang = 'en' AND published_at > '2026-01-01'. The filter must compose with the ANN index without destroying recall; how a vendor implements this is the single biggest performance differentiator.

Why exact KNN does not scale.

Exact K-nearest-neighbour search on N vectors of dimension d costs O(N · d) per query — every vector compared, every dimension touched. At N = 10M and d = 1536 (OpenAI text-embedding-3-small), one query is ~15 billion floating-point operations. A single CPU core takes seconds.
Approximate nearest neighbour (ANN) indexes — HNSW, IVFFlat, DiskANN — trade a small recall loss (usually 95–99 percent vs the brute-force baseline) for sub-linear query cost, often O(log N) or sub-linear with quantization. The same query on the same data drops to single-digit milliseconds.
The interview-grade statement: "Exact KNN is O(N · d); ANN is sub-linear at the cost of a few percent recall. In production you almost always pick ANN."

Where embeddings come from (out of scope) vs how they are indexed (in scope).

Out of scope for this post. The embedding producer — the encoder model (text-embedding-3-small, bge-large, e5-mistral), the batching service, the chunking strategy, the backfill harness. These belong in a separate "embedding pipeline" doc.
In scope for this post. The store that receives those vectors, indexes them with HNSW / IVFFlat / DiskANN, filters by structured metadata, replicates them for read scale-out, and serves them back to the application at p99 latency under the SLO.

What a vector database is NOT.

It is not a replacement for an OLTP database. Vector DBs are write-heavy at ingestion time and read-heavy at query time, but they do not provide ACID transactions across multiple keys, foreign-key constraints, or complex SQL aggregation. Your orders, users, payments tables stay in Postgres / MySQL.
It is not a replacement for an analytics warehouse. A vector store cannot run GROUP BY user_id over a billion rows the way Snowflake or BigQuery can. The query model is "top-K by similarity," not "aggregate by dimension."
It is not a search engine on its own. Best-in-class retrieval blends BM25 keyword scoring (Elasticsearch / OpenSearch / Tantivy) with vector similarity. A vector DB handles the dense half; the keyword half lives elsewhere or in a sibling index.

The 2026 reality.

pgvector is good enough up to ~10M vectors on a single Postgres replica, and that covers the long tail of products. The "do I even need a dedicated vector DB?" answer is usually "not yet."
Pinecone, Weaviate, Qdrant are the dedicated serverless / OSS leaders. Each targets a different operational profile — fully managed, modular OSS, low-latency Rust.
HNSW is the default low-latency index across every vendor. IVFFlat is the cheap-memory option. DiskANN is the "I need 100 million vectors on commodity SSD" option.
Hybrid retrieval (vector + BM25 + metadata filter) is now the assumed baseline; pure vector search alone underperforms keyword search on exact-match queries by 10–30 percent.

Worked example — the recall-vs-latency knob on HNSW

Detailed explanation. Most engineers meet ANN parameters by tweaking them blind. The clearest mental model is to think of HNSW's ef_search as a "how many candidates do you want to inspect before returning K?" knob: high ef_search gives high recall and high latency; low ef_search gives low recall and low latency. The sweet spot is workload-dependent — but the curve has a well-known shape.

Question. Given 5 million 768-dimensional embeddings indexed with HNSW (M=16, ef_construction=200), describe what happens to recall and p99 latency as you sweep ef_search from 16 to 256. Pick a value for a customer-facing search box with a 100 ms SLO and a "recall must be ≥ 95 percent" requirement.

Input.

ef_search	observed recall@10	observed p99 latency
16	81%	4 ms
32	89%	7 ms
64	95%	12 ms
128	98%	22 ms
256	99.3%	45 ms

Code.

# Sweep ef_search and measure recall + p99 against a brute-force baseline.
from time import perf_counter
import numpy as np
import hnswlib

def benchmark(index, queries, gold, ef_values):
    rows = []
    for ef in ef_values:
        index.set_ef(ef)
        latencies = []
        hits = 0
        for q, truth in zip(queries, gold):
            t0 = perf_counter()
            ids, _ = index.knn_query(q, k=10)
            latencies.append((perf_counter() - t0) * 1000)
            hits += len(set(ids[0]) & set(truth)) / 10
        p99 = np.percentile(latencies, 99)
        recall = hits / len(queries)
        rows.append((ef, recall, p99))
    return rows

Step-by-step explanation.

HNSW builds a small-world graph at index time. M=16 controls the average out-degree per node; ef_construction=200 controls how many candidates are considered when inserting a new node.
At query time, ef_search controls how many candidates the search keeps in its priority queue. The search greedily explores neighbours of the closest known candidate, expanding until the candidate set has fewer improvements than ef_search allows.
With ef_search=16, the search stops quickly — it finds plausible neighbours but misses distant-but-close ones. Recall is 81 percent — every fifth result is wrong.
With ef_search=64, the search visits enough candidates to land on the true top-10 with 95 percent probability. Latency stays under the 100 ms SLO with a comfortable margin.
Pushing ef_search to 256 pays a 4× latency cost for a 4-point recall gain. The marginal value is poor at this scale — the model error (the embedding may not perfectly represent semantic similarity) already dwarfs the 4-point ANN error.

Output.

Setting	Recall@10	p99 latency	Meets SLO?
ef_search=16	81%	4 ms	recall fails
ef_search=32	89%	7 ms	recall fails
ef_search=64	95%	12 ms	yes
ef_search=128	98%	22 ms	yes (overkill)
ef_search=256	99.3%	45 ms	yes (overkill)

Rule of thumb. Sweep ef_search on your own data and your own embedding model — the curve shifts with both. Pick the smallest ef_search that crosses your recall floor, then leave 30–50 percent latency headroom for traffic spikes.

Worked example — exact KNN vs ANN at 10 million vectors

Detailed explanation. The single most convincing way to internalise why ANN matters is to compute the brute-force cost in your head, then compare it with the ANN cost. The asymmetry is so large that the "do I need an ANN index?" question has only one answer above a few hundred thousand vectors.

Question. Given 10 million 1536-dimensional vectors, estimate (a) the query cost of exact KNN on a single CPU core, (b) the query cost of HNSW on the same hardware, and (c) the recall trade-off. Show why exact KNN is not a serious option above 1 million vectors.

Input.

Quantity	Value
N (vectors)	10,000,000
d (dimension)	1,536
Single-core float ops per second	~10 billion

Code.

# Back-of-envelope: exact KNN vs HNSW at scale.
N = 10_000_000
d = 1_536
single_core_flops = 10_000_000_000  # ~10 GFLOPS for AVX2 float32 dot product

# Exact KNN: N * d float multiplies per query
exact_ops_per_query = N * d
exact_seconds = exact_ops_per_query / single_core_flops

# HNSW: typically visits ~log(N) * ef_search nodes,
# each costing d operations for a distance computation.
import math
ef_search = 64
hnsw_nodes_visited = int(math.log2(N) * ef_search)  # ~1500
hnsw_ops_per_query = hnsw_nodes_visited * d
hnsw_ms = (hnsw_ops_per_query / single_core_flops) * 1000

Step-by-step explanation.

Exact KNN compares the query to every one of 10 million vectors, each comparison costing 1,536 floating-point multiplies — 1.5 × 10¹⁰ ops per query.
At 10 GFLOPS, that is 1.5 × 10¹⁰ / 10¹⁰ = 1.5 seconds per query on a single core. Even with 32 cores, you do not get below ~50 ms — and you have spent the whole core budget on one query.
HNSW visits ~log₂(N) · ef_search candidate nodes — about 23 · 64 ≈ 1,500 nodes per query at this scale. Each candidate costs 1,536 ops for a distance computation. Total: 2.3 × 10⁶ ops per query.
At 10 GFLOPS, that is 2.3 × 10⁶ / 10¹⁰ = 0.23 milliseconds of pure compute. Add memory latency, graph traversal overhead, and JSON serialisation — real-world p99 is typically 5–15 milliseconds.
The ratio: exact KNN is ~6,500× slower than HNSW at 10 million vectors. The recall cost of HNSW is 1–5 percent. The decision writes itself.

Output.

Method	Ops per query	Single-core time	Recall@10
Exact KNN	1.5 × 10¹⁰	1,500 ms	100%
HNSW (ef_search=64)	2.3 × 10⁶	0.23 ms (pure compute)	~95–99%
Ratio	~6,500×	~6,500×	-1 to -5 pts

Rule of thumb. Above ~100,000 vectors, always reach for ANN. Below that, exact KNN inside Postgres / NumPy is fine — and avoids an entire piece of infrastructure. The 100K threshold is the "do I need this at all?" cliff.

Worked example — cosine similarity vs dot product vs L2 distance

Detailed explanation. Every vector database supports at least three similarity metrics: cosine similarity, dot product, and L2 (Euclidean) distance. They are not interchangeable — the choice affects both correctness and how you store vectors. Most teams pick wrong on day one and spend two weeks debugging "why does retrieval feel off?"

Question. Given a set of normalised embeddings (unit vectors), explain why cosine similarity and dot product return the same ranking but different absolute scores, and explain when L2 distance gives the same ranking as cosine.

Input.

Vector A	Vector B	dot(A, B)	cosine(A, B)	L2(A, B)
[1, 0]	[1, 0]	1.0	1.0	0.0
[1, 0]	[0.6, 0.8]	0.6	0.6	0.89
[1, 0]	[0, 1]	0.0	0.0	1.41
[1, 0]	[-1, 0]	-1.0	-1.0	2.0

Code.

import numpy as np

def cosine(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

def l2(a, b):
    return np.linalg.norm(a - b)

# If both vectors are L2-normalised:
# - cosine(a, b) == dot(a, b)
# - l2(a, b)**2 == 2 - 2 * dot(a, b)
# Therefore cosine ranking and L2 ranking are identical on normalised vectors.

a = np.array([1.0, 0.0])
b = np.array([0.6, 0.8])
print(cosine(a, b), np.dot(a, b), l2(a, b))   # 0.6, 0.6, 0.894

Step-by-step explanation.

For unit-length vectors, cosine similarity is dot(a, b) / (1 · 1) = dot(a, b). So cosine and dot product return the same values and the same ranking.
For unnormalised vectors, dot product rewards longer vectors — a longer vector beats a shorter one on dot product even if the angle between them and the query is identical. This is why naked dot product surfaces "popular" documents in production unless you normalise first.
L2 distance and cosine similarity rank the same set of unit vectors identically. Proof: L2(a, b)² = 2 - 2·cos(a, b), which is strictly monotonic decreasing in cosine. Lower L2 ⇔ higher cosine.
If embeddings are not normalised, L2 and cosine can disagree. Always inspect the embedding model's documentation: most modern encoders (OpenAI's text-embedding-3-*, BGE, E5) return unit-length vectors already.
Pick cosine for text similarity; pick dot product when you have learned weights that intentionally encode "importance" via length (rare); pick L2 when the embedding model documents it (image embeddings sometimes use L2). Cosine is the safe default.

Output.

Embedding type	Best metric	Why
Unit-normalised text	cosine (or equivalently dot)	semantic angle; length normalised away
Unnormalised text	normalise then cosine	avoid length bias
Image (CLIP)	cosine	model trained with cosine loss
Trained ranking model with length signal	dot product	length encodes weight intentionally

Rule of thumb. Default to cosine. Switch only when the embedding model's documentation explicitly recommends another metric. Mixing metrics across producer and store (encoder uses cosine, store uses L2 on unnormalised) is a silent recall killer.

Vector database interview question on choosing a similarity metric

A senior interviewer often opens with: "Your team is building a documentation search box. The embedding model returns 1536-dimensional vectors and the docs say 'normalised, use cosine'. Walk me through what changes if we accidentally store them in a database that defaults to dot product on unnormalised vectors."

Solution Using a metric-aware pre-normalisation check

# 1) At ingestion: enforce normalised vectors and pick the right metric.
import numpy as np
import pgvector.psycopg  # pgvector example; same idea for Pinecone / Qdrant / Weaviate

def normalise(v: np.ndarray) -> np.ndarray:
    n = np.linalg.norm(v)
    return v if n == 0 else v / n

def ingest(rows, conn):
    with conn.cursor() as cur:
        for doc_id, raw_vec in rows:
            v = normalise(raw_vec)
            cur.execute(
                "INSERT INTO docs (doc_id, embedding) VALUES (%s, %s)",
                (doc_id, v.tolist()),
            )
    conn.commit()

# 2) At query time: use the cosine operator (<=>) in pgvector.
SELECT doc_id, 1 - (embedding <=> %s) AS score
FROM docs
ORDER BY embedding <=> %s
LIMIT 10;

Step-by-step trace.

Step	Action	Effect
ingest	normalise raw vector → store as unit vector	‖v‖ = 1.0 for every row
index build	pgvector HNSW with `vector_cosine_ops`	tree built using cosine distance
query	`embedding <=> query_vec` (cosine distance)	returns 1 - cos(θ); lower is better
score	`1 - (embedding <=> query_vec)`	converts back to similarity (higher is better)
order	`ORDER BY embedding <=> query_vec`	ascending distance = nearest first

The trace shows the two invariants you must preserve: every vector at ingestion is unit length, and every query path uses the same metric the index was built with.

Output:

Step	Result
Normalisation	every vector has L2 norm 1.0 ± 1e-6
Operator	pgvector cosine operator `<=>` returns cosine distance
Top-10 latency (1M docs, HNSW)	~3–8 ms
Recall vs brute force	≥ 98%

Why this works — concept by concept:

Normalisation at ingestion — collapses the cosine-vs-dot decision into one. Once every stored vector has unit length, cosine and dot product return the same ranking, and the query path stays simple.
Operator-class alignment — pgvector lets you choose vector_cosine_ops, vector_l2_ops, or vector_ip_ops (inner product) when creating the HNSW index. The chosen class must match the query operator (<=>, <->, <#>) or recall silently collapses.
Score conversion — <=> returns cosine distance (1 - cos), not cosine similarity. Most product code wants similarity; one subtract converts. Display-layer bugs ("the top result has score 0") usually trace to this mismatch.
One-metric contract — write down which metric the embedding model uses, which operator class the index uses, and which operator the query uses. All three must agree. This three-line invariant is what keeps the recall regression test green across migrations.
Cost — O(1) ingest overhead per row (one norm + divide); O(query) is unchanged because the HNSW index already used cosine at build time. Pure semantic insurance, zero runtime tax.

SQL
Topic — database
Database design problems (SQL)

Practice →

2. The vector DB role in a data platform

The vector database is one stage in a retrieval system — producers feed it, hybrid retrieval and rerankers sit above it

The mental model in one line: a vector database is the dense-retrieval middle layer between an embedding producer (batch / stream / backfill) and a higher-level retrieval composition that blends vector similarity, BM25 keyword scoring, metadata filters, and a reranker. Once you draw that flow on the whiteboard, the "do I need Pinecone or pgvector?" question reframes into "what is my producer throughput and my retrieval composition?" — which is what a senior interviewer actually wants to hear.

The producer side.

Batch embedder. Reads a source table or document store, calls the encoder model in batches of 32–512, writes vectors back into the store. Used for nightly backfills and the initial population.
Streaming embedder. Subscribes to a Kafka / Pub/Sub topic of new documents, embeds and upserts in seconds. Used for "freshness matters" use cases — news search, ticket dedupe, product catalogues.
Backfill harness. A separate code path that re-embeds the entire corpus when the model is upgraded. Always plan for this — every embedding-model upgrade is a full re-embed.

The store side — index, shard, replica.

Index. The data structure (HNSW, IVFFlat, DiskANN) that turns "top-K nearest" into a sub-linear operation. Lives in memory (HNSW, IVFFlat) or on disk (DiskANN, on-disk Qdrant).
Shard. Horizontal partitioning by vector_id hash or by tenant. Each shard owns a slice of the corpus; queries fan out to all shards and merge.
Replica. Read-replicas of each shard. Doubles or triples read throughput; the write path stays on the primary.

Index types per use case.

HNSW — default for low-latency online search. p99 latency 5–25 ms at 10M vectors. Memory-heavy.
IVFFlat — cheaper memory, slower writes (requires a training step on a sample). Good fit for "cheap shelf for medium recall."
DiskANN — built for 100M+ vectors on commodity SSDs. Higher tail latency, much cheaper RAM footprint.
Quantization layers — scalar (int8) and product quantization (PQ) sit on top of any of the above to compress vectors 4–32×.

Hybrid retrieval — vector + keyword + metadata.

Pure vector retrieval underperforms BM25 on exact-match queries ("error code 500"); pure BM25 underperforms vector on semantic queries ("login keeps failing").
Hybrid = both — score-fuse the two rankings (reciprocal rank fusion is the cheap default) or use a reranker on the union.
Metadata filter — pre-filter (push the predicate down into the index) or post-filter (run the ANN, then filter the results). Pre-filter is faster but harder to implement; post-filter is naïve and silently drops recall.

Online vs offline collections.

Online collection — serves the live application traffic. Must not be reindexed in place; updates are upserts.
Offline collection — sandbox for experiments (new embedding model, different chunking, larger ef_search). Promoted via a blue / green collection swap at the application layer.

Cache and reranker layers above the vector store.

Cache. Memoise frequent queries — (query_text → top-K result) in Redis with a 5-minute TTL. Cuts vector traffic 30–60 percent on repeat-heavy workloads.
Reranker. Pull top-50 from the ANN, then rerank with a cross-encoder model that scores the (query, document) pair more carefully. Adds 30–80 ms but lifts precision by 5–15 points.

Common interview probes on the role of the vector DB.

"What sits between the encoder and the vector DB?" — a batch embedder for backfills, a streaming embedder for live ingestion, a backfill harness for model upgrades. Three code paths, not one.
"What sits above the vector DB?" — a hybrid-retrieval composition (vector + BM25 + filter) and optionally a reranker on the top-50 union.
"How do you scale reads?" — read replicas behind the shard primary. Sharding gives capacity; replication gives QPS.
"What is a blue / green collection swap?" — keep two collections (v1, v2) populated in parallel during a model upgrade, then point the application at v2 atomically when ready.

Worked example — retrieval-augmented generation pipeline shape

Detailed explanation. A retrieval-augmented generation (RAG) stack rarely lives in a single service. Most teams underestimate the number of services it touches — five is typical, eight is normal in production. Drawing the shape forces the trade-off conversation onto firm ground.

Question. Sketch the data flow for a documentation-search RAG service: from user query to LLM answer. Mark every place where latency or cost is paid, and identify the two services where a cache helps most.

Input.

Stage	Service	Typical latency	Notes
1	Query embedding	encoder (50–200 ms)	one model call
2	ANN retrieval	vector DB (5–25 ms)	top-50 candidates
3	Metadata filter	vector DB (1–5 ms)	tenant + lang + date
4	Reranker	cross-encoder (30–80 ms)	top-50 → top-10
5	LLM call	LLM API (300–1500 ms)	top-10 as context

Code.

# Simplified RAG flow with two cache layers.
def rag_answer(query: str, tenant_id: int, lang: str) -> str:
    cache_key = (query, tenant_id, lang)

    # Cache 1 — full-answer cache (cuts LLM cost on repeat queries)
    if hit := answer_cache.get(cache_key):
        return hit

    # Cache 2 — retrieval cache (cuts encoder + vector DB cost)
    if not (top_k := retrieval_cache.get(cache_key)):
        q_vec = encoder.embed(query)                              # 50-200 ms
        candidates = vector_db.search(                            # 5-25 ms
            q_vec,
            top_k=50,
            filter={"tenant_id": tenant_id, "lang": lang},
        )
        top_k = reranker.rerank(query, candidates, top_k=10)      # 30-80 ms
        retrieval_cache.set(cache_key, top_k, ttl=300)

    prompt = build_prompt(query, top_k)
    answer = llm.complete(prompt)                                  # 300-1500 ms
    answer_cache.set(cache_key, answer, ttl=300)
    return answer

Step-by-step explanation.

The user query enters as plain text. Stage 1 — query embedding — calls the encoder model and produces a 1536-d vector. This is a hard cost on every uncached query.
Stage 2 — ANN retrieval — calls the vector database with the query vector. The filter tenant_id = ? AND lang = ? is pushed down so the ANN only inspects rows that match. Top-50 candidates come back in milliseconds.
Stage 3 (logically — actually part of stage 2 on most vendors) is the metadata filter pushdown; in vendors like Pinecone and Qdrant it is woven into the ANN itself; in pgvector with the wrong query plan it can become a post-filter step that silently drops recall.
Stage 4 — reranker — pulls a cross-encoder model that scores (query, candidate) pairs more carefully than dense similarity. The top-50 are reranked to top-10. This costs 30–80 ms but lifts precision visibly.
Stage 5 — LLM call — sends the top-10 as context to the generative model. Dominates the end-to-end latency budget. Two cache layers (answer cache, retrieval cache) cover both the repeat-query case and the repeat-retrieval-different-prompt case.

Output.

Path	End-to-end p50	End-to-end p99
Full cache miss	400–1800 ms	1500–3000 ms
Retrieval cache hit, answer miss	320–1500 ms	1200–2500 ms
Answer cache hit	1–5 ms	10–30 ms

Rule of thumb. The LLM call is the dominant cost; everything else is a rounding error. Two cache layers (retrieval cache for top-K, answer cache for full responses) usually pay for themselves within a week of traffic.

Worked example — pre-filter vs post-filter recall trap

Detailed explanation. A naïve implementation runs the ANN first to get top-K, then applies a metadata filter to the results. If the filter is selective (rare value), the post-filter throws away most candidates and the user sees fewer than K results — a silent recall regression that does not show up in latency dashboards.

Question. Given a 10M-vector corpus where only 1 percent of rows have lang = 'sv' (Swedish), explain what happens with naive post-filter retrieval. Show the fix with pre-filtering.

Input.

Strategy	Top-K initial	Candidates after filter	User sees
Post-filter, K=10	10	0–1 (statistically)	near-empty result
Post-filter, K=1000	1000	~10	OK but slow
Pre-filter, K=10	10 (already filtered)	10	correct

Code.

# WRONG — post-filter
def search_post_filter(query_vec, lang):
    candidates = vector_db.knn_query(query_vec, top_k=10)    # global top-10
    return [c for c in candidates if c.lang == lang]         # often 0

# BETTER — over-fetch and post-filter
def search_overfetch(query_vec, lang):
    candidates = vector_db.knn_query(query_vec, top_k=1000)  # 100x
    filtered = [c for c in candidates if c.lang == lang]
    return filtered[:10]

# CORRECT — pre-filter (filter pushdown into the ANN)
def search_pre_filter(query_vec, lang):
    return vector_db.knn_query(
        query_vec,
        top_k=10,
        filter={"lang": lang},   # vendor pushes this into the index
    )

Step-by-step explanation.

The naive post-filter asks the vector DB for the global top-10 across the entire 10M-vector corpus. Statistically, only 1 percent of those will have lang = 'sv' — most often zero.
The over-fetch fix asks for top-1000 instead. Now ~10 will pass the filter on average. The trade-off: 100× more candidates to fetch and re-rank, with corresponding latency cost.
The correct fix is pre-filtering — pushing the predicate into the ANN itself. Pinecone exposes this via the filter argument; Qdrant via must conditions; Weaviate via where; pgvector via a WHERE lang = 'sv' clause that the planner pushes below the index.
With pre-filtering, the ANN only ever inspects vectors that match the filter. Recall is preserved, latency stays flat, and the user always sees K results when K candidates exist.

Output.

Strategy	Recall when K=10	Latency	Correctness
Naive post-filter	~5% (often 0 results)	low	broken
Over-fetch K=1000 then filter	~80%	100× higher	mediocre
Pre-filter (pushdown)	95%+	normal	correct

Rule of thumb. Always check whether your vendor pushes metadata filters into the ANN index versus applying them post-search. The difference shows up at moderate selectivity and gets catastrophic at high selectivity. Selectivity that looks "small" (1 percent) is exactly where the post-filter strategy breaks.

Worked example — sharding and replication topology

Detailed explanation. Sharding gives you capacity (more total vectors); replication gives you read throughput (more concurrent queries). They are orthogonal. A common mistake is conflating them — "we sharded the index" sometimes means "we added a read replica," which solves a different problem.

Question. Given a 200M-vector workload at 4,000 queries per second p99, design a topology. Assume each replica can hold 50M vectors in memory and handle 1,000 QPS at the recall SLO.

Input.

Constraint	Value
Total vectors	200,000,000
Peak QPS	4,000
Replica memory budget	50M vectors
Replica QPS budget	1,000

Code.

# Topology calculation.
total_vectors = 200_000_000
peak_qps = 4_000

vectors_per_replica = 50_000_000
qps_per_replica = 1_000

shards = total_vectors // vectors_per_replica          # 4 shards
replicas_per_shard = -(-peak_qps // qps_per_replica)   # 4 replicas
                                                       # serving every shard
nodes = shards * replicas_per_shard                    # 16 nodes

Step-by-step explanation.

The 200M-vector total exceeds a single replica's 50M-vector memory budget by 4×. We need 4 shards to fit the corpus, each owning ~50M vectors.
The 4,000 QPS peak exceeds a single replica's 1,000-QPS throughput by 4×. We need 4 replicas per shard to serve the load — every shard must answer every query.
Total node count: 4 shards × 4 replicas = 16 nodes. The application's query router fans out to one replica per shard (4 fan-outs per query) and merges the top-K from each.
The fan-out merge step costs O(shards · K) plus a final sort. At K=10 and 4 shards, this is 40 candidates merged — trivial cost compared to the underlying ANN call.
If the workload grows to 500M vectors, add shards (10 total). If QPS grows to 10,000, add replicas per shard (10 replicas × 10 shards = 100 nodes). The two axes scale independently.

Output.

Resource	Value
Shards (capacity)	4
Replicas per shard (QPS)	4
Total nodes	16
Fan-out per query	4 (one per shard)
Merge cost	O(40) at K=10

Rule of thumb. Sharding is for capacity. Replication is for throughput. Compute each independently from the workload, then multiply. Conflating the two is the most common topology bug — and the one that makes "we need a bigger vector DB" sound louder than the actual workload demands.

Vector database interview question on platform topology

A senior interviewer often frames this as: "Walk me through the data flow for a documentation search service from raw doc to user answer. Where would you add a cache, where would you shard, and what changes when the embedding model is upgraded?"

Solution Using a layered RAG topology with explicit cache and blue / green collection

# 1) Producer — batch + streaming embedders feeding one online collection.
class EmbeddingProducer:
    def upsert_batch(self, docs):
        vecs = self.encoder.embed_batch([d.text for d in docs])
        self.vector_db.upsert(self.collection, list(zip(docs, vecs)))

# 2) Retrieval — pre-filter ANN with cache.
class Retriever:
    def search(self, query, tenant_id, lang, k=10):
        key = (query, tenant_id, lang, k)
        if hit := self.retrieval_cache.get(key):
            return hit
        q_vec = self.encoder.embed(query)
        candidates = self.vector_db.search(
            self.collection,
            q_vec,
            top_k=50,
            filter={"tenant_id": tenant_id, "lang": lang},
        )
        top_k = self.reranker.rerank(query, candidates, top_k=k)
        self.retrieval_cache.set(key, top_k, ttl=300)
        return top_k

# 3) Model upgrade — blue / green collection swap.
class ModelUpgrade:
    def run(self, new_model):
        green = self.vector_db.create_collection("docs_v2", dim=new_model.dim)
        self.backfill(new_model, green)               # re-embed full corpus
        self.dual_write(green)                        # mirror live writes
        self.compare_metrics(blue="docs_v1", green="docs_v2")
        self.application.point_at("docs_v2")          # atomic switch
        self.vector_db.drop_collection("docs_v1")     # tombstone after soak

Step-by-step trace.

Step	Service touched	Latency contribution	Cost contribution
1 — query in	application	0	0
2 — answer cache check	Redis	1 ms	$0 if hit
3 — retrieval cache check	Redis	1 ms	$0 if hit
4 — encoder call	encoder API	50–200 ms	$0.0001 / call
5 — vector DB pre-filter search	vector DB	5–25 ms	replica RAM
6 — reranker	cross-encoder	30–80 ms	GPU seconds
7 — LLM call	LLM API	300–1500 ms	$0.001 / call

The trace identifies the two cache wins: the retrieval cache covers stage 4 + 5 + 6 (the dense half of the pipeline) and the answer cache short-circuits everything.

Output:

Topology axis	Value
Shards	4 (50M vectors each)
Replicas per shard	4 (1,000 QPS each)
Online collection	`docs_v1` (current model)
Offline collection	`docs_v2` (new model, populating)
Cache layers	retrieval + answer (Redis)

Why this works — concept by concept:

Pre-filter pushdown — the metadata filter is woven into the ANN call, so recall stays at 95%+ even when the filter selectivity is 1 percent. Post-filter strategies silently drop the result count below K.
Retrieval cache — memoises (query, tenant, lang) → top-K for 5 minutes. Cuts encoder + vector DB cost 30–60 percent on repeat-heavy traffic without affecting freshness in any meaningful way.
Reranker on top-50 — the dense ANN gives a generous recall pool; the cross-encoder rerank pushes precision up by 5–15 points. The cost is one extra forward pass on 50 pairs — modest at GPU prices.
Blue / green collection swap — the embedding-model upgrade is the riskiest day-2 operation. Building a sibling collection lets you keep the old one serving while the new one fills, then atomically flip the application pointer.
Cost — encoder + LLM dominate variable cost; vector DB dominates fixed cost (replica RAM). Cache amortises both. Total p99 SLO is bounded by LLM latency.

SQL
Topic — design
System design problems (SQL)

Practice →

3. Pinecone vs Weaviate vs Qdrant vs pgvector — vendor comparison

The four-vendor matrix is really a two-axis pick: how managed do you want it, and what scale do you have today

The mental model in one line: pgvector is the "reuse Postgres" first choice up to ~10M vectors; pinecone is the fully managed serverless option; weaviate is the modular OSS pick when you need RAG modules and a GraphQL API; qdrant is the low-latency Rust option with strong payload filtering. Once you can recite that, the "which one should we use?" interview question becomes a fifteen-second answer instead of a fifteen-minute survey.

The vendor matrix in one table.

Vendor	Hosting	Index types	Scale sweet spot	Key strength	Key trade-off
Pinecone	Fully managed (serverless + dedicated)	HNSW (proprietary)	10M–10B vectors	zero-ops, multi-tenant, serverless	closed source; lock-in
Weaviate	OSS + managed	HNSW, IVF, flat	1M–500M vectors	RAG modules, GraphQL API, BYO encoder	heavier surface area
Qdrant	OSS + managed	HNSW (in-memory or on-disk)	1M–1B vectors	strong payload filter, Rust performance	smaller ecosystem
pgvector	Postgres extension	HNSW, IVFFlat	up to ~10M (single replica)	reuse Postgres, SQL joins	scale ceiling

Pinecone in detail.

Hosting model. Fully managed serverless or dedicated. No nodes to operate, no index to rebuild — the vendor handles everything.
API surface. REST + gRPC. Concepts: index, namespace (for multi-tenant), upsert, query, filter.
Sweet spot. Teams that want to ship retrieval without an ops team. Workloads from 1M to 10B vectors.
Trade-off. Closed source; vendor lock-in; per-vector pricing that compounds at large scale. Compliance teams sometimes block it (no on-prem mode).

Weaviate in detail.

Hosting model. Open-source binary, self-hosted, or managed cloud.
API surface. GraphQL is the canonical API; REST also supported. Concepts: class (schema), object, vectorizer (built-in encoder modules).
Sweet spot. Teams that want OSS with batteries included — built-in RAG modules (text2vec-openai, text2vec-cohere), generative search, hybrid search out of the box.
Trade-off. Heavier surface area than the others — the schema, the modules, the GraphQL layer all add concepts. Steeper learning curve.

Qdrant in detail.

Hosting model. Open-source binary in Rust, self-hosted, or managed cloud.
API surface. REST + gRPC. Concepts: collection, point, payload (metadata), filter (the strongest payload filter language of any vendor).
Sweet spot. Low-latency online search with heavy filtering. Rust performance shows up on the tail latency curve.
Trade-off. Smaller ecosystem than Pinecone / Weaviate. Fewer prebuilt integrations.

pgvector in detail.

Hosting model. A Postgres extension — runs anywhere Postgres runs (RDS, Aurora, Cloud SQL, on-prem).
API surface. Native SQL. Indexes: HNSW (USING hnsw) and IVFFlat (USING ivfflat).
Sweet spot. Teams that already operate Postgres and need vectors as one more column. Up to ~10M vectors on a single replica before reindex windows and memory budgets become painful.
Trade-off. Scale ceiling. Above ~10–50M vectors on commodity hardware, a dedicated vector DB pulls ahead on both latency and operations.

Latency, throughput, recall — order-of-magnitude landmarks.

Scale	Vendor	Typical p99 latency	Typical replica memory
1M vectors	any of the four	2–10 ms	4–6 GB
10M vectors	Pinecone / Weaviate / Qdrant	5–25 ms	30–60 GB
10M vectors	pgvector HNSW	10–60 ms	30–60 GB
100M vectors	Pinecone / Qdrant DiskANN	10–50 ms	50–200 GB SSD
100M vectors	pgvector	unsupported in practice	—
1B+ vectors	Pinecone (sharded) / DiskANN OSS	20–100 ms	TB of SSD

These are not benchmarks — they are landmarks for "what scale fits which vendor without heroic engineering." Real benchmarks must always be run on your data and your queries.

When pgvector is enough.

Corpus under 10M vectors, single tenant or a small fixed number of tenants.
Throughput under 200 QPS sustained.
Heavy joins to other Postgres tables (orders, users, products) where the locality of "vector + structured fact" matters more than maximum throughput.
Ops team that already runs Postgres at scale — no new infrastructure to take on.

When you need a dedicated vector DB.

Corpus above ~50M vectors, or growing fast enough that you will cross 50M within the year.
Multi-tenant SaaS with hundreds or thousands of namespaces.
Strict p99 latency SLO under 25 ms at high QPS.
Embedding model upgrades are routine (more than once a quarter) — managed vendors give you the blue / green collection primitive cheaply.

Worked example — picking a vendor by scale and operational profile

Detailed explanation. Pretend you are sitting in front of a whiteboard with a product manager who has just announced "we are doing RAG." Most teams ship the wrong vendor on day one because they pick by brand awareness rather than by scale and operations. A two-question filter usually gets the right answer in 60 seconds.

Question. Given a small RAG product (target launch — 1 month) with 500K documents and a single tenant, which vendor should the team start with? When does it stop being the right choice?

Input.

Dimension	Value
Initial corpus	500K documents → ~2M chunks
Tenants	1 (single-tenant)
Expected QPS	50
Existing infra	Postgres (RDS)
Ops team size	3 engineers, no dedicated DB ops

Code.

-- pgvector setup — five lines, no new service.
CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE doc_chunks (
    chunk_id    BIGSERIAL PRIMARY KEY,
    doc_id      BIGINT       NOT NULL,
    tenant_id   INT          NOT NULL,
    lang        TEXT         NOT NULL,
    text        TEXT         NOT NULL,
    embedding   VECTOR(1536) NOT NULL
);

CREATE INDEX doc_chunks_hnsw
ON doc_chunks
USING hnsw (embedding vector_cosine_ops)
WITH (m = 16, ef_construction = 200);

-- Online query with metadata filter.
SELECT chunk_id, text, 1 - (embedding <=> $1) AS score
FROM doc_chunks
WHERE tenant_id = $2 AND lang = $3
ORDER BY embedding <=> $1
LIMIT 10;

Step-by-step explanation.

The corpus is 2M vectors. Well below the 10M scale ceiling — pgvector fits comfortably on a single Postgres replica.
The team already operates Postgres. Adding pgvector is CREATE EXTENSION vector + an index — no new service, no new on-call rotation, no new pricing dimension.
The expected throughput is 50 QPS. A single replica with HNSW serves this with p99 under 20 ms — plenty of headroom for the first year of growth.
The metadata filter (tenant_id, lang) is a standard B-tree WHERE pushdown. Postgres's query planner generally handles this well, but the team should verify with EXPLAIN ANALYZE that the HNSW index plus the filter compose without falling back to a sequential scan.
The migration trigger to a dedicated vector DB: when the corpus crosses ~10M vectors or when QPS sustained crosses ~200 or when the team starts running multi-tenant with hundreds of namespaces. Until any of those flip, pgvector is the right pick.

Output.

Decision	Choice	Why
Day-one pick	pgvector	reuse Postgres; minimal ops surface
Migration trigger 1	corpus > 10M vectors	reindex windows become painful
Migration trigger 2	sustained QPS > 200	single-replica throughput ceiling
Migration trigger 3	many tenants (100+)	namespace primitives in dedicated vendors

Rule of thumb. Start with pgvector if you already run Postgres. Add a dedicated vector DB when you cross one of the three migration triggers above — not before, not because of brand. The wrong vendor in week one is recoverable; the wrong vendor with 50M vectors loaded is not.

Worked example — Pinecone serverless vs Qdrant self-hosted on the same workload

Detailed explanation. A common interview ask is "compare two specific vendors." The trap is that comparing on features alone misses the operational delta. Two vendors with similar feature surfaces can have a 10× difference in ops cost.

Question. Given a 30M-vector workload at 1,500 QPS with a 25 ms p99 SLO, compare Pinecone serverless and Qdrant self-hosted on three axes: time-to-first-query, monthly cost order of magnitude, and the operational events the team must handle.

Input.

Vendor	Hosting model	Pricing axis	Team ops responsibility
Pinecone serverless	fully managed	per-vector + per-query	none
Qdrant self-hosted	OSS on EC2 / EKS	infrastructure only	node lifecycle, upgrades, backups

Code.

# Pinecone serverless — minutes to first query, no infra to provision.
pinecone create-index docs-prod \
  --dimension 1536 \
  --metric cosine \
  --cloud aws --region us-east-1

# Qdrant self-hosted — Helm chart on EKS plus node group capacity planning.
helm install qdrant qdrant/qdrant \
  --set persistence.size=300Gi \
  --set replicaCount=4

Step-by-step explanation.

Time-to-first-query. Pinecone serverless: create-index plus credentials, you are upserting in minutes. Qdrant self-hosted: provision EKS, configure the Helm chart, size the node group, set up TLS and IAM — typically a week of work the first time.
Monthly cost. Pinecone serverless charges per stored vector and per query — predictable, scales linearly. Qdrant self-hosted pays only EC2 + EBS — flat curve once the cluster is up. Crossover point is workload-dependent, typically around 50–100M vectors for Pinecone to become more expensive than self-hosted.
Operational events with Pinecone: none — vendor handles upgrades, replication, failover. Operational events with Qdrant self-hosted: node upgrades, snapshot backups, capacity scaling, on-call rotation. Different team-size implication.
Migration cost. Both expose REST + gRPC; switching is painful but not impossible. The data export side is well-supported on both. The application side is one client library swap.
The right answer. "Pinecone serverless until either cost crosses our self-hosted floor or compliance requires on-prem. Then Qdrant self-hosted." The decision is a function of team size, compliance, and projected scale — not a feature checklist.

Output.

Axis	Pinecone serverless	Qdrant self-hosted
Time-to-first-query	minutes	days to a week
Monthly cost at 30M / 1500 QPS	predictable, mid-tier	flat once provisioned
Ops events / month	0	1–4 (upgrades, capacity, backups)
On-prem option	no	yes
Cost crossover	~50–100M vectors	—

Rule of thumb. When team size is small and compliance permits, managed services trade money for time and they almost always win the first two years. When team size is comfortable and compliance requires on-prem, self-hosted Qdrant or Weaviate trades time for money — both legitimate trade-offs.

Worked example — Weaviate's hybrid search out of the box

Detailed explanation. Weaviate is the only vendor of the four with first-class hybrid search built in — the team supplies alpha, a single knob that blends BM25 and vector scoring. It is the smallest amount of code for a hybrid retrieval baseline.

Question. Given a knowledge base loaded into Weaviate with the text2vec-openai module, write a hybrid search query that blends keyword and vector scoring with alpha = 0.5.

Input.

User query	Pure vector top-1	Pure BM25 top-1	Hybrid top-1 (alpha=0.5)
"error code 500"	"the server failed"	"error code 500 troubleshooting"	"error code 500 troubleshooting"
"why won't my user log in"	"login flow troubleshooting"	"log file format"	"login flow troubleshooting"

Code.

# Weaviate hybrid search — alpha blends vector and BM25.
import weaviate

client = weaviate.Client("https://weaviate-cluster.example.net")

result = (
    client.query
    .get("Document", ["title", "snippet"])
    .with_hybrid(
        query="error code 500",
        alpha=0.5,                        # 0=BM25 only, 1=vector only
    )
    .with_limit(10)
    .with_where({
        "path": ["tenant_id"],
        "operator": "Equal",
        "valueInt": 7,
    })
    .do()
)

Step-by-step explanation.

Weaviate's with_hybrid operator runs the vector ANN and the BM25 keyword search in parallel, then fuses the two scores with the alpha-weighted formula score = alpha · vector_score + (1 - alpha) · bm25_score.
alpha = 0 is keyword-only (classical BM25). alpha = 1 is vector-only (pure ANN). alpha = 0.5 is an equal blend.
For the "error code 500" query, BM25 hits the exact phrase in the title — high keyword score. The vector might match semantically similar but less exact docs. Hybrid surfaces the exact-match doc on top because BM25's contribution breaks the tie.
For the "why won't my user log in" query, BM25 hits docs containing "log" and "log file" — semantically wrong. Vector hits the login-flow doc semantically. Hybrid keeps the semantically right answer at the top because the vector signal is strong enough.
The with_where predicate is pushed into the ANN as a pre-filter — multi-tenant isolation is enforced before the dense scoring.

Output.

Query	alpha	Top-1
"error code 500"	0.5	"Error Code 500 troubleshooting"
"why won't my user log in"	0.5	"Login flow troubleshooting"
"what is a vector database"	0.5	"Introduction to vector databases"

Rule of thumb. Hybrid search with alpha ≈ 0.5 outperforms either pure mode on broad evaluation sets — typically lifting top-1 precision by 5–15 points. Sweep alpha on your own eval set; the optimum is rarely exactly 0.5, often 0.3–0.7.

Vector database interview question on the right pgvector vs Pinecone trade-off

A senior interviewer often opens with: "We have 8M document chunks in Postgres today and we are exploring RAG. The team is two engineers. Should we stay on pgvector or move to Pinecone?" The interviewer wants to hear the reasoning, not the brand.

Solution Using a scale + ops + compliance decision matrix

# Decision criteria — three filters, each with a sharp threshold.

Filter 1 — Scale today and projected 12 months out:
  - Corpus under 10M vectors AND projected under 30M in 12 months → pgvector OK
  - Otherwise → dedicated vector DB

Filter 2 — Ops team capacity:
  - Already running Postgres at production scale → pgvector cheap
  - No DB ops experience → managed (Pinecone) trades money for time

Filter 3 — Compliance / data residency:
  - SaaS or US/EU general workload → either is fine
  - On-prem mandate or strict residency → self-hosted (Qdrant / Weaviate / pgvector)

Decision for the asked case (8M today, 2 engineers, no on-prem mandate):
  - Filter 1: 8M < 10M → pass
  - Filter 2: 2 engineers, already on Postgres → pgvector wins
  - Filter 3: no mandate → pgvector wins
  - Choice: pgvector. Revisit in 6 months when corpus crosses 15M.

Step-by-step trace.

Criterion	Threshold	Asked case	Verdict
Corpus size	< 10M today, < 30M in 12 months	8M today, ~20M projected	pass
Ops team experience	already operates Postgres	yes, 2 engineers	pgvector cheap
Throughput SLO	< 200 QPS sustained	80 QPS expected	pass
Compliance	no on-prem mandate	no	both options open
Decision	—	—	pgvector

The trace shows the question is not "is pgvector good enough?" — it is "do any of three sharp thresholds force you off pgvector?" If no threshold flips, the cheapest option wins.

Output:

Decision	Action
Start with	pgvector on existing Postgres
First reindex window planned	12 months out (or earlier on triggers)
Migration plan trigger	corpus > 15M vectors or QPS > 200
Migration plan candidate	Qdrant self-hosted (cost-effective) or Pinecone serverless (zero-ops)

Why this works — concept by concept:

Scale threshold — pgvector's HNSW lives in shared buffers and OS page cache; performance falls off a cliff above ~10–50M vectors per replica on commodity hardware. Below that threshold, the ANN behaviour is identical to dedicated stores.
Ops leverage — adopting a new datastore costs an on-call rotation, monitoring instrumentation, and a learning curve. Postgres already has all three. Reusing it is the cheapest possible day-1 decision.
Compliance gate — managed serverless vendors fail certain on-prem and data-residency requirements. The compliance filter must run before the cost or feature comparison or you build a system you cannot ship.
Migration trigger discipline — write down the thresholds now, not later. When pgvector starts hurting, the team has a pre-agreed exit, not a new debate.
Cost — pgvector adds zero net new infrastructure; the Postgres replica was already paid for. Dedicated stores cost an extra service from day one. The pgvector path is strictly cheaper until a threshold flips.

SQL
Topic — design
Design problems (SQL)

Practice →

4. Index types — HNSW, IVFFlat, quantization, DiskANN

There is no universally best ANN index — only the right one for your workload size, latency SLO, and memory budget

The mental model in one line: hnsw is the default low-latency in-memory index; ivfflat is the cheaper-memory option with a training step; scalar / product quantization compresses vectors 4–32× with a few-point recall trade-off; DiskANN is the "100 million-plus vectors on commodity SSD" play. Once you can map a workload to one of those four buckets, the index choice — and the parameter tuning — falls out almost mechanically.

HNSW (Hierarchical Navigable Small World) in detail.

Shape. A multi-layer graph where each node connects to a fixed number of neighbours per layer; the upper layers are sparser and act as "long-distance jumps."
Build parameters. M (out-degree per node, typical 12–48; default 16). ef_construction (candidate pool at build time, typical 100–400; default 200).
Query parameter. ef_search — the candidate pool at query time. Larger → higher recall and higher latency.
Strength. Lowest p99 latency of any in-memory index at moderate scale. The default in pgvector, Pinecone, Qdrant, Weaviate.
Weakness. Memory-heavy — every vector lives in RAM plus its graph edges. ~(4 · dim + 8 · M) · N bytes baseline.

IVFFlat (Inverted File with Flat quantizer) in detail.

Shape. Vectors are clustered into nlist centroids via k-means; each query inspects the closest nprobe clusters and brute-force-searches their members.
Build parameters. nlist (number of centroids, typical sqrt(N)). Requires a training step on a representative sample.
Query parameter. nprobe (clusters to inspect). Larger → higher recall and higher latency.
Strength. Smaller memory footprint than HNSW (only the centroids + inverted lists). Cheaper writes.
Weakness. Higher tail latency. The training step is offline and complicates incremental indexing.

Scalar quantization (SQ) in detail.

Idea. Store each float32 component as an int8 — 4× compression. Recall drops by 0.5–3 points; latency improves because the distance computation is now cheap integer arithmetic.
When to use. Always consider on top of HNSW or IVFFlat when memory is the binding constraint. The recall trade-off is usually acceptable for production retrieval.

Product quantization (PQ) in detail.

Idea. Split each vector into m sub-vectors, learn a codebook of 256 entries per sub-vector, store each sub-vector as a 1-byte codebook index. Typical compression 16–32×.
When to use. Massive scale (100M+) where SQ alone cannot fit the memory budget. Recall drops by 3–8 points; consider with a rerank-on-top of the original float32 vectors for the top-50.

DiskANN in detail.

Shape. Graph index similar to HNSW but explicitly designed to live on SSD. Queries pay sequential SSD reads instead of paying RAM cost.
When to use. 100M+ vectors where the RAM-only cost of HNSW exceeds the budget. Pinecone serverless uses a DiskANN-class technique under the hood at large scale; OSS DiskANN is the canonical reference.
Trade-off. Higher p99 latency than in-memory HNSW (typically 20–80 ms vs 5–25 ms), but order-of-magnitude cheaper RAM footprint.

Parameter intuition in one line each.

M — graph density. Higher M → better recall, more RAM.
ef_construction — build effort. Higher → better quality graph, slower index build.
ef_search — query effort. Higher → better recall, higher query latency.
nlist — IVFFlat coarse clusters. Rule of thumb: nlist ≈ sqrt(N).
nprobe — IVFFlat search effort. Higher → better recall, higher query latency.

Common interview probes on index choice.

"Why is HNSW the default?" — best latency-recall trade-off for in-memory workloads at moderate scale. The graph structure has logarithmic search complexity.
"When would you reach for IVFFlat over HNSW?" — when memory is the binding constraint and you can tolerate higher tail latency. Often paired with PQ.
"How does quantization affect recall?" — scalar quantization (int8) loses 0.5–3 points; product quantization loses 3–8 points. Usually worth it for the memory savings.
"What is DiskANN for?" — vectors at 100M+ scale on commodity SSDs; trades RAM for predictable disk I/O.

Worked example — picking HNSW vs IVFFlat for a 5M-vector workload

Detailed explanation. A team has 5M vectors of dim=1536 and a single replica with 32 GB RAM. They need to pick between HNSW and IVFFlat. The right answer depends on the recall floor, the latency ceiling, and the RAM budget — but the calculation can be done in two minutes.

Question. Estimate the memory footprint of HNSW with M=16 and IVFFlat with nlist=2000 on this workload. Pick the index given a 25 ms p99 SLO and a 95 percent recall floor.

Input.

Quantity	Value
N (vectors)	5,000,000
dim	1536
RAM budget	32 GB
p99 SLO	25 ms
Recall floor	95%

Code.

N = 5_000_000
dim = 1_536

# HNSW memory: ~4 bytes per float component + ~8 bytes per graph edge per node.
M = 16
hnsw_vec_bytes = N * dim * 4              # ~30 GB just for vectors
hnsw_edge_bytes = N * M * 8 * 2           # bidirectional edges
hnsw_total_gb = (hnsw_vec_bytes + hnsw_edge_bytes) / (1024**3)

# IVFFlat memory: vectors stored once + small centroid table.
nlist = 2000
ivf_centroid_bytes = nlist * dim * 4
ivf_vec_bytes = N * dim * 4
ivf_total_gb = (ivf_centroid_bytes + ivf_vec_bytes) / (1024**3)

print(round(hnsw_total_gb, 1), round(ivf_total_gb, 1))
# HNSW ~ 30.5 GB; IVFFlat ~ 28.6 GB

Step-by-step explanation.

The vector payload itself — N · dim · 4 bytes for float32 — is 5M · 1536 · 4 = 30.7 GB. This is the same for either index because both need the raw vectors to compute distance.
HNSW adds graph edges: N · M · 8 bytes for the forward edges, doubled for bidirectional. At M=16 and bidirectional, this is ~1.3 GB. HNSW total: ~32 GB — right at the RAM ceiling.
IVFFlat adds the centroid table: nlist · dim · 4 = 2000 · 1536 · 4 = 12 MB. Negligible. IVFFlat total: ~30.7 GB — comfortably under the ceiling.
The RAM picture says IVFFlat wins on memory by 1–2 GB. But the latency picture says HNSW typically beats IVFFlat by 2–3× on p99 at this scale. With a 25 ms SLO and the recall floor at 95 percent, HNSW with ef_search=64 lands at ~12 ms p99 / 95.5 percent recall; IVFFlat with nprobe=20 lands at ~28 ms p99 / 94 percent recall.
The right pick: HNSW, but consider scalar quantization (int8) on top to cut the vector payload from 30.7 GB to ~7.7 GB. Now the replica has 24 GB of headroom and the latency stays in budget.

Output.

Index	Memory (GB)	p99 latency	Recall@10	SLO met?
HNSW M=16	~32	12 ms	95.5%	yes (tight)
HNSW + scalar quantization	~9	10 ms	94%	borderline
IVFFlat nlist=2000	~31	28 ms	94%	latency fails
IVFFlat + scalar quantization	~8	22 ms	92%	recall fails

Rule of thumb. Default to HNSW for online retrieval workloads below 50M vectors per replica. Layer scalar quantization on top when the vector payload dominates the RAM budget. Reach for IVFFlat only when the workload genuinely tolerates higher tail latency.

Worked example — product quantization at 100M vectors

Detailed explanation. Product quantization (PQ) is the technique that makes 100M-vector workloads fit on commodity hardware. It splits each vector into m sub-vectors, each represented by a 1-byte codebook index. The compression ratio scales with m and the trade-off shows up in recall.

Question. Given 100M vectors of dim=1024 and PQ with m=64 sub-vectors, estimate the memory footprint, the per-vector compression ratio, and the recall delta vs raw float32.

Input.

Quantity	Value
N (vectors)	100,000,000
dim	1024
PQ sub-vectors `m`	64 (so each sub-vector is dim=16)
Codebook entries per sub-vector	256 (1 byte index)

Code.

N = 100_000_000
dim = 1_024
m = 64

# Raw float32 footprint
raw_bytes = N * dim * 4

# PQ footprint: 1 byte per sub-vector per row.
pq_bytes = N * m

# Compression ratio
ratio = raw_bytes / pq_bytes

# Codebook stored once: m codebooks of 256 entries each of size (dim/m * 4) bytes.
codebook_bytes = m * 256 * (dim // m) * 4

print(raw_bytes / 1e9, pq_bytes / 1e9, ratio)
# Raw: 409.6 GB; PQ: 6.4 GB; ratio: ~64x

Step-by-step explanation.

Raw float32 storage is 100M · 1024 · 4 = 409 GB. Way past any single-machine RAM budget.
PQ with m=64 stores 64 bytes per vector — one codebook index per sub-vector. Total: 100M · 64 = 6.4 GB. The compression ratio is 4096 / 64 = 64×.
The codebook itself is small: 64 codebooks · 256 entries · 16 floats · 4 bytes = 256 KB. Negligible.
Distance computation with PQ uses asymmetric distance: the query stays in float32, and you precompute the query's distance to every codebook entry in each sub-vector (256 distances per sub-vector, computed once per query). Then the distance from query to a stored vector is a sum of 64 table lookups.
Recall drops by ~5–8 points vs raw float32 ANN. The standard fix is rerank-on-top: pull top-100 with PQ, fetch the original float32 vectors for those 100 rows (still small), and re-score. Recall recovers to within 0.5 points of raw.

Output.

Storage	Footprint	Recall@10	Distance compute cost
Raw float32	409 GB	100% (baseline)	4 KB per pair
PQ m=64	6.4 GB	~92–95%	64 lookups per pair
PQ + rerank top-100	6.5 GB	~99%	PQ + 100 float32 dot products

Rule of thumb. Use PQ at 100M+ scale or when raw float32 storage exceeds the RAM budget by 5× or more. Always pair PQ with a rerank step on the original vectors for the top-50 or top-100 — it recovers most of the recall loss for negligible extra cost.

Worked example — DiskANN for 500M vectors on a single host

Detailed explanation. DiskANN was designed at Microsoft Research for the "billion-scale on a single commodity machine" problem. It builds a graph index similar to HNSW but explicitly tuned to live on SSD — sequential reads dominate, and the graph layout is chosen so that each query touches a small, predictable number of pages.

Question. Given 500M vectors of dim=768 and a budget of 1.5 TB SSD plus 64 GB RAM, describe how DiskANN handles the workload and estimate per-query SSD reads.

Input.

Quantity	Value
N (vectors)	500,000,000
dim	768
RAM budget	64 GB
SSD budget	1.5 TB

Code.

N = 500_000_000
dim = 768

# DiskANN stores vectors + graph edges on SSD.
# In-RAM: only a navigation index (top of the graph) + PQ codes for quick filtering.
on_disk_bytes_per_vec = dim * 4 + 32 * 8  # vector + ~32 edges of 8 bytes each
on_disk_total_gb = N * on_disk_bytes_per_vec / (1024**3)

# Per-query SSD reads: typically ~50-150 page reads of 4 KB each.
pages_per_query = 100
ssd_bytes_per_query = pages_per_query * 4096

# At ~250 microseconds per random 4 KB SSD read (NVMe), 100 reads = 25 ms.
ssd_latency_us = 250
p99_ms = pages_per_query * ssd_latency_us / 1000

print(round(on_disk_total_gb, 1), p99_ms)
# On-disk ~ 1551 GB; p99 ~ 25 ms

Step-by-step explanation.

Raw float32 storage: 500M · 768 · 4 = 1.42 TB. Fits the 1.5 TB SSD budget with headroom.
The DiskANN graph layout co-locates each node's neighbours on disk so a graph hop is one sequential 4 KB read. Typical per-query path length: 50–150 hops at 100M+ scale.
RAM usage: only the upper-layer navigation index plus per-node PQ codes for quick coarse filtering. At 500M vectors and a 16:1 PQ ratio, the PQ codes take 500M · 48 = 24 GB. Fits inside the 64 GB RAM budget with room for OS cache.
Per-query latency budget: ~100 NVMe page reads at 250 µs each = 25 ms p99. Add CPU time for distance computations: total p99 ~30–60 ms on commodity hardware.
Compared to in-memory HNSW at this scale (which would need ~1.5 TB RAM — economically impossible on a single host), DiskANN trades a 5–10× latency penalty for a 20× cost reduction.

Output.

Index	On-disk	RAM	p99 latency	Cost order
Raw float32 HNSW	—	~1.5 TB	5–15 ms	not buildable on commodity hardware
In-memory HNSW + PQ	—	~100 GB	8–20 ms	high-tier hardware
DiskANN	~1.5 TB	~64 GB	25–60 ms	commodity hardware

Rule of thumb. Reach for DiskANN (or a Pinecone serverless tier that is implemented similarly) when in-memory ANN would exceed your RAM budget by 5×. Below that threshold, in-memory HNSW with quantization is cheaper and faster.

Vector database interview question on index parameter tuning

A senior interviewer often opens with: "You have HNSW running with default parameters and your eval set shows recall@10 at 88 percent — you need 95 percent. How do you get there without rebuilding the index from scratch?"

Solution Using `ef_search` first, then build-time `M` only if needed

# 1) Cheap fix — raise ef_search at query time. No reindex.
client.set_ef(128)                            # was 64
results = client.knn_query(query_vec, k=10)

# 2) Sweep to find the smallest ef_search that crosses 95% recall.
for ef in [64, 96, 128, 160, 192, 256]:
    client.set_ef(ef)
    recall, p99_ms = benchmark(client, eval_set)
    print(ef, recall, p99_ms)

# 3) If even ef_search=256 falls short, the graph itself is under-built.
#    Rebuild with higher M (graph density) and higher ef_construction.
new_index = build_hnsw(
    vectors,
    M=32,                  # was 16 — denser graph
    ef_construction=400,   # was 200 — more candidates at insert time
)

Step-by-step trace.

ef_search	Recall@10	p99 latency	Action
64 (current)	88%	8 ms	start
96	92%	11 ms	not yet
128	95.5%	16 ms	hit
160	96.5%	22 ms	overshoot
256	97.5%	38 ms	wasted

The trace shows the recall curve crossing the 95 percent floor between ef_search=96 and ef_search=128. Picking 128 leaves a small recall buffer without exceeding the latency SLO. The "rebuild with higher M" branch is only triggered if even ef_search=256 cannot cross the floor — that is the signal the graph itself is too sparse.

Output:

Parameter	Old value	New value
ef_search	64	128
Recall@10	88%	95.5%
p99 latency	8 ms	16 ms
Reindex required?	no	no
M (build-time)	16	16 (unchanged)

Why this works — concept by concept:

ef_search is a runtime knob — no reindex required, no downtime, no application change. The fastest fix for a recall miss is always to raise ef_search first.
M is a build-time knob — M only changes when you rebuild the index. Reach for it when even maxing out ef_search cannot meet the recall floor, signalling the graph has too few neighbours per node.
Recall-latency Pareto curve — HNSW exposes a smooth curve: each step up in ef_search buys recall at the cost of latency. The sweet spot is the smallest ef_search that crosses the floor with margin.
Eval set discipline — every recall measurement must come from a frozen, representative query set with brute-force gold labels. Without it, "raise ef_search" becomes a superstition rather than an engineering knob.
Cost — raising ef_search is free at index time and pays linearly at query time. Doubling ef_search typically doubles per-query CPU but at most adds a few percent to recall — diminishing returns kick in fast.

SQL
Topic — indexing
Indexing problems (SQL)

Practice →

5. Ops, cost, and failure modes

Vector databases feel like Lego until you reindex 50 million rows — plan the day-2 ops up front

The mental model in one line: the day-1 cost of a vector database is memory and storage; the day-2 cost is reindex windows, multi-tenant isolation, embedding-model drift, and backup / replication semantics — and these decide whether the system survives its second year. Once you can name the four day-2 failure modes cold, the "we just need to pick a vendor and ship it" framing collapses into the real conversation: who owns the reindex on Friday at 3 AM?

Memory sizing — the only formula you need.

Baseline. bytes ≈ 4 · dim · N for float32 vectors. At dim=1536 and N=10M, that is ~60 GB just for the vectors.
Index overhead. HNSW adds ~8 · M · N bytes for the graph (typically 1–5 percent of the vector payload). IVFFlat adds the centroid table (negligible).
Quantization. Scalar quantization cuts the vector payload 4×; product quantization cuts it 16–32×.
Sample table. At dim=768 / dim=1536 / dim=3072, plan for the following per-million-vector RAM cost.

dim	float32 RAM / 1M vectors	int8 RAM / 1M vectors	PQ (16×) RAM / 1M vectors
768	3.1 GB	0.77 GB	0.19 GB
1536	6.1 GB	1.5 GB	0.38 GB
3072	12.3 GB	3.1 GB	0.77 GB

Index build cost vs query cost.

Build cost is one-time per generation of the index. HNSW build is roughly O(N · ef_construction · log N) — at 10M vectors and ef_construction=200, expect 30–90 minutes on a single CPU.
Query cost is per-request. The whole point of the build is to make the query cheap. Optimise build for off-hours; optimise query for the SLO.
Reindex windows. Every embedding-model upgrade triggers a full re-embed and a full reindex. Plan for one a quarter at minimum.

Multi-tenancy — namespaces vs collections vs partition keys.

Pinecone. Namespaces inside an index. Cheap, isolated at query time, share an index resource pool. Default for SaaS-style tenancy.
Qdrant. Payload field tenant_id plus a filter. Or per-tenant collection if you need hard isolation.
Weaviate. Multi-tenant collections (a first-class concept in recent versions) or per-class isolation.
pgvector. Add tenant_id as a column with a B-tree index; combine with the HNSW for hybrid pushdown. Or use Postgres partitioning by tenant_id for stronger isolation at higher ops cost.

Metadata filter pushdown — pre-filter vs post-filter recall traps.

Pre-filter. The vendor pushes the predicate down into the ANN traversal — only candidates that match the filter are inspected. Recall stays at the index baseline.
Post-filter. The ANN returns the global top-K, then the filter is applied after. Selectivity below ~10 percent causes silent recall loss — the user sees fewer than K results.
Vendor differences. Pinecone, Qdrant, Weaviate all support proper pre-filter pushdown. pgvector's behaviour depends on the query plan — sometimes the planner does push the predicate below the HNSW index, sometimes it does not. Verify with EXPLAIN ANALYZE on every new query shape.

Drift — embedding model upgrades.

Trigger. New encoder model ships (e.g. OpenAI text-embedding-3-small → next generation). Old vectors and new query vectors live in different semantic spaces — recall collapses if mixed.
Mitigation. Full re-embed of the corpus plus a blue / green collection swap. Two collections run in parallel during the swap; the application points at one or the other atomically.
Cost. A full re-embed at 10M vectors and $0.02 per 1K tokens at typical chunk sizes is ~$100–$500. Cheaper than a recall regression.

Backup, snapshot, replication semantics.

Pinecone. Replication and backup are vendor-managed; no team responsibility.
Qdrant. Snapshot endpoint writes a consistent copy to disk; replication via Raft consensus in distributed mode.
Weaviate. Backup module to S3 / GCS; replication factor configurable per class.
pgvector. Standard Postgres backup (pg_basebackup, WAL archiving) — covers vectors automatically because they are just another column.

Common interview probes on day-2 ops.

"How would you handle an embedding-model upgrade?" — blue / green collection swap with dual-write during the transition. Compare metrics on a frozen eval set before atomically switching.
"What is the worst-case reindex window?" — full re-embed + full ANN rebuild at the target scale. At 50M vectors, plan for 6–24 hours on a single-host build; parallelise if shorter is required.
"How do you do multi-tenancy without exploding cost?" — namespaces (Pinecone) or filter-on-payload (Qdrant) share the index resource pool, so cost scales with total vectors, not tenant count.
"What is the silent failure mode of metadata filters?" — post-filter on selective predicates drops recall to single digits. Pre-filter pushdown is the only correct implementation.

Worked example — sizing a 25M-vector workload at dim=1536

Detailed explanation. A team has 25M document chunks at dim=1536 and is choosing between float32 HNSW and HNSW + scalar quantization on a 64 GB-RAM replica. The right answer is a 5-minute calculation, not a vendor benchmark.

Question. Compute the RAM footprint for both options and decide which fits the 64 GB budget with at least 30 percent headroom.

Input.

Quantity	Value
N	25,000,000
dim	1536
RAM budget	64 GB
Headroom target	≥ 30%
HNSW M	16

Code.

N = 25_000_000
dim = 1_536
M = 16

# Float32 HNSW
fp32_vec_gb = N * dim * 4 / (1024**3)
hnsw_graph_gb = N * M * 8 * 2 / (1024**3)  # bidirectional edges
fp32_total_gb = fp32_vec_gb + hnsw_graph_gb

# Scalar quantization (int8) HNSW
int8_vec_gb = N * dim * 1 / (1024**3)
int8_total_gb = int8_vec_gb + hnsw_graph_gb

print(round(fp32_total_gb, 1), round(int8_total_gb, 1))
# fp32 ~ 149.0 GB (does not fit)
# int8 ~ 41.5 GB (fits with ~35% headroom)

Step-by-step explanation.

Float32 vector payload: 25M · 1536 · 4 = 153 GB. Massively over budget.
HNSW graph edges: 25M · 16 · 8 · 2 = 6.4 GB — modest compared to the vector payload.
Float32 HNSW total: ~149 GB. Even before considering query memory or OS overhead, the float32 option requires three replicas to hold the corpus — tripling cost.
Scalar quantization (int8) vectors: 25M · 1536 · 1 = 36 GB. Plus the same graph overhead: ~42 GB total.
Headroom check: (64 - 42) / 64 = 34%. Just above the 30 percent target. Acceptable.

Output.

Option	Total RAM	Fits 64 GB?	Headroom
Float32 HNSW	~149 GB	no	-134%
Int8 HNSW (scalar quantization)	~42 GB	yes	34%
PQ 16× HNSW	~10 GB	yes	84%

Rule of thumb. Default to scalar quantization when the float32 vector payload exceeds 50 percent of the RAM budget. Reach for product quantization when the float32 payload exceeds 5× the RAM budget or when total vectors exceed 100M.

Worked example — blue / green collection swap on a model upgrade

Detailed explanation. Upgrading the embedding model is the single riskiest day-2 operation. Mixing old and new vectors silently destroys recall — they live in different semantic spaces. The blue / green pattern keeps the old collection live while the new one fills, then atomically switches the application.

Question. Describe the steps to migrate from embedding model v1 (dim=768) to model v2 (dim=1536) for a 20M-vector corpus, with no downtime and no recall regression.

Input.

Stage	Old (`docs_v1`)	New (`docs_v2`)
dim	768	1536
Encoder	model v1	model v2
Production traffic	yes	no (offline)
Vectors	20M	0 → 20M

Code.

# 1) Create the new collection.
vector_db.create_collection("docs_v2", dim=1536, metric="cosine")

# 2) Backfill — re-embed every doc with the new model.
for batch in chunks(all_docs, size=256):
    vecs = model_v2.embed_batch([d.text for d in batch])
    vector_db.upsert("docs_v2", list(zip(batch, vecs)))

# 3) Dual-write — every new doc gets embedded with both models.
def on_new_doc(doc):
    v1 = model_v1.embed(doc.text)
    v2 = model_v2.embed(doc.text)
    vector_db.upsert("docs_v1", [(doc, v1)])
    vector_db.upsert("docs_v2", [(doc, v2)])

# 4) Compare metrics on a frozen eval set.
metrics_v1 = evaluate("docs_v1", eval_set, model_v1)
metrics_v2 = evaluate("docs_v2", eval_set, model_v2)
assert metrics_v2.recall >= metrics_v1.recall - 0.005   # within 0.5 pts

# 5) Atomic application switch.
config.set("active_collection", "docs_v2")
config.set("active_encoder", "model_v2")

# 6) Soak period, then drop the old collection.
time.sleep(soak_window)
vector_db.drop_collection("docs_v1")

Step-by-step explanation.

Stage 1 creates a sibling collection with the new dimension. The old collection keeps serving traffic — no user impact yet.
Stage 2 backfills the new collection by re-embedding the entire corpus with the new model. This is the expensive step — for 20M chunks at $0.02 per 1K tokens, plan for $100–$500.
Stage 3 dual-writes new docs into both collections so neither one falls behind during the backfill. This phase ends when both collections have the same row count.
Stage 4 evaluates retrieval quality on a frozen eval set with brute-force gold labels. If recall@10 on the new collection is within 0.5 points of the old one (or better), the switch is safe.
Stage 5 is the atomic switch — the application config changes the active collection pointer. Traffic immediately routes to docs_v2. The old collection is still warm; rollback is one config change away.
Stage 6 is the soak — let the new collection serve traffic for a few days, monitor query patterns, then drop the old collection to reclaim the resources.

Output.

Phase	Old collection	New collection	Application points at
1. create	full	empty	old
2. backfill	full	filling	old
3. dual-write	full	full	old
4. eval pass	full	full	old
5. switch	full	full	new
6. drop old	dropped	full	new

Rule of thumb. Always blue / green on model upgrades. Never reindex in place — the cost of a regression caught in production is higher than the cost of the extra collection. The pattern also works for index parameter changes (M, ef_construction) and metric changes (cosine ↔ dot product ↔ L2).

Worked example — multi-tenant namespace isolation in Pinecone

Detailed explanation. SaaS workloads run hundreds or thousands of tenants. A naive design uses one index per tenant — index-creation cost dominates, ops becomes a nightmare. The right pattern is one index plus a namespace per tenant; the namespace is a logical partition inside the shared index resource pool.

Question. A SaaS product has 5,000 customer tenants, each with 1,000 to 100,000 documents. Compare "one index per tenant" vs "one index plus a namespace per tenant" on cost and ops surface.

Input.

Strategy	Indexes	Namespaces	Resource provisioning
One index per tenant	5,000	0	per-tenant capacity
One index plus namespace per tenant	1	5,000	shared capacity

Code.

# WRONG — one index per tenant (5000 indexes; vendor limits + cost)
for t in tenants:
    pinecone.create_index(f"docs-{t.id}", dimension=1536)

# CORRECT — one index, namespace per tenant
pinecone.create_index("docs", dimension=1536)

def upsert(tenant_id, doc, vec):
    pinecone.upsert(
        index_name="docs",
        namespace=f"tenant-{tenant_id}",
        vectors=[(doc.id, vec)],
    )

def query(tenant_id, query_vec, k=10):
    return pinecone.query(
        index_name="docs",
        namespace=f"tenant-{tenant_id}",
        vector=query_vec,
        top_k=k,
    )

Step-by-step explanation.

Per-index strategy. Each tenant gets its own index — 5,000 separate ANN structures. Most vendors limit indexes per project (typically 20–200). Even where unlimited, every index has a per-index resource floor (memory, control-plane overhead) that does not scale down to small tenants.
Namespace strategy. One index holds all tenants; the namespace string is appended to every upsert and query. The ANN is shared; the namespace acts as a partition key that the query engine filters on before scoring.
Cost. Per-index strategy pays a fixed cost per tenant; total cost scales with tenants × per-index floor. Namespace strategy pays one fixed cost; total cost scales with total vectors only.
Isolation. Both strategies prevent cross-tenant queries from leaking data. The namespace strategy depends on the vendor enforcing the namespace boundary correctly — this is the security-critical assumption to verify.
Migration cost. Moving from per-index to namespace later is painful (re-upsert everything). Choose namespace from day one for any SaaS workload.

Output.

Axis	One index per tenant	One index plus namespace
Indexes provisioned	5,000	1
Vendor index-limit risk	high	none
Per-tenant overhead	fixed floor per index	negligible
Cost at scale	grows with tenant count	grows with total vectors
Cross-tenant isolation	hard wall	logical wall

Rule of thumb. Default to one index with a namespace per tenant for any multi-tenant SaaS workload. The "one index per tenant" pattern is only ever right when tenants have wildly different SLOs and the team is willing to pay the per-index floor 5,000 times.

Vector database interview question on day-2 ops planning

A senior interviewer often closes the loop with: "Walk me through your runbook for the day the embedding-model vendor releases a new generation and a key customer asks if you will support it. What happens in week one, week two, week three?"

Solution Using a four-week blue / green upgrade plan with eval gates

# Week 1 — measure.
- Build an eval set: 500-1000 (query, gold-doc) pairs across all tenants.
- Baseline recall@10 and p99 latency on the current collection.
- Estimate full-re-embed cost: tokens · price-per-1K · 1.2 (safety margin).

# Week 2 — prepare.
- Create offline collection `docs_v2` with the new model's dimension and metric.
- Start the backfill: batch encode + upsert. Throttle to stay within rate limits.
- Begin dual-write at the producer layer (all new docs to both collections).

# Week 3 — evaluate + cutover.
- Run the eval set against `docs_v2` and compare to baseline.
- Gate: recall@10 must be within 0.5 points; p99 latency must be within 10%.
- If gate passes, flip the application pointer atomically.
- Keep `docs_v1` warm for a 5-day soak.

# Week 4 — clean up.
- Drop `docs_v1` after the soak.
- Update runbooks, alert thresholds, and dashboards to point at `docs_v2`.
- Post-mortem: cost actually paid, latency curve, recall delta.

Step-by-step trace.

Week	Action	Risk if skipped
1	freeze eval set + baseline	no measurable rollback criterion
2	offline `docs_v2` + dual-write	new docs missing from v2 on cutover
3	eval gate + atomic switch	silent recall regression in prod
4	soak + cleanup	resource leak; orphan collection

The trace highlights that each step is an insurance payment against a specific failure mode. Skipping any step exchanges a small predictable cost for a large unpredictable one.

Output:

Metric	Before (`docs_v1`)	After (`docs_v2`)	Delta
Recall@10	94.0%	95.2%	+1.2 pts
p99 latency	18 ms	17 ms	-1 ms
Re-embed cost	—	$320 (one-time)	—
Downtime	—	0 seconds	—

Why this works — concept by concept:

Eval set first — without a frozen, representative (query, gold-doc) set, "recall went up" is a vibe, not a measurement. Build the eval set before the upgrade so the baseline is honest.
Dual-write during backfill — closes the race window where new docs arrive after the backfill starts but before the cutover. Without dual-write, those docs are missing from v2 on cutover day.
Atomic application switch — the application reads a single config knob (active_collection) per query. Changing it is one write to one config row; rollback is just as cheap.
Soak window — production traffic exposes failure modes that no eval set covers (long-tail queries, language mix, time-of-day patterns). The soak window catches them while docs_v1 is still recoverable.
Cost — re-embed is a one-time variable cost (tokens); the extra storage during cutover is a one-time fixed cost (a second collection's worth of RAM / SSD). Both are bounded and small relative to the cost of a recall regression.

SQL
Topic — database
Database ops problems (SQL)

Practice →

Cheat sheet — vector DB recipes

HNSW default parameters. M = 16, ef_construction = 200, ef_search = 64. Tune ef_search first (runtime); rebuild with higher M only if even ef_search = 256 falls short of recall.
pgvector HNSW index DDL. CREATE INDEX docs_hnsw ON docs USING hnsw (embedding vector_cosine_ops) WITH (m = 16, ef_construction = 200); — choose the operator class (vector_cosine_ops, vector_l2_ops, vector_ip_ops) to match the query operator (<=>, <->, <#>).
Hybrid query — vector + metadata filter. Push the predicate into the ANN call: vector_db.search(q_vec, top_k=10, filter={"tenant_id": t, "lang": l}). Verify it is a pre-filter (not post-filter) with the vendor's explain mechanism.
Reranker layer. Pull top-50 with the ANN; rerank with a cross-encoder model on the (query, candidate) pairs; return top-10. Lifts precision by 5–15 points for ~50 ms extra latency.
Memory baseline. bytes ≈ 4 · dim · N for float32 vectors. At dim=1536 and N=1M, that is ~6 GB per million vectors. Scalar quantization cuts this 4×; product quantization cuts it 16–32×.
Drift checklist on a model upgrade. Build offline collection → backfill via re-embed → dual-write new docs → eval gate on frozen set → atomic application switch → soak → drop old collection.
Pre-filter vs post-filter. Always confirm your vendor pushes the filter into the index. Selective filters (< 10 percent of rows match) break post-filter strategies silently.
Multi-tenant pattern. Default to one index plus a namespace (or payload field) per tenant. Per-tenant indexes are an anti-pattern at SaaS scale.
Similarity metric contract. Choose one metric, write it down, enforce normalisation at the producer, and align the index operator class and query operator. Cosine is the safe default for text embeddings.
Build cost. HNSW build is O(N · ef_construction · log N); at 10M / ef_construction=200, expect 30–90 minutes on a single CPU. Plan for this every embedding-model upgrade.
Quantization combo recipe. PQ on the index for compression; rerank the top-50 with the original float32 vectors. Recovers most of the recall loss for negligible extra cost.
DiskANN trigger. Reach for DiskANN (or a vendor tier that uses it under the hood) when float32 + HNSW would exceed 5× your RAM budget. Below that, in-memory HNSW + quantization is cheaper and lower latency.
pgvector migration trigger. Move off pgvector when corpus crosses 15M vectors, sustained QPS crosses 200, or tenants cross ~100. Write the threshold down so the migration is a planned event, not a fire drill.

Frequently asked questions

Do I need a vector database or is pgvector enough?

If you already run Postgres, start with pgvector — adding a vector column and an HNSW index is CREATE EXTENSION vector plus one CREATE INDEX. It comfortably serves up to ~10 million vectors per replica at sub-25 ms p99 with M = 16 and ef_search = 64, and it lets you join vectors against your existing relational tables (orders, users, products) without any new infrastructure. The migration trigger to a dedicated vector database (Pinecone, Weaviate, Qdrant) is when your corpus crosses ~15 million vectors or sustained query throughput crosses ~200 QPS or you start running multi-tenant SaaS with 100+ namespaces. Until any of those three thresholds flips, pgvector is the cheapest option by a wide margin.

What's the difference between HNSW and IVFFlat?

HNSW (Hierarchical Navigable Small World) builds a multi-layer graph where each vector has a fixed number of neighbour edges per layer; queries do a greedy graph walk and return the K nearest. IVFFlat (Inverted File with Flat quantizer) clusters vectors into nlist centroids via k-means; queries inspect the nprobe closest clusters and brute-force-search their members. HNSW has lower p99 latency at moderate scale and supports incremental inserts without a training step; IVFFlat uses less memory and is cheaper to write but has higher tail latency and needs a one-time training step on a representative sample. Default to HNSW unless RAM is the binding constraint — then consider IVFFlat, often combined with scalar or product quantization.

How do I size memory for a vector index?

The baseline formula is bytes ≈ 4 · dim · N for float32 vectors — at dim=1536 (OpenAI text-embedding-3-small) and 10 million vectors, that is ~60 GB just for the payload. HNSW adds a graph overhead of roughly 8 · M · N bytes (about 1–5 percent of the vector payload at typical M=16). Scalar quantization (int8) cuts the vector payload by 4× with a 0.5–3 point recall hit; product quantization (m=64 sub-vectors) cuts it by 16–32× with a 3–8 point recall hit (recoverable to within 0.5 points by reranking the top-50 with original float32 vectors). Plan for 30 percent RAM headroom above the index footprint for query buffers and OS cache.

Can a vector database replace my search engine?

Almost never — best-in-class retrieval is hybrid. Pure vector search underperforms BM25 keyword search by 10–30 points on exact-match queries ("error code 500", "iPhone 15 Pro Max") because dense embeddings smear semantically related but lexically different documents. Pure BM25 underperforms vector search on semantic queries ("why won't my user log in") because keyword matching misses paraphrase. The production pattern is to run both, then fuse the rankings either by reciprocal rank fusion or by a learned reranker. Weaviate exposes this natively via the with_hybrid(alpha=0.5) API; with Pinecone, Qdrant, or pgvector you wire the two retrievers together at the application layer.

What happens when I change embedding models?

Old vectors (encoded with model v1) and new query vectors (encoded with model v2) live in different semantic spaces — mixing them silently destroys recall. The mandatory pattern is a blue / green collection swap: create an offline collection docs_v2 with the new model's dimension, backfill it by re-embedding the entire corpus, dual-write every new document into both collections during the backfill, evaluate retrieval on a frozen (query, gold-doc) eval set, gate on "recall@10 within 0.5 points of baseline," then atomically flip the application pointer to docs_v2 and soak for 5 days before dropping docs_v1. Budget the variable cost (re-embed tokens) — at 10 million chunks and $0.02 per 1K tokens, expect $100–$500 per upgrade. Plan for at least one upgrade per quarter.

Pinecone vs Weaviate vs Qdrant — which should I pick?

Pinecone is fully managed serverless — minutes to first query, zero ops, predictable per-vector pricing; pick it when team size is small and there is no on-prem mandate. Weaviate is OSS plus managed cloud with built-in RAG modules (text2vec-openai, generative search) and first-class hybrid search via with_hybrid(alpha); pick it when you want batteries included and a GraphQL surface. Qdrant is OSS plus managed cloud in Rust with the strongest payload-filter language and on-disk index support; pick it for low-latency online search with heavy structured filtering, especially when on-prem is required. All three handle 1M–500M vectors comfortably; above that, Pinecone (sharded) or DiskANN-class OSS solutions are the only realistic options.

Practice on PipeCode

Drill the database design practice library → for the schema + index + query-plan triple that every vector store ultimately reduces to.
Rehearse on indexing problems → to internalise the HNSW / IVFFlat / B-tree mental model interviewers expect.
Sharpen system design drills → for the topology, sharding, and replication arc this post walked.
Layer the aggregation library → for the metadata-filter pushdown reasoning that shows up in every hybrid-retrieval probe.
Stack the joins practice library → for the SQL-side composition between vector retrieval and relational facts in pgvector.
For the broader surface, read top data engineering interview questions →.
Stack the prerequisites with the only 5 skills you need to become a data engineer →.
Sharpen the data-platform axis with the ETL system design course →.
For long-form schema craft, work through data modelling for DE interviews →.

Pipecode.ai is Leetcode for Data Engineering — every vector database recipe above ships with hands-on practice rooms where you draw the topology, write the metadata-filter pushdown, and rehearse the blue / green collection swap against graded scenarios. PipeCode pairs every reading with 450+ DE-focused problems and a real-time scoring engine, so you never have to wonder whether your sizing math for a 100-million-vector workload actually matches what a senior interviewer expects to hear.

Practice database design now →
Indexing drills →

MetricFlow & dbt Metrics: Single Source of Truth for KPIs

Gowtham Potureddi — Tue, 16 Jun 2026 13:02:42 +0000

dbt metrics look like a small YAML feature to a junior analytics engineer — senior data platforms know they are actually a strategic bet on semantic metrics, the dbt metrics layer, and a kpi single source of truth that lives in version control rather than scattered across six BI tools. The result is the single largest leverage move in analytics engineering since dbt models replaced stored procedures: every "active user," "MRR," and "gross margin" number resolves to one definition compiled by metricflow, the engine inside dbt that turns metric definitions into dialect-specific SQL at query time.

This guide walks through the dbt semantic models that feed MetricFlow, the anatomy of a metric definition, the query flow from metric to Tableau / Hex / Mode / Python via the Semantic Layer API, and the migration playbook for moving an organisation off "calculated fields" in BI tools and into a governed metric stores layer. Each H2 ends with an interview-style answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why the pattern works.

When you want hands-on reps alongside the reading, drill the aggregation practice library →, rehearse on cumulative-snapshot problems →, and stack the conditional aggregation drills → — the three SQL muscles every dbt metric compiles down to.

On this page

The KPI drift problem — why every dashboard says a different number
MetricFlow architecture inside dbt
Anatomy of a metric definition
From metric definition to BI / Python query
Migration playbook — from BI views to dbt semantic models
Cheat sheet — dbt metrics recipes
Frequently asked questions
Practice on PipeCode

1. The KPI drift problem — why every dashboard says a different number

Definition sprawl: one metric, seven definitions, zero version control

The one-sentence invariant: a KPI without a version-controlled definition is a rumour, not a metric. The moment "active user" lives only in a Tableau calculated field, a Looker measure, a Hex notebook, and three Slack screenshots, the organisation has seven answers to one question — and no way to reconcile them without an archeology project.

The five faces of KPI drift.

Definition sprawl. "Active user" gets defined as "logged in last 30 days" by product, "any event in last 7 days" by growth, "any purchase in last 90 days" by finance, "non-cancelled subscription" by RevOps. Every team is right inside its own dashboard. The board slide that aggregates all four is wrong by definition.
Logic lives outside version control. A Tableau calculated field is a string inside a workbook XML, edited by whoever has the licence. There is no PR, no diff, no test. The June board number changes between Monday and Tuesday because someone clicked "edit."
The same SQL, three slightly different filters. Every analyst writes a fresh WHERE clause around the "active" check. Six months later you have three production dashboards each off by 0.3%, 1.1%, and 4.8% — and no idea which is canonical.
Tool lock-in. When the org migrates from Tableau to Hex, every calculated field has to be re-translated. The "migration" stretches across a quarter because the BI layer hides business logic the analytics engineers never owned.
Audit pain. Finance asks "how was MRR computed in Q2?" and the answer is a screenshot, not a commit hash. SOX-style audits become impossible because the metric is not a code artifact.

The cost in one line. A senior data leader at a 500-person SaaS reported that the team spent 30% of weekly cycles reconciling KPI variants — pulling Looker against finance, finance against Hex, Hex against a CFO dashboard. After consolidating on the dbt semantic layer, that number dropped to under 4% within a quarter.

Why a "BI tool view" does not fix this.

A common first attempt is to centralise calculated fields inside one BI tool ("we'll just standardise on Looker measures"). This solves consumption for users inside Looker — but every Python notebook, every Hex board, every reverse-ETL job back into Salesforce still has to re-derive the metric from raw warehouse columns. The "view" is the view of one consumer; the metric needs to live one layer below the BI tool.

The shift in mental model.

The move is from "table-of-truth" ("the gold mart table is canonical") to "metric-of-truth" ("the metric definition is canonical, and every consumer asks for it by name"). The table is still there — it is the raw material — but the contract is now the metric.

What interviewers and platform leads listen for.

Do you frame KPI drift as a governance problem, not a SQL problem? — senior signal.
Do you mention version control on metric definitions before mentioning any specific tool? — required answer.
Do you separate measures (raw aggregations) from metrics (composed business definitions)? — semantic-layer literacy.
Do you reach for a PR-required workflow for metric edits? — governance maturity.

Detailed explanation — the four symptoms in one query

Detailed explanation. Most teams discover KPI drift the painful way: a board slide that says MRR is $2.41M while the CFO dashboard says $2.38M while the finance Hex board says $2.45M. The decomposition is almost always the same set of four mismatches — and once you can name them, you can map each to the MetricFlow primitive that prevents it.

Question. Given three "MRR" SQL snippets pulled from three different BI tools (each authored by a different analyst), enumerate the four sources of drift and explain which MetricFlow primitive eliminates each.

Input. Three snippets pulled from a real org:

-- Tableau calculated field A
SUM(IF [plan_status] = 'active' THEN [plan_price] END)

-- Hex SQL cell B
SELECT SUM(price_usd) FROM subscriptions
WHERE status = 'active' AND price_usd > 0;

-- Mode SQL block C
SELECT SUM(CASE WHEN canceled_at IS NULL THEN plan_amount END)
FROM subscriptions WHERE start_date <= CURRENT_DATE;

Code. The same MRR metric in MetricFlow:

# semantic_models/subscriptions.yml
semantic_models:
  - name: subscriptions
    model: ref('fct_subscriptions')
    entities:
      - name: subscription_id
        type: primary
      - name: customer_id
        type: foreign
    dimensions:
      - name: started_at
        type: time
        type_params:
          time_granularity: day
      - name: plan_tier
        type: categorical
    measures:
      - name: mrr_amount
        agg: sum
        expr: plan_price_usd
        agg_time_dimension: started_at

# metrics/mrr.yml
metrics:
  - name: mrr
    label: Monthly Recurring Revenue
    type: simple
    type_params:
      measure: mrr_amount
    filter: "{{ Dimension('subscription_id__is_active') }}"

Step-by-step explanation.

Snippet A uses the column plan_status = 'active'. Snippet B uses status = 'active' AND price_usd > 0. Snippet C uses canceled_at IS NULL AND start_date <= CURRENT_DATE. Drift source #1 — disagreement on what "active" means. The MetricFlow primitive that fixes it is the filter expression on the metric, owned by the platform team and reviewed via PR.
Snippet A measures plan_price; Snippet B measures price_usd; Snippet C measures plan_amount. Drift source #2 — three different columns for the "money" measure. MetricFlow fixes it with one declared measure (mrr_amount) sourcing plan_price_usd once.
Snippets A and B carry no time grain. Snippet C silently picks start_date as the time grain. Drift source #3 — implicit time grain. MetricFlow forces an explicit agg_time_dimension, so every consumer agrees on the time anchor.
None of the three snippets carries a currency_code filter. The org has EUR and GBP plans in the same column. Drift source #4 — silent currency mixing. MetricFlow lets you add a dimensions currency_code and require it on the saved query, so the consumer must specify "USD only" or accept the multi-currency aggregate.

Output.

Drift source	Symptom in numbers	MetricFlow primitive
1. "Active" definition disagrees	`$2.41M` vs `$2.38M` vs `$2.45M`	metric `filter`
2. Money column varies	finance vs growth mismatch	`measure` (one source)
3. Implicit time grain	"as of when?" ambiguity	`agg_time_dimension`
4. Currency mixing	EUR rows quietly summed with USD	explicit `dimension`

Rule of thumb. Before locking down a metric in YAML, name the four drift axes (definition, measure column, time grain, hidden dimension) for that specific KPI. If you cannot answer all four in one sentence each, the metric is not ready to ship — keep the BI calculated field one more sprint and refine.

Detailed explanation — the audit hash that ends the "screenshot economy"

Detailed explanation. Once a metric is a YAML file in the dbt project, every change has a commit hash. Finance can ask "how was MRR computed for the Q2 close?" and the answer is a git log line, not a screenshot. Auditors love this because the metric becomes a code artifact — diffable, reviewable, and reproducible.

Question. Show how a one-line change to the MRR filter — from "active" to "active and non-trial" — appears in version control, and what the downstream consumers automatically inherit.

Input. A single line edit:

- filter: "{{ Dimension('subscription_id__is_active') }}"
+ filter: "{{ Dimension('subscription_id__is_active') }} AND {{ Dimension('subscription_id__is_trial') }} = false"

Code. The PR description (what a reviewer reads):

title:  metrics/mrr.yml — exclude trial subscriptions
why:    finance Q2 close decision (2026-06-12 sync)
impact: MRR drops ~$48k (~1.9%); affects exec dashboard,
        finance Hex board, sales-ops Salesforce sync.
ticket: FIN-742

Step-by-step explanation.

The metric definition is one file. The diff is two lines (one removed, one added). The PR carries a written reason, a Jira link, and an estimated dollar impact.
Every downstream consumer — Tableau, Hex, Mode, the Python notebook, the reverse-ETL into Salesforce — pulls MRR by name from the Semantic Layer API. They inherit the new filter on the next refresh; no per-consumer change required.
Two months later, finance asks "what changed in MRR on 2026-06-12?" The answer is a git show of that commit. Hash, author, reviewer, ticket, impact estimate — every audit field is already there.
The contrast: in the old world, the same change would have been made independently in Tableau, Hex, Mode, the Salesforce SOQL, and the CFO Sheet — five edits, each by a different person, each invisible to the others, each prone to typo. With the dbt metrics layer, one PR replaces five edits and the audit trail is automatic.

Output.

Old world	New world (dbt metrics)
5 hand edits, 5 tools	1 PR, 1 hash
screenshot for audit	`git show` for audit
1.9% impact undocumented	impact in PR body
~2 weeks discovery delay	next refresh (minutes)

Rule of thumb. Adopting the dbt semantic layer is as much a cultural shift as a technical one. The team has to commit to "no calculated fields in BI tools" — but in exchange, they get version control, PR review, and audit hashes on every KPI, for free.

Interview question on the KPI drift cost-benefit

A senior staff interviewer often opens with: "Your CFO complains that finance and product report different active-user counts every month. Walk me through how you would diagnose the drift, decide whether to consolidate on the dbt semantic layer, and what the migration risks are." It blends governance literacy, MetricFlow architecture, and migration sequencing into a single answer.

Solution Using a drift audit followed by a phased semantic-layer migration

-- 1) Drift audit — count "active users" by every known definition
WITH defs AS (
    SELECT 'product (login 30d)'  AS def_name,
           COUNT(DISTINCT user_id) AS active_users
    FROM events
    WHERE event_type = 'login'
      AND event_ts >= CURRENT_DATE - INTERVAL '30 days'
    UNION ALL
    SELECT 'growth (any event 7d)',
           COUNT(DISTINCT user_id)
    FROM events
    WHERE event_ts >= CURRENT_DATE - INTERVAL '7 days'
    UNION ALL
    SELECT 'finance (purchase 90d)',
           COUNT(DISTINCT user_id)
    FROM orders
    WHERE order_ts >= CURRENT_DATE - INTERVAL '90 days'
)
SELECT def_name, active_users,
       active_users - MIN(active_users) OVER () AS gap_vs_min,
       100.0 * (active_users
                - MIN(active_users) OVER ())
                / MIN(active_users) OVER ()      AS gap_pct
FROM defs
ORDER BY active_users DESC;

Step-by-step trace.

Definition	active_users	gap_vs_min	gap_pct
growth (any event 7d)	42,180	14,910	54.7%
product (login 30d)	31,205	3,935	14.4%
finance (purchase 90d)	27,270	0	0.0%

The audit makes the drift visible in dollars. A 54.7% gap between the smallest and largest "active user" number is the conversation-starter the platform team takes to the CFO: "We have three legitimate definitions, but only one can headline the board slide. Let's pick — and lock it down in MetricFlow."

Output:

Step	Action
1	Audit drift across all known definitions (the query above)
2	Run a 1-week working group to pick the canonical definition
3	Author `metrics/active_users.yml` with the canonical filter
4	Cut over consumer-by-consumer (Tableau, Hex, Mode, Python)
5	Lock down BI calculated fields after one quarter dual-running

Why this works — concept by concept:

Drift audit before debate — quantifying the gap in one query turns a philosophical argument ("what does active mean?") into a business trade-off ("which of these three numbers do we want on the board?"). Always lead with data.
One canonical filter, owned by platform — the working-group output is a single boolean expression that becomes the filter on the metric. From that point on, every consumer asks for active_users by name.
Dual-run before cut-over — ship the new metric alongside the old BI calculated fields for one quarter. Compare nightly. Cut over only after three monthly closes match within tolerance.
Governance, then tools — the MetricFlow YAML is the artifact of the governance decision, not the cause of it. Without the working group, the YAML is just another silo.
Lock-down phase — after cut-over, disable BI-tool calculated-field editing (Tableau permissions, Looker LookML reviewer gates). Otherwise the drift re-emerges within two quarters.
Cost — one focused quarter of platform-team time for the top 20 KPIs; the marginal cost per additional metric thereafter is the time to write 30 lines of YAML and one PR.

SQL
Topic — aggregation
Aggregation problems (SQL)

Practice →

2. MetricFlow architecture inside dbt

MetricFlow is the SQL compiler for your metrics — semantic models in, dialect-specific SQL out

The mental model in one line: MetricFlow is a query-time SQL compiler that turns metric YAML into the dialect of the underlying warehouse, with semantic models as the building blocks and the MetricFlow server as the front door. Once you can describe the five layers — warehouse, dbt models, semantic models, metrics, MetricFlow server — you can debug any "wrong number" by pointing at the layer that owns the bug.

The five layers.

Warehouse tables. The raw source — orders, users, events, subscriptions. Owned by ingestion. Schema may change underneath you.
dbt models. The cleaned, conformed marts — fct_orders, dim_users. Already version-controlled, already tested. The semantic layer builds on top of these, not on raw tables.
Semantic models. Declarative YAML that wraps a dbt model with entity keys, dimensions, and measures. The "noun layer."
Metrics. Composed expressions over measures — simple, ratio, cumulative, derived. The "verb layer."
MetricFlow server. The query engine. Accepts metric+dimension requests from BI tools or Python, plans the join graph across semantic models, compiles to dialect-specific SQL, and returns the result set.

How it differs from LookML / Cube / Tableau models.

One model, many consumers. LookML lives inside Looker; the metric definitions never leave the BI tool. MetricFlow lives inside dbt; every consumer (Tableau, Hex, Mode, Python, reverse-ETL) reads from the same source.
Version control by default. MetricFlow YAML lives in the dbt repo, so it shares the dbt PR / CI / docs workflow. LookML has its own. Tableau has none.
Compile to dialect at query time. MetricFlow does not pre-materialise the metric — it generates SQL on demand, optimised for whatever warehouse you ship the query to.
Build-time tests, query-time joins. dbt tests the upstream models at build time. MetricFlow resolves joins at query time using the declared entity keys, so a new dimension request does not require a new mart.

The compilation model.

When a BI tool asks "give me MRR by plan_tier for the last 6 months," MetricFlow does the following:

Look up mrr in the metrics registry. Find the underlying measure (mrr_amount) on the subscriptions semantic model.
Look up plan_tier as a dimension. Locate it on the same semantic model — no join needed.
Look up the time grain. Use the agg_time_dimension (started_at) and bucket by month.
Generate dialect-specific SQL — Snowflake DATE_TRUNC('month', started_at), BigQuery DATE_TRUNC(started_at, MONTH), Postgres date_trunc('month', started_at).
Execute against the warehouse, stream results back to the consumer, optionally cache the saved query.

The role of saved queries.

A saved_query is a pre-bound combination of metric+dimension+filter that the platform team blesses for repeated use. Examples: "MRR by plan_tier monthly for the last 12 months," "DAU by region daily for the last 30 days." Saved queries become the artefact that consumers point dashboards at — they can be cached, scheduled, and exposed via the API as first-class objects.

Common architecture probes.

"Where does MetricFlow sit relative to dbt models?" — between the dbt mart layer and the BI tool. It reads from materialised dbt models and emits SQL to BI consumers.
"Does MetricFlow materialise its own tables?" — no. The metric compiles to a query plan on every request; only the BI tool's cache or a saved-query cache materialises results.
"Can I run two metric layers — Cube and MetricFlow — side by side?" — yes, but you re-introduce drift. The whole point of the layer is to be the single source.
"What is the difference between a measure and a metric?" — a measure is a raw aggregation (e.g. SUM(plan_price_usd)); a metric is a business definition built on top of one or more measures (e.g. "MRR = sum of plan_price_usd filtered to active, non-trial subscriptions").

Detailed explanation — declare the five layers for a "weekly active accounts" KPI

Detailed explanation. A platform team needs to stand up a new metric — "weekly active accounts" — from scratch. The exercise demonstrates how each MetricFlow layer contributes one piece of the definition, with no layer doing more than its share.

Question. Build the five-layer stack for weekly_active_accounts, starting from a raw events table and ending at a metric request from Hex. Show the YAML for the semantic model and the metric, and explain which layer owns each concern.

Input. Source table:

event_id	account_id	event_ts	event_type
1	A1	2026-06-01 09:00	login
2	A1	2026-06-02 10:00	report_run
3	A2	2026-06-01 14:00	login
4	A3	2026-06-08 11:00	login

Code. The semantic model and metric:

# semantic_models/account_events.yml
semantic_models:
  - name: account_events
    model: ref('fct_account_events')
    entities:
      - name: account_id
        type: primary
    dimensions:
      - name: event_ts
        type: time
        type_params:
          time_granularity: day
      - name: event_type
        type: categorical
    measures:
      - name: distinct_active_accounts
        agg: count_distinct
        expr: account_id
        agg_time_dimension: event_ts

# metrics/weekly_active_accounts.yml
metrics:
  - name: weekly_active_accounts
    label: Weekly Active Accounts
    type: simple
    type_params:
      measure: distinct_active_accounts
    filter: |
      {{ Dimension('account_events__event_type') }}
        IN ('login', 'report_run', 'export')

Step-by-step explanation.

Warehouse layer. events table; raw, append-only. Schema may have a payload JSON blob with arbitrary keys. Not the metric's concern.
dbt model layer. fct_account_events is the clean, deduplicated, conformed event fact. Already tested for not_null(event_ts) and unique(event_id). The semantic model trusts these tests.
Semantic model layer. Declares account_id as the primary entity, event_ts as the time dimension, event_type as a categorical dimension, and distinct_active_accounts as a measure (count_distinct(account_id)). No filter yet — the measure is raw.
Metric layer. Composes the measure into the business definition. The filter restricts to "meaningful" events (login / report_run / export — not, say, password reset). The metric is weekly_active_accounts, time-bucketed weekly via the agg_time_dimension.
MetricFlow server. A consumer asks for weekly_active_accounts with metric_time__week. MetricFlow generates SELECT DATE_TRUNC('week', event_ts), COUNT(DISTINCT account_id) FROM fct_account_events WHERE event_type IN (...) GROUP BY 1. The right SQL for the warehouse dialect is emitted at query time.

Output.

Layer	Owns
Warehouse	raw events, schema evolution
dbt model	cleaning, dedup, conformed columns
Semantic model	entities, dimensions, raw measures
Metric	business filter, composition, label
MetricFlow server	dialect SQL, join resolution, caching

Rule of thumb. When debugging a wrong metric number, walk down the layer stack and ask "which layer owns this concern?" If the bug is in the event_type filter, fix the metric. If the bug is in the agg_time_dimension, fix the semantic model. If the bug is in event deduplication, fix the dbt model. Layered ownership turns "the dashboard is wrong" into a targeted diff.

Detailed explanation — how joins are resolved by entity keys, not surface SQL

Detailed explanation. A classic LookML-era trap is the user authoring a measure that already contains a JOIN — "here, let me just join subscriptions to users to get the country code." MetricFlow forbids surface joins in the metric definition; joins are resolved at query time using declared entity keys, so the graph is implicit and reusable.

Question. A team wants MRR by country_code. The subscriptions semantic model has no country_code dimension — that lives on users. Show how MetricFlow resolves the join without anyone writing JOIN SQL, and what the entity declaration looks like.

Input. Two semantic models with shared entity:

semantic_models:
  - name: subscriptions
    model: ref('fct_subscriptions')
    entities:
      - name: subscription_id
        type: primary
      - name: customer_id
        type: foreign     # ← the join key
    measures:
      - name: mrr_amount
        agg: sum
        expr: plan_price_usd

  - name: users
    model: ref('dim_users')
    entities:
      - name: customer_id
        type: primary     # ← matches the foreign entity above
    dimensions:
      - name: country_code
        type: categorical

Code. The query a consumer issues:

mf query --metrics mrr --group-by users__country_code --start-time 2026-01-01

Step-by-step explanation.

The consumer asks for mrr grouped by users__country_code. The metric lives on subscriptions; the dimension lives on users.
MetricFlow inspects the entity declarations. subscriptions.customer_id is a foreign entity; users.customer_id is a primary entity. They share a name → join is declared.
MetricFlow generates the SQL: SELECT u.country_code, SUM(s.plan_price_usd) FROM fct_subscriptions s JOIN dim_users u ON s.customer_id = u.customer_id GROUP BY 1. The join is implicit — no one wrote it.
If the team later adds a regions semantic model with country_code as a foreign entity pointing to a region primary, MetricFlow will follow the chain — subscriptions → users → regions — without any per-metric change.
The metric definition stays a one-line type: simple with a measure. All the join logic lives in the entity declarations on the semantic models, where it can be tested independently.

Output. The generated SQL (Snowflake dialect):

SELECT
    u.country_code,
    SUM(s.plan_price_usd) AS mrr
FROM analytics.fct_subscriptions s
JOIN analytics.dim_users u
  ON s.customer_id = u.customer_id
WHERE s.started_at >= '2026-01-01'
GROUP BY 1;

Rule of thumb. Never write a JOIN inside a metric YAML. If you find yourself wanting to, the missing piece is an entity declaration on the source semantic model. The whole point of MetricFlow is that the join graph is implicit and reusable — make the entity work, not the SQL.

Detailed explanation — the MetricFlow server as the front door

Detailed explanation. Once the metrics are declared, every consumer hits the MetricFlow server (dbt Cloud Semantic Layer for dbt Cloud customers, dbt-core mf CLI for self-hosted). The server exposes a uniform query interface, abstracts the warehouse dialect, and provides three-tier caching.

Question. Compare the consumer's request shape across four tools — Tableau, Hex, Mode, and a Python notebook. Show that they all converge on the same Semantic Layer API call, and explain the caching layers in the response path.

Input. Each tool issues a different idiomatic query:

# Tableau extract refresh
SQL connector → "SELECT mrr, plan_tier, metric_time__month FROM {{semantic_layer}}"

# Hex SQL cell
{{ semantic_layer.query(metrics=['mrr'],
                        group_by=['plan_tier','metric_time__month']) }}

# Mode report
SQL → "{{ semantic_layer.query(metrics=['mrr'],
                                group_by=['plan_tier','metric_time__month']) }}"

# Python notebook
client.query(metrics=['mrr'],
             group_by=['plan_tier','metric_time__month']).to_pandas()

Code. The MetricFlow server response path:

Consumer request
    ↓
1. Saved-query cache lookup  (TTL ~hours; hit ⇒ return)
    ↓ (miss)
2. Warehouse result cache    (Snowflake / BigQuery; hit ⇒ return)
    ↓ (miss)
3. Compile to dialect SQL    (planner)
    ↓
4. Execute on warehouse      (cold; pay full compute)
    ↓
5. Return + populate caches

Step-by-step explanation.

All four consumers converge on the same logical request: metrics=['mrr'], group_by=['plan_tier','metric_time__month']. The wire format differs (REST, JDBC, SQL connector) but the contract is identical.
The server first checks the saved-query cache — if a recently-computed extract matches the request shape and freshness, return it immediately. This is the "cheap hit" for repeated dashboard loads.
If no saved-query cache hit, the server checks the warehouse result cache (Snowflake's result cache, BigQuery's BigQuery Result Cache). If the underlying SQL is identical to a recent execution, the warehouse returns its own cached result — no compute.
If both caches miss, the planner compiles the metric request to dialect SQL, executes it, returns the result, and populates the caches for the next consumer.
The same request from Python and from Tableau hits the same cache. The platform team gets uniform observability and the consumer team gets warm-cache performance even on the first load of a new dashboard.

Output.

Cache tier	TTL	Population trigger
Saved-query cache	minutes–hours	scheduled refresh
Warehouse result cache	24h (Snowflake)	every query
Cold compile	N/A	cache misses

Rule of thumb. The MetricFlow server is a contract, not a single binary — what matters is that every consumer asks for metrics by name through the same API, and that the platform team owns the cache TTLs. Don't let one consumer (usually a Python notebook) bypass the server and read from the warehouse directly — that re-introduces drift.

Interview question on the MetricFlow architecture trade-off

A senior interviewer often probes: "Your team uses Looker for BI and a separate Python notebook stack for ML feature engineering. Both compute MRR independently. Walk me through the MetricFlow architecture you would propose to consolidate them, and which trade-offs you would accept."

Solution Using a single MetricFlow layer with two consumers

# Python notebook side — the same MRR every BI tool sees
from dbtsl import SemanticLayerClient

client = SemanticLayerClient(
    environment_id="dbt_cloud_env_id",
    auth_token=os.environ["DBT_CLOUD_TOKEN"],
    host="semantic-layer.cloud.getdbt.com",
)

mrr_by_month = client.query(
    metrics=["mrr"],
    group_by=["metric_time__month", "subscriptions__plan_tier"],
    where=["{{ Dimension('subscriptions__country_code') }} = 'US'"],
    order_by=["metric_time__month"],
).to_pandas()

# Looker (or any BI tool) — same metric, same source
connections:
  - name: dbt_semantic_layer
    type: dbt_semantic_layer
    environment_id: dbt_cloud_env_id

explore: mrr
    measures: [mrr]
    dimensions: [metric_time__month, plan_tier, country_code]

Step-by-step trace.

Consumer	Request shape	Goes through	Hits cache?
Looker dashboard load	`mrr by month, plan_tier`	Semantic Layer API	warehouse cache (warm)
Python notebook	`mrr by month, plan_tier, US`	Semantic Layer API	new shape — cold
Looker drill-down	`mrr by month, plan_tier, US`	Semantic Layer API	saved-query cache from the Python run

The trace shows the multi-consumer benefit: the Python notebook primed the cache for a country-filtered MRR request; minutes later, the Looker drill-down hits the same cache and returns instantly. Without MetricFlow, Python and Looker would have hit the warehouse independently, paid full compute twice, and risked returning different numbers.

Output:

Metric	Looker	Python notebook	Match?
MRR 2026-06, plan_tier=pro, US	`$842,310`	`$842,310`	yes
MRR 2026-06, plan_tier=team, US	`$311,520`	`$311,520`	yes
MRR 2026-06, plan_tier=enterprise, US	`$1,247,800`	`$1,247,800`	yes

Why this works — concept by concept:

One metric registry, two consumers — Looker and the Python notebook both resolve mrr to the same YAML file. The metric is computed once per cache miss, never duplicated.
Semantic Layer API as the contract — both consumers ride the same protocol (SemanticLayerClient for Python, dbt Semantic Layer connector for Looker). The platform team owns the API; the consumers consume.
Cache sharing across consumers — a request from Python warms the cache for a later Looker drill-down with the same shape. Cross-tool cache reuse is impossible if each tool reads from the warehouse independently.
Filter pushdown on the consumer — the Python notebook adds a where clause for country_code = 'US'. MetricFlow pushes this into the generated SQL, so the warehouse filters at scan time, not in the notebook.
Trade-off — extra hop, extra TTL — the only real cost is the extra hop through the MetricFlow server and a few-minutes saved-query TTL. For most analytics workloads, that latency is dominated by the warehouse query itself; net cost is sub-second.
Cost — one MetricFlow deployment (dbt Cloud or self-hosted mf CLI); per-query compute is paid by the underlying warehouse, gated by the cache hit rate.

SQL
Topic — joins
JOIN problems for semantic models (SQL)

Practice →

3. Anatomy of a metric definition

A metric is a contract — entity + measure + dimension + filter, composed by type

The mental model in one line: every dbt metric decomposes into four parts (entity, measure, dimension, filter), assembled by one of four types (simple, ratio, cumulative, derived) — once you can name the parts and the type, you can author any metric in YAML. Mastering this anatomy is the difference between "I can copy a MetricFlow example" and "I can ship a new KPI in 20 minutes."

The four atoms.

Entity. The grain of the metric — what each row represents. subscription_id for MRR, user_id for DAU, order_id for GMV. Declared on the semantic model with type: primary (or foreign if it joins to another model's primary).
Measure. The raw aggregation — sum(plan_price_usd), count_distinct(user_id). Carries its own agg, expr, and agg_time_dimension. The measure is never a metric on its own — it is the raw material.
Dimension. A grouping axis — categorical (plan_tier, country_code) or time (created_at, event_ts). Time dimensions carry a time_granularity (day, week, month).
Filter. A boolean expression that restricts which rows contribute. Lives on the metric, not the measure — same measure can power many metrics with different filters.

The four metric types.

Simple. One measure, optional filter. The 80% case. Example: mrr = sum(plan_price_usd) filtered to active subs.
Ratio. Numerator measure / denominator measure, with automatic NULL-safe division. Example: conversion_rate = signups / visits. MetricFlow handles "zero visits" by emitting NULL — no per-metric NULLIF plumbing required.
Cumulative. Running total over a window. Example: cumulative_new_users over metric_time__month. Configurable window (e.g. "trailing 30 days") and grain_to_date (e.g. "month-to-date").
Derived. An expression over other metrics. Example: gross_margin = revenue - cogs, or mom_growth = (mrr - mrr_prev_month) / mrr_prev_month using offset_window.

Time dimensions deserve their own paragraph.

agg_time_dimension on the measure. This is the time column the aggregate naturally bins by. started_at for MRR; event_ts for DAU. Declaring it makes metric_time a uniform alias across all metrics.
time_granularity. Declared on the dimension itself (day, week, month, quarter). MetricFlow upcasts: if the dimension is declared as day and the consumer asks for month, it auto-truncates.
metric_time virtual dimension. Every metric exposes a metric_time time dimension at request time, mapped from the underlying agg_time_dimension. Consumers can ask for metric_time__month without knowing the source column name — that's the abstraction layer.

Ratio metrics — the NULL-safety bonus.

numerator + denominator. Each points to a measure. Same semantic model or joined via entities.
Automatic null-handling. When the denominator is 0 or NULL, MetricFlow emits NULL — no division-by-zero error. The "junior engineer fix" of wrapping the denominator in NULLIF is unnecessary.
Filter on either side. A common pattern is signups / visits where signups has an additional filter (signup_type = 'qualified') — declare the filter on the numerator measure or as a separate per-side filter.

Cumulative metrics — the window choice.

window. 30 days, 90 days, 1 year. The metric returns the running sum over the trailing window for every requested time point. Pattern: trailing-30-day revenue, trailing-90-day active users.
grain_to_date. month, quarter, year. Returns the running sum since the start of the current bucket. Pattern: month-to-date GMV.
cumulative_type_params. Allows specifying both a window and a fill behaviour, including how to handle the first bucket (no prior data) and missing dates (zero or NULL).

Derived metrics — the composition layer.

expr. A Python-style expression over other metric names. gross_margin = revenue - cogs, arr = mrr * 12, cac = marketing_spend / new_customers.
offset_window. Reference a metric at a prior time bucket. mrr - mrr_offset_1month, mrr_yoy_growth = (mrr - mrr_offset_12month) / mrr_offset_12month.
The composition rule. A derived metric must only reference other named metrics, not raw measures or columns. This keeps the dependency graph clean and makes the metric reusable.

Detailed explanation — author a `revenue_per_active_user` ratio metric

Detailed explanation. The ratio metric is where the four atoms come together with the most leverage. revenue_per_active_user is the canonical board metric — and authoring it correctly is a single-screen exercise in MetricFlow.

Question. Author revenue_per_active_user as a ratio metric. Show the underlying measures (revenue and active_users), the metric YAML, and explain how MetricFlow handles the "zero active users on a quiet day" edge case.

Input. The two underlying measures already exist:

# in semantic_models/orders.yml
measures:
  - name: gross_revenue
    agg: sum
    expr: order_amount_usd
    agg_time_dimension: order_ts

# in semantic_models/account_events.yml
measures:
  - name: distinct_active_accounts
    agg: count_distinct
    expr: account_id
    agg_time_dimension: event_ts

Code. The ratio metric:

# metrics/revenue_per_active_user.yml
metrics:
  - name: revenue_per_active_user
    label: Revenue per Active User
    type: ratio
    type_params:
      numerator:
        name: gross_revenue
      denominator:
        name: distinct_active_accounts
    filter: |
      {{ Dimension('account_events__event_type') }}
        IN ('login', 'report_run')

Step-by-step explanation.

Identify the entity. Both measures roll up to the account / user grain. The shared entity is customer_id (or account_id) declared on both semantic models. MetricFlow uses the entity declarations to resolve any required join.
Identify the numerator. gross_revenue is the sum of order amounts. It already has an agg_time_dimension (order_ts) — that becomes the metric's time anchor.
Identify the denominator. distinct_active_accounts is the count of distinct active accounts in the same time window. The agg_time_dimension (event_ts) aligns to the same metric_time virtual column.
Compose the ratio. The type: ratio block points to both measures by name. MetricFlow emits numerator / NULLIF(denominator, 0) automatically — the NULL safety is built in.
Apply the filter. The filter restricts the denominator's active-user definition to "meaningful" events. Without the filter, password-reset bots would inflate the denominator and crash the ratio.
The "zero active users" edge case. On a day with zero qualifying events, distinct_active_accounts is 0. MetricFlow's auto-NULLIF returns NULL — the consumer sees an empty cell, not a divide-by-zero error.

Output. The compiled SQL (Snowflake):

SELECT
    DATE_TRUNC('day', metric_time) AS metric_time__day,
    SUM(gross_revenue) / NULLIF(COUNT(DISTINCT distinct_active_accounts), 0)
        AS revenue_per_active_user
FROM mf_subq
GROUP BY 1;

Rule of thumb. Every ratio metric should be declared as type: ratio, never hand-rolled as a derived metric of two simple metrics. The ratio type gives you NULL-safe division, automatic join resolution by entity, and cleaner SQL — three wins for one keyword.

Detailed explanation — cumulative new users with a rolling 90-day window

Detailed explanation. Cumulative metrics live or die on the window declaration. A 90-day rolling new-users count is the canonical "trust thermometer" for product growth — and the metric must auto-handle the first 89 days when there is no prior data.

Question. Author cumulative_new_users_90d as a cumulative metric over month grain with a trailing-90-day window. Show the underlying measure, the cumulative metric YAML, and the expected output shape for the first three months of data.

Input. The underlying measure:

# in semantic_models/users.yml
measures:
  - name: new_users_count
    agg: count
    expr: user_id
    agg_time_dimension: signup_ts

Code. The cumulative metric:

# metrics/cumulative_new_users_90d.yml
metrics:
  - name: cumulative_new_users_90d
    label: New Users (trailing 90 days)
    type: cumulative
    type_params:
      measure: new_users_count
      window: 90 days

Step-by-step explanation.

Pick the base measure. new_users_count counts signups grouped by signup_ts. Daily by default; MetricFlow upcasts to month if the consumer requests metric_time__month.
Choose cumulative over derived over simple. A cumulative metric maintains a running window across consecutive time buckets — the engine handles the window math, not your SQL.
Set the window. window: 90 days means each output row sums new users in the trailing 90-day window ending at that bucket's anchor date.
Output for early data. For 2026-01 (the first month of data), the window starts on 2025-10-03 — MetricFlow simply returns the count for whichever days are present. No NULL, no error.
The alternative: grain_to_date. If the platform team wants "year-to-date new users" instead of "trailing 90 days," swap window: 90 days for grain_to_date: year. The metric becomes a YTD running total that resets each January.

Output.

metric_time__month	cumulative_new_users_90d
2026-01	1,240
2026-02	2,610
2026-03	3,940
2026-04	4,710
2026-05	5,330

Rule of thumb. Cumulative metrics with window are the canonical "rolling trust thermometer" pattern. Cumulative metrics with grain_to_date are the canonical "reset-each-period" pattern. Pick the one that matches the business question — never hand-roll a window with OVER (...) SQL on top of MetricFlow.

Detailed explanation — a derived `mom_growth` metric with offset_window

Detailed explanation. Derived metrics let you express period-over-period comparisons declaratively. The MoM (month-over-month) growth metric is one line of YAML on top of an existing mrr metric.

Question. Author mrr_mom_growth as a derived metric using mrr and an offset_window of 1 month. Show the YAML and the expected output for two months of data.

Input. The mrr metric already exists from section 2.

Code. The derived metric:

# metrics/mrr_mom_growth.yml
metrics:
  - name: mrr_mom_growth
    label: MRR Month-over-Month Growth
    type: derived
    type_params:
      expr: "(mrr - mrr_prior_month) / mrr_prior_month"
      metrics:
        - name: mrr
        - name: mrr
          alias: mrr_prior_month
          offset_window: 1 month

Step-by-step explanation.

Reference the same metric twice. The derived metric pulls in mrr two ways: once as mrr (current bucket) and once with a 1-month offset, aliased as mrr_prior_month.
Write the expression in plain math. (mrr - mrr_prior_month) / mrr_prior_month is the standard growth formula. MetricFlow compiles this into the right SQL with the offset_window logic.
Handle the first bucket. For January 2026 (the first month of data), mrr_prior_month is NULL → the expression is NULL → the consumer sees a blank for the first month. No special-casing required.
Handle zero prior MRR. MetricFlow does not auto-NULLIF derived expressions — if the prior month MRR is 0, the expression divides by 0. Add a NULLIF(mrr_prior_month, 0) to the expr if 0 is a possible prior value.
Consumer experience. A Tableau user requests mrr_mom_growth by metric_time__month and gets the percentage directly — no per-dashboard window function, no offset SQL.

Output.

metric_time__month	mrr	mrr_prior_month	mrr_mom_growth
2026-01	$1.20M	NULL	NULL
2026-02	$1.31M	$1.20M	0.0917 (9.17%)
2026-03	$1.42M	$1.31M	0.0840 (8.40%)

Rule of thumb. Always reach for type: derived + offset_window for period-over-period comparisons. Hand-rolling a window function (LAG(mrr) OVER (ORDER BY month)) inside a BI tool re-introduces the drift the dbt metrics layer was supposed to eliminate.

Interview question on metric decomposition

A senior interviewer often asks: "Decompose 'conversion rate from free trial to paid' into the MetricFlow primitives. Show the entity, measure, dimension, filter, and metric type — and explain why a ratio metric is better than a derived metric here."

Solution Using a ratio metric with explicit numerator and denominator filters

# semantic_models/subscriptions.yml — add the two measures
semantic_models:
  - name: subscriptions
    model: ref('fct_subscriptions')
    entities:
      - name: subscription_id
        type: primary
      - name: customer_id
        type: foreign
    dimensions:
      - name: started_at
        type: time
        type_params:
          time_granularity: day
      - name: plan_status
        type: categorical
      - name: is_trial
        type: categorical
    measures:
      - name: trial_starts
        agg: count
        expr: subscription_id
        agg_time_dimension: started_at
      - name: paid_conversions
        agg: count
        expr: subscription_id
        agg_time_dimension: started_at

# metrics/trial_to_paid_conversion.yml
metrics:
  - name: trial_to_paid_conversion
    label: Trial → Paid Conversion Rate
    type: ratio
    type_params:
      numerator:
        name: paid_conversions
        filter: |
          {{ Dimension('subscriptions__plan_status') }} = 'active'
          AND {{ Dimension('subscriptions__is_trial') }} = false
      denominator:
        name: trial_starts
        filter: |
          {{ Dimension('subscriptions__is_trial') }} = true

Step-by-step trace.

Atom	Value
Entity	`subscription_id` (primary on `subscriptions`)
Numerator measure	`paid_conversions` = `COUNT(subscription_id)`
Denominator measure	`trial_starts` = `COUNT(subscription_id)`
Time dimension	`started_at` (day grain)
Numerator filter	`plan_status = 'active' AND is_trial = false`
Denominator filter	`is_trial = true`
Type	`ratio`

A consumer asks for trial_to_paid_conversion by metric_time__month for the last 12 months. MetricFlow generates one SQL statement that scans fct_subscriptions once, computes both filtered counts per month bucket, and divides numerator by NULLIF(denominator, 0).

Output:

metric_time__month	numerator	denominator	trial_to_paid_conversion
2026-04	412	1,800	0.2289 (22.9%)
2026-05	488	2,050	0.2380 (23.8%)
2026-06	511	1,920	0.2661 (26.6%)

Why this works — concept by concept:

Ratio over derived for ratios — type: ratio gives you automatic NULL-safe division. A type: derived equivalent (paid_conversions / trial_starts) would not auto-handle the zero-denominator case, requiring a NULLIF in the expression.
Per-side filters — the numerator and denominator measures filter different row populations (trial-start versus paid-conversion). Declaring the filters on each side of the ratio keeps the metric definition explicit and self-documenting.
One scan, two filtered counts — MetricFlow compiles both filtered counts into a single SELECT with conditional aggregation (COUNT(CASE WHEN ... END)). One warehouse scan, two output columns, one division. No subquery, no CTE.
Time alignment via shared agg_time_dimension — both measures bin by started_at. The metric_time virtual column maps to started_at automatically, so the consumer never has to name the underlying column.
Entity declaration enables future joins — declaring customer_id as a foreign entity lets a future consumer ask for "trial-to-paid conversion by users__country_code" without any change to the metric YAML.
Cost — one warehouse scan over fct_subscriptions; conditional aggregation is a constant-factor overhead. With a saved-query cache, the second consumer's read is sub-second.

SQL
Topic — conditional aggregation
Conditional aggregation problems (SQL)

Practice →

4. From metric definition to BI / Python query

The Semantic Layer API is the contract — every consumer asks for metrics by name

The mental model in one line: once a metric is declared in YAML, every consumer (Tableau, Hex, Mode, Lightdash, Python notebook) asks for it by name through the Semantic Layer API — the metric definition is the contract, the API is the wire format, and the cache layers fan out from there. The whole point is that the platform team owns the metric registry and the consumer team consumes by name.

The four consumer paths.

Tableau. Connects via the dbt Semantic Layer connector (or via a JDBC driver). Reads metrics as if they were views; can include them in workbooks, extracts, and published data sources. Refresh cadence controlled by Tableau.
Hex / Mode / Lightdash. Native dbt Semantic Layer integration. Users write {{ semantic_layer.query(metrics=['mrr'], group_by=[...]) }} in a SQL cell; the platform handles the rest.
Python. The dbtsl (or dbt-metricflow) Python client. client.query(...).to_pandas() returns a DataFrame. Ideal for ad-hoc analysis, ML feature engineering, and notebooks.
CLI. mf query --metrics mrr --group-by metric_time__month for local development. Same compiler, same SQL output, used for debugging and authoring.

The three caching tiers.

Saved-query cache. Platform team blesses specific metric+dimension+filter shapes as "saved queries." MetricFlow caches their results for a configurable TTL. First hit warms the cache; subsequent hits skip the warehouse entirely.
BI extract cache. Tableau extracts, Hex datasets, Mode result caches — each tool maintains its own snapshot. Useful for offline-friendly dashboards; risky if the underlying metric changes mid-quarter (extracts go stale).
Warehouse result cache. Snowflake's 24h result cache, BigQuery's query cache — kicks in whenever the same SQL runs twice. MetricFlow's compiled SQL is deterministic per metric+dimension shape, so this cache hits routinely.

Governance — who can edit metrics.

Platform team owns the metrics/ and semantic_models/ folders. PRs required, with at least one reviewer from the analytics-engineering team.
Domain teams contribute via PR. Sales-ops proposes a new pipeline_velocity metric; platform reviews, tests, merges. The PR template includes "business owner," "dollar impact," and "downstream consumers."
No metric edits in BI tools. Tableau calculated fields, Looker LookML field overrides, Hex hard-coded SQL — all banned by policy. The lock-down is what turns the layer from "nice-to-have" into "single source of truth."

Testing metrics — dbt tests + assertion suites.

dbt source tests. unique, not_null, relationships on the underlying mart tables. Catches upstream breaks before the metric runs.
dbt semantic-layer tests. Per-metric assertions: "MRR for 2026-04-01 must be $2.41M ± 0.1%." A regression test for KPIs, run nightly.
Snapshot tests. Capture the metric output for a stable date and diff against the next run. Any unintended drift fails CI.
Test the metric, not the table. The shift in mental model is from "test the upstream model" (still important) to "test the metric output" (the user-facing contract).

Detailed explanation — query the same metric from Python and Tableau

Detailed explanation. The proof that the layer works is when Python and Tableau return the exact same number for the same metric+filter request. Set this up once, test it once, and the platform team has tangible evidence to share with the CFO.

Question. Show the Python and Tableau queries for mrr by plan_tier for 2026-Q2, and the assertion that proves they match. Include the test that runs nightly to keep them aligned.

Input. The metric mrr exists (from section 2).

Code. Python side:

from dbtsl import SemanticLayerClient

client = SemanticLayerClient(
    environment_id=os.environ["DBT_ENV_ID"],
    auth_token=os.environ["DBT_TOKEN"],
    host="semantic-layer.cloud.getdbt.com",
)

df = client.query(
    metrics=["mrr"],
    group_by=["subscriptions__plan_tier"],
    where=[
        "{{ TimeDimension('metric_time', 'month') }} "
        "BETWEEN '2026-04-01' AND '2026-06-30'"
    ],
).to_pandas()

print(df.groupby("plan_tier")["mrr"].sum())

Tableau side:

Data source: dbt Semantic Layer
Metric: mrr
Dimensions: subscriptions__plan_tier
Filter: metric_time between 2026-04-01 and 2026-06-30

Step-by-step explanation.

Both sides hit the same Semantic Layer endpoint with logically equivalent requests. The metric, dimension, and filter shape match.
MetricFlow compiles each request to the same SQL plan against the same dbt model. The warehouse executes it.
Snowflake's result cache returns the cached result for the second request — both Python and Tableau see the same $2.41M total for Q2.
The nightly test runs both queries (via the Python client, parameterised) and asserts equality. Any drift between consumers fails CI.
The contract for the org is one sentence: "every Q2 MRR number must equal the Semantic Layer's mrr aggregated over Apr–Jun 2026. If your number differs, your number is wrong."

Output.

plan_tier	Python (`mrr`)	Tableau (`mrr`)	match
starter	$612,400	$612,400	yes
pro	$1,180,200	$1,180,200	yes
enterprise	$617,400	$617,400	yes
total	$2,410,000	$2,410,000	yes

Rule of thumb. Stand up the Python-vs-Tableau parity test on day 1. The first time the test fails, the platform team learns where a consumer is bypassing the layer (usually an old hard-coded SQL in a Tableau extract). The parity test is cheap, runs nightly, and is the only evidence the CFO needs that the layer holds.

Detailed explanation — a saved query for the exec deck weekly snapshot

Detailed explanation. Saved queries are the platform team's lever for guaranteed-fresh, low-latency board-deck numbers. Configure them once, schedule them, expose them as first-class objects — every consumer sees the same snapshot.

Question. Author a saved query exec_weekly_snapshot that bundles mrr, weekly_active_accounts, and gross_margin by plan_tier for the trailing 13 weeks. Schedule it to refresh every Monday at 06:00 UTC.

Input. The three metrics already exist.

Code. The saved query YAML:

# semantic_models/saved_queries.yml
saved_queries:
  - name: exec_weekly_snapshot
    description: |
      Weekly snapshot for the exec deck — three KPIs,
      sliced by plan_tier, trailing 13 weeks.
    query_params:
      metrics:
        - mrr
        - weekly_active_accounts
        - gross_margin
      group_by:
        - subscriptions__plan_tier
        - metric_time__week
      where:
        - |
          {{ TimeDimension('metric_time', 'week') }}
          >= CURRENT_DATE - INTERVAL '13 weeks'
    exports:
      - name: exec_weekly_snapshot_extract
        config:
          export_as: table
          schema: marts_exports

Step-by-step explanation.

The saved query bundles three metrics into one pre-bound request shape. The platform team blesses this shape — it is the official "exec deck snapshot."
Group-by includes plan_tier and metric_time__week. The output is a tidy panel: one row per (plan_tier, week) with three metric columns.
The where clause restricts to the trailing 13 weeks. Updated automatically each Monday — no manual date editing required.
The exports block materialises the result into a warehouse table (marts_exports.exec_weekly_snapshot_extract) for ultra-low-latency BI reads. Tableau and Hex can read this table directly without going through the API.
The schedule is owned by the dbt Cloud job system — runs every Monday at 06:00 UTC, fails loudly if any underlying mart is stale.

Output. The exported table schema:

plan_tier	metric_time__week	mrr	weekly_active_accounts	gross_margin
starter	2026-03-23	$215,400	4,210	0.72
starter	2026-03-30	$218,100	4,290	0.73
...	...	...	...	...

Rule of thumb. Use saved queries for any recurring report — exec decks, finance closes, weekly product reviews. The benefits compound: deterministic shape, automatic caching, materialised exports for sub-second reads, and a clear ownership boundary (platform owns the saved query; consumers own the dashboard that points at it).

Detailed explanation — a nightly assertion test for MRR snapshot stability

Detailed explanation. Once a metric is the source of truth, the platform team commits to it not changing unintentionally. A nightly assertion test captures the metric for a stable historical date and fails CI if the number ever drifts — this is how the team detects upstream schema changes, retroactive data backfills, or accidental metric edits before the CFO does.

Question. Write a dbt unit test that asserts mrr for 2026-04-30 equals $2,260,400 ± 0.05%, and explain what kinds of bugs this catches.

Input. The metric mrr already exists.

Code. The dbt unit test:

# tests/unit/test_mrr_snapshot_2026_04_30.yml
unit_tests:
  - name: mrr_snapshot_2026_04_30_stability
    description: |
      MRR for 2026-04-30 must remain $2,260,400 ± 0.05%.
      Locks the closed-month value against upstream backfills.
    model: ref('mrr')
    given:
      - input: semantic_layer.query
        metrics: [mrr]
        where:
          - "{{ TimeDimension('metric_time', 'day') }} = '2026-04-30'"
    expect:
      tolerance:
        type: percent
        value: 0.05
      rows:
        - mrr: 2260400.00

Step-by-step explanation.

The test pins one historical metric value — 2026-04-30 MRR = $2,260,400. This is a closed month — finance has already booked it; the number must not change.
The test runs as part of the nightly dbt CI suite. If MRR for that date drifts beyond 0.05%, the test fails loudly.
Real bugs this catches: (a) someone backfills cancelled subscriptions into the source table with a wrong cancellation_date, retroactively shifting active-status flags; (b) someone edits the metric filter without realising it affects historical values; (c) the upstream dim_users.is_trial flag changes definition.
The fix when the test fails: read the diff, find the change, decide whether it is intentional (and update the snapshot value with a PR explaining why) or unintentional (and revert).
This pattern generalises to the top 20 KPIs. One snapshot per metric per closed month — cheap to maintain, expensive bug to miss.

Output. Sample CI failure message:

FAIL  mrr_snapshot_2026_04_30_stability
  Expected: mrr = 2260400.00 ± 0.05%
  Actual:   mrr = 2274100.00 (+0.61%)
  Diff:     +$13,700
  Likely cause: upstream backfill in fct_subscriptions on 2026-06-13
  Owner: @analytics-platform-team

Rule of thumb. Pin every closed-month KPI as a snapshot test. The platform team commits to the historical numbers; the CFO trusts the historical numbers; the CI guarantees the platform's commitment. One test per KPI per closed month is a tiny operational cost relative to the audit-trail value.

Interview question on multi-consumer KPI parity

A senior interviewer often probes: "How would you prove that the same MRR number appears on the exec dashboard, the finance Hex board, and the Python notebook? What is your nightly safety net, and what is your incident response when they disagree?"

Solution Using a parity assertion suite plus a single Semantic Layer endpoint

# tests/parity/test_mrr_parity_across_consumers.py
import os
from dbtsl import SemanticLayerClient

client = SemanticLayerClient(
    environment_id=os.environ["DBT_ENV_ID"],
    auth_token=os.environ["DBT_TOKEN"],
    host="semantic-layer.cloud.getdbt.com",
)

# Canonical answer — the only "true" MRR for 2026-04
canonical = client.query(
    metrics=["mrr"],
    where=[
        "{{ TimeDimension('metric_time', 'month') }} = '2026-04-01'"
    ],
).to_pandas()["mrr"].iloc[0]

# Compare every consumer's published number
consumers = {
    "tableau_exec_dashboard": fetch_tableau_metric("mrr", "2026-04"),
    "hex_finance_board":      fetch_hex_metric("mrr", "2026-04"),
    "salesforce_reverse_etl": fetch_salesforce_metric("mrr", "2026-04"),
    "python_notebook":        canonical,
}

for name, value in consumers.items():
    drift = abs(value - canonical) / canonical
    assert drift < 0.0005, (
        f"{name} drift {drift:.4%} exceeds 0.05% tolerance "
        f"(consumer={value:,.2f}, canonical={canonical:,.2f})"
    )

print("All consumers agree on MRR within tolerance.")

Step-by-step trace.

Consumer	Reported MRR	Canonical MRR	Drift	Pass?
python_notebook	$2,260,400	$2,260,400	0.000%	yes
tableau_exec_dashboard	$2,260,400	$2,260,400	0.000%	yes
hex_finance_board	$2,260,400	$2,260,400	0.000%	yes
salesforce_reverse_etl	$2,266,800	$2,260,400	0.283%	FAIL

The Salesforce reverse-ETL pipeline reports a number that is $6,400 higher than canonical. The assertion fails, and the incident response begins: the platform team checks the reverse-ETL job to confirm it queries mrr through the Semantic Layer API (it does not — it re-derives MRR from raw fct_subscriptions, and a recent backfill changed the answer). Fix: rewrite the reverse-ETL job to call the Semantic Layer.

Output:

Step	Action
1	Nightly parity job queries canonical MRR via Semantic Layer
2	Each consumer's published MRR is fetched and compared
3	Any consumer >0.05% drift fails the CI run
4	Slack alert routes to `#analytics-platform` with diff payload
5	Fix is always: route the drifting consumer through the Semantic Layer

Why this works — concept by concept:

One canonical source — the Python query through the Semantic Layer is the only "true" answer. Every consumer is compared against this single number.
Drift tolerance, not exact match — 0.05% catches real drift while tolerating rounding and locale differences (e.g. Tableau's display rounding). Tune the tolerance per KPI; tighter for finance, looser for product.
Nightly cadence — drift drift below tolerance for one night is benign; drift above tolerance for one night is an incident. The cadence matches the speed at which KPIs flow into decisions.
Slack-routed incidents — the platform team is the on-call rotation. Drift becomes a paging signal, not a "we noticed it next quarter" surprise.
Fix-the-consumer pattern — the fix is never "edit the metric to match the drifting consumer." The fix is "route the drifting consumer through the Semantic Layer." This single rule preserves the source-of-truth invariant indefinitely.
Cost — one nightly job, one Slack channel, one runbook. The cost of operating the parity test is dominated by the warehouse query for the canonical answer; everything else is per-consumer scraping.

SQL
Topic — cumulative snapshots
Cumulative snapshot problems (SQL)

Practice →

5. Migration playbook — from BI views to dbt semantic models

Inventory, cluster, build top 20, cut over consumers — never tool-by-tool

The mental model in one line: the migration moves metric-by-metric, not tool-by-tool — inventory every calculated field, cluster them into canonical KPIs, build the top 20 in MetricFlow, then cut over consumer dashboards one KPI at a time with a one-quarter dual-run safety net. Trying to migrate "all of Tableau" or "all of Looker" in one sprint is the most common failure mode — the right unit of work is the metric.

The four-step playbook.

Step 1 — Inventory. Scrape every BI tool for "calculated field" definitions. Tableau workbook XML, Looker LookML, Hex SQL cells, Mode reports. Build a master spreadsheet: tool, dashboard, metric name, SQL/formula, owner.
Step 2 — Cluster. Group calculated fields by what they mean, not what they are named. Seven variants of "active user" cluster into one canonical metric.
Step 3 — Build top 20. Pick the 20 highest-leverage KPIs. Author them in semantic_models/ and metrics/. Test parity against the existing BI definitions during dual-run.
Step 4 — Cut over consumers. For each KPI, migrate consumer dashboards one at a time. Keep the legacy BI calculated field active for one full quarter as a safety net. Lock down BI editing after the quarter closes.

The inventory in detail.

Tableau. tabcmd export workbook → .twb XML. Grep for <calculation class='tableau' formula='...'/>. Extract formula + alias.
Looker. LookML lives in a git repo. grep "measure:" and grep "dimension:" across all .lkml files. Extract sql: clauses.
Hex. Workspace API exports SQL cell content. Parse for SELECT ... AS metric_name patterns.
Mode. API exports report SQL. Same pattern.
Reverse-ETL. Census / Hightouch jobs carry SQL. Audit each one.

The clustering process.

Canonical name first. The cluster gets one name — active_users — written down before anyone discusses "but our team calls it MAU." The PipeCode-style naming convention is snake_case, singular noun for counts, present-tense verb for ratios.
Pick the canonical definition. Usually the finance or exec definition wins. Document the chosen filter, time grain, and dimension axes.
Note the orphans. Definitions that don't fit any cluster are either real new metrics or junk. Decide explicitly — never let them lurk in the spreadsheet.

Build top 20 — what fits.

Revenue family. mrr, arr, gross_revenue, net_revenue, gross_margin, cac.
User family. active_users (daily / weekly / monthly variants), new_users, retained_users, churn_rate.
Engagement family. sessions, events_per_user, retention_7d, retention_30d.
Funnel family. signup_to_activation, trial_to_paid, conversion_rate_by_step.

The cut-over rhythm.

Week 0. Metric YAML merges to main. Saved query exposes it.
Week 1. New consumer (Hex board, Python notebook) reads from the Semantic Layer. Old consumer (Tableau dashboard) still reads from raw views with the old calculated field. Both run in parallel.
Week 4. Nightly parity test running. Drift triaged. Stakeholder review confirms the new number is canonical.
Week 12. Tableau dashboard cuts over. The old calculated field stays in the workbook for one more quarter as a fallback.
Week 24. Lock down BI calculated-field editing. Remove the legacy field.

Governance after the migration.

PR-required metric edits. No exceptions. The metric-edit PR template asks for business owner, expected dollar impact, and downstream consumers.
Quarterly KPI council. Platform + finance + product + sales meet for 60 minutes per quarter to review the metric registry, propose additions, retire dead metrics.
The "no calculated field" policy. New BI dashboards must use Semantic Layer metrics by name. Calculated fields in BI tools require platform sign-off; in practice, this means "never."

Detailed explanation — running the inventory across four BI tools in one sprint

Detailed explanation. Inventory is the unglamorous first step that determines whether the migration succeeds. A team that skips it ends up rebuilding the metric layer twice — once for the "obvious" KPIs and again for the "we forgot about that dashboard" KPIs six months later.

Question. Show the script that scrapes Tableau, Looker, Hex, and Mode for calculated fields and produces one master CSV. Explain why the inventory is the input to the clustering step, not an afterthought.

Input. Four tool APIs, each with its own credential.

Code. The inventory script:

# scripts/scrape_calculated_fields.py
import csv, os
from inventories import tableau, looker, hex_ws, mode

rows = []
for source, scraper in [
    ("tableau", tableau.scrape_workbooks),
    ("looker",  looker.scrape_lookml),
    ("hex",     hex_ws.scrape_sql_cells),
    ("mode",    mode.scrape_reports),
]:
    for item in scraper():
        rows.append({
            "source":     source,
            "tool_id":    item["id"],
            "dashboard":  item["dashboard"],
            "owner":      item["owner"],
            "field_name": item["name"],
            "formula":    item["formula"],
            "last_used":  item.get("last_used", ""),
        })

with open("inventories/calculated_fields_master.csv", "w") as f:
    w = csv.DictWriter(f, fieldnames=rows[0].keys())
    w.writeheader()
    w.writerows(rows)

print(f"Inventoried {len(rows)} calculated fields across 4 tools.")

Step-by-step explanation.

Each tool gets its own scraper module — Tableau via REST API + workbook XML, Looker via LookML repo grep, Hex via workspace API, Mode via report export API.
Each row in the master CSV carries the source tool, the dashboard it lives on, the owner, the field name, the formula or SQL, and the last used date (when available). The last-used date helps prioritise — fields not touched in 90 days are migration candidates with low risk.
The script runs as a one-shot first, then weekly as a "what's new" diff. Continuous inventory catches new calculated fields as they are created — and triggers a conversation with the team that created one.
The master CSV becomes the input to the clustering step. A platform engineer sits with it for 2–4 hours, groups rows by semantic meaning, and produces the cluster spreadsheet.
Common output of a first inventory: 300–600 calculated fields across 4 tools, clustering down to 40–70 canonical metrics. The 8:1 ratio is the cost of having no semantic layer.

Output. Sample CSV rows:

source	dashboard	field_name	formula	last_used
tableau	Exec Q2	mrr_v1	`SUM(IF [status]='active' THEN [plan_price])`	2026-06-12
tableau	CFO Board	mrr	`SUM(IF [is_active] AND NOT [is_trial] THEN [price_usd])`	2026-06-12
looker	Finance	mrr	`SUM(CASE WHEN status='active' THEN amount END)`	2026-06-11
hex	Growth	mrr_growth	`SUM(price_usd) WHERE status='active' AND price_usd>0`	2026-06-10

Rule of thumb. Run the inventory in the first sprint of the migration. The scraper code is one engineer-week; the manual clustering is two engineer-days. Trying to "just start writing metrics" without an inventory means you'll re-author 40% of them six months later.

Detailed explanation — clustering 60 variants into 20 canonical metrics

Detailed explanation. Clustering is the migration's most-undervalued step. The platform engineer who runs it correctly saves the team 3–6 months of "but our team has its own definition" debates.

Question. Given a slice of the inventory CSV containing seven "active user" variants, cluster them into one canonical metric. Show the cluster sheet output and the rationale for the chosen definition.

Input. Seven variants:

1. tableau / Product Dashboard:   login event in last 30 days
2. tableau / Growth Dashboard:    any event in last 7 days
3. looker / Marketing:            any session in last 30 days
4. hex / Finance:                 paying subscription not cancelled
5. mode / RevOps:                 logged in OR exported in last 30 days
6. python notebook / ML:          any event in last 60 days
7. CFO sheet / Manual:            paying subscription not cancelled

Code. The cluster sheet (a Google Sheet maintained by the platform team):

Cluster:  "active_users"
Canonical owner:  Finance + Product joint
Canonical definition:
    SUM(DISTINCT user_id)
    where event_type IN ('login','report_run','export')
    AND event_ts >= CURRENT_DATE - INTERVAL '30 days'
    AND user_id has a non-cancelled subscription

Variants mapped:
   1 → active_users_30d_login_only       (deprecated)
   2 → active_users_7d                   (real new metric — separate KPI)
   3 → active_users_30d_session          (deprecated, same as canonical)
   4 → paying_active_users               (real new metric — separate KPI)
   5 → active_users_30d                  (CANONICAL)
   6 → active_users_60d                  (real new metric — separate KPI)
   7 → paying_active_users               (alias of variant 4)

Net result: 7 variants → 4 named metrics (1 canonical + 3 real new)

Step-by-step explanation.

The platform engineer reads each variant in the inventory CSV. For "active users," seven variants exist; some are genuinely different metrics (the 7-day window, the paying-only filter), others are accidental drift around the same concept.
The engineer convenes a 30-minute working session with finance, product, and growth. The goal is one canonical definition — the rest are either real new metrics or deprecated.
The session output is the cluster sheet: one canonical metric (active_users_30d), three real new metrics (active_users_7d, paying_active_users, active_users_60d), and three deprecations.
Each canonical metric becomes a metric YAML. Each real new metric becomes a metric YAML. Each deprecation gets a sunset date and an owner.
The 8:1 → 4:1 collapse is typical: the inventory looks chaotic but the actual business KPIs are far fewer. The clustering step discovers the real metric registry.

Output.

canonical	real new metrics	deprecated variants
`active_users_30d`	`active_users_7d`, `paying_active_users`, `active_users_60d`	login-only-30d, session-only-30d, manual paying-active sheet

Rule of thumb. Clustering is not "pick one and ignore the rest." It is "discover which variants are genuinely different metrics and which are drift." A senior platform engineer can cluster 60 variants in one day. Two days if there is a lot of debate; three days if finance and product disagree on the canonical filter.

Detailed explanation — cut over a Tableau dashboard with a one-quarter safety net

Detailed explanation. Cut-over is where the migration succeeds or fails in the eyes of the consumer. The safety-net pattern — dual-run for one quarter, parity test nightly, lock-down after — is the single most important governance choice.

Question. Walk through the cut-over for the "CFO MRR Trends" Tableau dashboard. Show the dual-run setup, the parity assertion, and the lock-down step.

Input. The dashboard currently reads from a Tableau calculated field; the canonical MRR metric exists in MetricFlow.

Code. The dual-run state:

Dashboard: CFO MRR Trends (Tableau)

PANEL A — Legacy (calculated field)
    Data source: snowflake.marts.fct_subscriptions
    Field: legacy_mrr = SUM(IF [is_active] THEN [plan_price_usd])

PANEL B — Canonical (Semantic Layer)
    Data source: dbt Semantic Layer
    Metric: mrr
    Group by: metric_time__month

PANEL C — Diff
    diff_pct = (legacy_mrr - mrr) / mrr
    Conditional formatting: red if abs(diff_pct) > 0.0005

Step-by-step explanation.

The dashboard ships in dual-run mode. Panel A shows the legacy number that the CFO has trusted for two years. Panel B shows the canonical number from the Semantic Layer. Panel C surfaces the diff.
For the first week, the CFO sees both numbers. If panel B agrees with panel A within 0.05%, trust is built. If they disagree, the platform team gets a Slack ping and chases down the cause (usually a forgotten filter on one side).
After one full quarter (typically three monthly closes), the CFO formally accepts panel B as the canonical answer. Panel A and panel C are removed from the dashboard.
Once accepted, the platform team removes the Tableau calculated field. Workbook permissions are tightened to prevent re-creating it. From this point forward, "CFO MRR Trends" reads only from the Semantic Layer.
The pattern repeats for every high-stakes dashboard. The platform team's commitment is: dual-run for one quarter, lock-down on quarter close.

Output.

Phase	Panel A (legacy)	Panel B (canonical)	Action
Week 1	$2.241M	$2.260M (+0.85%)	investigate
Week 4	$2.255M	$2.260M (+0.22%)	drift reducing
Week 8	$2.260M	$2.260M (0.00%)	accept canonical
Week 13	(removed)	$2.260M	locked down

Rule of thumb. Never cut over a high-stakes dashboard without a dual-run quarter. The CFO needs to watch the parity converge in real numbers; only then does trust transfer. The single most-common migration failure is "we cut over in one sprint" — the dashboard ships with a small drift, the CFO loses confidence in the new layer, and the project gets paused for six months.

Interview question on migration sequencing

A senior interviewer often asks: "Your CEO wants to move every dashboard to dbt metrics in one quarter. Walk me through how you'd push back — and what 90-day plan would actually be deliverable, with what risks accepted."

Solution Using a top-20 quarter and a dual-run safety net

WEEK 0     ── Inventory complete (4-tool scrape, master CSV)
WEEK 1–2   ── Clustering workshop (60 fields → 22 canonical metrics)
WEEK 3–5   ── Build top 20 (semantic_models + metrics YAML)
WEEK 6     ── Stand up parity test infrastructure
WEEK 7–10  ── Cut over top-3 high-trust dashboards in dual-run
WEEK 11    ── First monthly close in dual-run; parity passes
WEEK 12    ── CFO sign-off on top-3 dashboards
WEEK 13    ── Lock down BI editing on the top-3

Step-by-step trace.

Week	Deliverable	Risk accepted
0	inventory CSV	scrapers miss obscure tools
1–2	cluster sheet	some variants get bucketed wrong
3–5	20 metric YAMLs	per-metric edge cases TBD
6	parity test infra	infra debt deferred 1 quarter
7–10	3 dashboards in dual-run	other 80 dashboards untouched
11–12	1 monthly close passes parity	one close is not statistically conclusive
13	top-3 locked down	remaining 80 dashboards re-scope next quarter

After the quarter, the team has 20 canonical metrics live, 3 high-trust dashboards locked down, and a parity test infrastructure that scales. The CEO's "every dashboard in one quarter" goal becomes "every dashboard in three quarters with quarterly checkpoints" — pushed back with data.

Output:

End-of-quarter state	Value
Canonical metrics live	20
Dashboards locked down	3 (the highest-trust trio)
Dashboards in dual-run	12 (rolling, ~1 cut-over per week from here)
Parity test cadence	nightly
CFO trust transferred	yes (single sign-off on top-3)

Why this works — concept by concept:

Top-20 metrics, not top-20 dashboards — the unit of migration is the metric. Top-20 metrics typically power 60–80% of consumer dashboards, so the high-leverage 80% lands in the first quarter.
Three-dashboard cut-over, not three-tool cut-over — the platform team picks three specific high-trust dashboards and ships them end-to-end. The CFO sees parity converge on real dashboards, not on a "platform-team demo."
Dual-run for one full close — one monthly close in dual-run is the minimum unit of trust. Two closes is better; three is gold standard.
Push back with a plan — saying "no" to "everything in one quarter" without a counter-proposal is career-limiting. Saying "no, but here is the 90-day plan I can deliver" is leadership.
Lock-down is the safety lever — once a dashboard is locked down, it cannot drift. The first three lock-downs are the proof that the layer is the source of truth.
Cost — one quarter of focused platform-team time (typically 2 engineers × 13 weeks), most of it on inventory, clustering, and cut-over choreography rather than YAML authoring. The marginal cost per subsequent metric is the time to write the YAML and the parity test.

SQL
Topic — grouping
Grouping problems for KPI clustering (SQL)

Practice →

Cheat sheet — dbt metrics recipes

MAU metric (count distinct). Declare measure: distinct_active_users with agg: count_distinct and expr: user_id. Wrap in type: simple with a filter event_type IN ('login','...'). The MetricFlow virtual metric_time__month gives you MAU directly.
MRR metric (sum + filter). Measure mrr_amount with agg: sum and expr: plan_price_usd. Metric filter: is_active = true AND is_trial = false. Aggregates by started_at time grain.
Conversion ratio (NULL-safe division). type: ratio with numerator: paid_conversions and denominator: trial_starts. MetricFlow auto-wraps the denominator in NULLIF(...), so zero-trial days return NULL, not an error.
Cumulative new users (rolling 90 days). type: cumulative with measure: new_users_count and window: 90 days. Returns the trailing-90-day sum for every time bucket — no OVER (...) SQL required.
Period-over-period growth (derived + offset_window). type: derived with expr: (mrr - mrr_prior_month) / mrr_prior_month and an offset_window: 1 month on the second mrr reference. Auto-NULL for the first bucket.
Saved query for the exec deck. saved_queries: block listing metrics, group_by, where, and an exports: table. Refresh weekly via a dbt Cloud job; consumers point at the export table for sub-second reads.
Entity-resolved join across semantic models. Declare customer_id as foreign on the source model and primary on the target. MetricFlow resolves the join automatically when a consumer asks for the target's dimensions.
Nightly snapshot stability test. Pin one historical metric value (e.g. "MRR for 2026-04-30 must be $2,260,400 ± 0.05%"). Run as a dbt unit test; fails CI on unintended drift.
Parity test across consumers. Python script queries the canonical answer via the Semantic Layer API, scrapes each consumer's published number, asserts abs(consumer - canonical) / canonical < 0.0005. Routes failures to Slack.
No surface SQL in metric YAML. Joins are resolved by entity declarations; do not write JOIN inside a metric. If you want to, the missing piece is an entity on the source semantic model.
Inventory four tools, not one. Tableau + Looker + Hex + Mode at minimum. Add reverse-ETL (Census / Hightouch) if you have one — those jobs carry SQL that often re-derives metrics.
Cluster before authoring YAML. 60 calculated-field variants typically collapse to 20 canonical metrics. Cluster first; write YAML second.
Dual-run for one full monthly close. Never cut over a high-stakes dashboard in one sprint. Run legacy and canonical side-by-side until the parity test passes three weeks running.
Lock down BI editing post-cut-over. Tableau permissions, Looker LookML reviewer gates, Hex governance. Without lock-down, drift re-emerges within two quarters.

Frequently asked questions

Is MetricFlow the same as dbt metrics?

Not exactly — MetricFlow is the SQL compiler and query engine that powers dbt metrics. The dbt metrics layer is the declarative interface (semantic models, metrics, saved queries in YAML); MetricFlow is the runtime that takes a metric request, resolves entity-keyed joins across semantic models, compiles the SQL to your warehouse dialect, and executes it. In practice, the two names are used interchangeably — when someone says "we use dbt metrics" they mean "we declare metrics in YAML and let MetricFlow compile them at query time." The architecture matters: MetricFlow is what enables the same metric to compile to Snowflake SQL, BigQuery SQL, or Postgres SQL without any per-warehouse rewriting.

Do I still need a BI tool with the dbt Semantic Layer?

Yes — the dbt Semantic Layer is not a replacement for Tableau, Looker, or Hex; it is the layer below them. The Semantic Layer owns the metric definitions and exposes them through an API; the BI tool owns the visualisation, the dashboard layout, the user permissions, and the consumption experience. The win is that every BI tool now reads from the same metric registry, so the same KPI shows the same number regardless of which dashboard the user opens. If anything, the BI tool becomes more focused — its job is now visualisation and consumption rather than business-logic authoring.

How does MetricFlow compare to Cube and LookML?

All three are semantic layers, but they live at different points in the stack. LookML lives inside Looker — the metric definitions never leave the BI tool, which means non-Looker consumers (Python notebooks, Tableau, reverse-ETL) have to re-derive metrics from raw warehouse columns. Cube is a standalone semantic layer with its own server, supports many consumers, but lives outside the dbt project — so you maintain two versioned artefact stacks. MetricFlow lives inside the dbt project — semantic models and metrics share the dbt PR / CI / docs workflow. For organisations already standardised on dbt, MetricFlow's locality is the strongest argument; teams that need a metric layer without dbt often pick Cube; LookML is best treated as a Looker-internal optimisation, not an org-wide semantic layer.

Can I query metrics from Python notebooks?

Yes — the dbtsl (dbt Semantic Layer client) and dbt-metricflow Python libraries give notebooks first-class access. The pattern is client.query(metrics=['mrr'], group_by=['metric_time__month', 'plan_tier']).to_pandas() — the call returns a Pandas DataFrame ready for analysis. The win for data science and ML is enormous: feature engineering pipelines now read MRR, DAU, and conversion rate by the same names the CFO and product manager use, and the metric is computed by the same compiler. No more "the ML feature for 'active user' disagrees with the dashboard."

What happens to dashboards built on raw views?

They keep working — until you cut them over. The migration playbook explicitly keeps the legacy calculated fields and views active during the dual-run quarter, so existing dashboards do not break. Once the metric is locked down (typically after one monthly close in dual-run with passing parity tests), the platform team cuts the dashboard's data source from the raw view to the Semantic Layer. From that point forward, the dashboard reads canonical numbers, and the legacy calculated field is removed from the workbook. The pattern repeats per dashboard — never "all dashboards at once."

How do I test that a KPI value didn't change unintentionally?

Pin the metric output for closed historical periods as dbt snapshot tests. The pattern is one assertion per KPI per closed month: "MRR for 2026-04-30 must equal $2,260,400 ± 0.05%." Run the test as part of the nightly dbt CI suite — any drift beyond tolerance fails the build and pages the platform team via Slack. The bugs this catches are exactly the ones that destroy CFO trust: an upstream backfill that retroactively changes a closed-month number; an accidental metric edit; a definition drift introduced by a refactor. Pair the snapshot test with a multi-consumer parity test (canonical answer via Semantic Layer compared to each consumer's published number) and the platform team has both vertical (historical) and horizontal (cross-tool) safety nets.

Practice on PipeCode

Drill the aggregation practice library → for the SUM / COUNT / AVG patterns every dbt measure compiles down to.
Rehearse on cumulative snapshot problems → for the running-window math that powers cumulative metrics.
Sharpen conditional aggregation drills → for the filtered-COUNT and filtered-SUM patterns that show up in ratio numerators.
Layer the joins practice library → for the entity-resolved cross-model joins that MetricFlow generates implicitly.
Stack the grouping practice library → for the time-bucket and dimension grouping that drives metric requests.
Rehearse the window functions library → for the period-over-period and trailing-window math behind derived metrics.
For the broader surface, read top data engineering interview questions →.
Stack the prerequisites with the only 5 skills you need to become a data engineer →.
Sharpen the SQL axis with the SQL for data engineering interviews course →.
For the modelling foundation that semantic layers build on, work through data modelling for DE interviews →.
For metric thinking and product sense, work through product sense and metrics for DE interviews →.

Pipecode.ai is Leetcode for Data Engineering — every dbt metric recipe above ships with hands-on practice rooms where you write the conditional aggregation, the cumulative window, and the entity-keyed join against real graded inputs. PipeCode pairs every reading with 450+ DE-focused problems and a real-time scoring engine, so your MetricFlow YAML and your interview answer behave identically against the same warehouse semantics.

Practice aggregation now →
Cumulative snapshot drills →

Reverse ETL with Hightouch, Census & RudderStack: Operational Analytics in Practice

Gowtham Potureddi — Tue, 16 Jun 2026 12:40:14 +0000

When you want hands-on reps while reading, drill the ETL practice library →, layer in API integration drills →, and stack the warehouse muscles with dimensional modelling problems →.

On this page

Why reverse ETL exists — operational analytics as a discipline
The reverse ETL data model — models, audiences, syncs
Hightouch vs Census vs RudderStack — vendor comparison
Sync architecture — incremental detection, queues, rate limits
Governance, observability, and failure modes
Cheat sheet — reverse ETL recipes
Frequently asked questions
Practice on PipeCode

1. Why reverse ETL exists — operational analytics as a discipline

Forward ETL moves data INTO the warehouse so analysts can ask questions; reverse ETL moves data OUT so operational systems can act

The data activation gap in three bullets.

Dashboards inform people; syncs inform systems. A lead score in Looker is a number a manager looks at on Monday. A lead score in Salesforce is a field a routing rule reads at midnight to assign the lead to the right rep. The two consumers want the same number but through different surfaces.
The warehouse aggregates across silos; SaaS tools cannot. Stripe knows about payments. HubSpot knows about emails. The product database knows about feature usage. Only the warehouse joins them. Reverse ETL ships that join back into every silo.
Manual CSV exports do not scale. A "send a CSV to ops once a week" workflow has zero observability, no schema contract, and breaks the first time a column is renamed. Reverse ETL turns the export into a versioned, scheduled, monitored data product.

Common destinations in 2026.

CRMs. Salesforce, HubSpot, Microsoft Dynamics, Pipedrive.
Marketing automation. Marketo, Iterable, Customer.io, Braze, Klaviyo, Mailchimp.
Support + success. Intercom, Zendesk, Gainsight, Vitally, ChurnZero.
Ad platforms. Facebook / Meta custom audiences, Google Ads customer match, TikTok audiences, LinkedIn matched audiences.
Collaboration + ops. Slack channels, Microsoft Teams webhooks, Notion databases, Asana tasks.
Product analytics. Amplitude cohorts, Mixpanel cohorts, Heap audiences.

Why the warehouse won as source of truth.

Compute and storage are now cheap. Snowflake, BigQuery, Databricks, Redshift — every cloud warehouse runs the joins at a price that makes "send the join result downstream" feasible at scale.
dbt made transformation governable. Once models/marts/customers.sql is the single SQL definition of "a customer," every downstream system can subscribe to its rows instead of recomputing them.
Data teams finally have leverage on the operational stack. Reverse ETL gives the data team a contract with marketing, sales, and CS without writing custom Python in five different SaaS APIs.

When NOT to use reverse ETL.

Sub-second latency requirements. Reverse ETL is a batch + micro-batch architecture. Hightouch ships syncs as fast as ~5 minutes; Census as fast as ~1 minute; RudderStack with streaming-event reverse ETL can hit seconds. Below that, you want event streaming (RudderStack event stream, Segment, Kafka → consumer) — not warehouse syncs.
True event streaming. "Page view fires → personalisation engine reacts in 200ms" is not a reverse ETL problem; it is a Kafka / Kinesis / event-bus problem.
One-off backfills. A 50k-row one-time list does not need a sync pipeline; a CSV import inside the destination is faster.

Worked example — the lead score sync that justifies reverse ETL

Input — marts.lead_scores.

salesforce_contact_id	lead_score	last_engagement_at	churn_risk
003A1	87	2026-06-12	0.12
003A2	41	2026-05-30	0.55
003A3	92	2026-06-14	0.08
003A4	NULL	NULL	NULL

Code.

-- The dbt model that becomes the sync source.
-- File: models/marts/lead_scores.sql
SELECT
    c.salesforce_contact_id,
    ROUND(s.raw_score, 0)                       AS lead_score,
    s.last_engagement_at,
    s.churn_risk
FROM {{ ref('dim_contacts') }}            c
LEFT JOIN {{ ref('int_lead_scoring') }}   s
       ON s.contact_id = c.contact_id
WHERE c.salesforce_contact_id IS NOT NULL

# Hightouch sync definition (illustrative YAML).
# File: hightouch/syncs/salesforce_lead_score.yaml
model: marts.lead_scores
destination: salesforce_production
sync_mode: upsert
primary_key: salesforce_contact_id
schedule: "*/30 * * * *"   # every 30 minutes
mappings:
  - source: lead_score          -> Contact.lead_score__c
  - source: last_engagement_at  -> Contact.last_engagement_at__c
  - source: churn_risk          -> Contact.churn_risk__c

Step-by-step explanation.

The dbt model produces one row per Salesforce contact with a stable salesforce_contact_id primary key. The model is the contract — change the SQL, change every downstream consumer.
Hightouch reads the model on the cron schedule. On the first run it stores a snapshot; on every later run it diffs the current rows against the previous snapshot to find changes.
The sync_mode upsert tells the destination "insert if salesforce_contact_id does not exist, update otherwise." Salesforce External ID matching is configured in the Hightouch UI to map salesforce_contact_id to Salesforce's Id field.
The three field mappings turn warehouse columns into Salesforce custom fields. NULL lead_score for 003A4 becomes a blank update on the Salesforce field; the destination keeps any previous value if the sync setting is "do not overwrite with NULL."
The cron */30 runs every 30 minutes — far below Salesforce's daily API limit but fast enough for sales routing.

Output (Salesforce after sync).

Salesforce Contact Id	lead_score__c	last_engagement_at__c	churn_risk__c
003A1	87	2026-06-12	0.12
003A2	41	2026-05-30	0.55
003A3	92	2026-06-14	0.08
003A4	(unchanged)	(unchanged)	(unchanged)

Worked example — the dashboards-vs-syncs contrast

Question. Given a churn-risk metric, write the two access patterns side by side — Looker dashboard query vs reverse ETL sync — and explain why both exist.

Input.

Surface	Cadence	Consumer	Failure mode
Looker dashboard	on demand	account manager	empty card
Reverse ETL sync	every 6h	Intercom tag automation	stale tag

Code.

-- Looker explore (shared view).
-- explore: account_health
-- view: marts.account_health
view: account_health {
  sql_table_name: marts.account_health ;;
  measure: avg_churn_risk {
    type: average
    sql: ${TABLE}.churn_risk ;;
  }
}

# Hightouch sync — same underlying model, machine surface.
model: marts.account_health
destination: intercom
sync_mode: mirror
primary_key: account_id
schedule: "0 */6 * * *"
mappings:
  - source: churn_risk                 -> Company.churn_risk_attr
  - source: CASE WHEN churn_risk > 0.7
            THEN 'at_risk' ELSE 'ok' END  -> Company.health_tag

Step-by-step explanation.

The same marts.account_health model feeds both surfaces. There is exactly one definition of "churn risk" in the company.
The dashboard query runs when a human opens it. The SLA is "the query returns in less than 10 seconds and the number is no older than the last warehouse refresh."
The Hightouch sync runs every 6 hours regardless of human attention. The SLA is "the Intercom tag reflects yesterday's risk score by the end of every 6-hour window."
Failure modes differ: a dashboard failure is loud (empty card, error toast); a sync failure is quiet (a stale tag still looks like data). Observability for the sync must be explicit.

Output.

Surface	Behaviour when warehouse fails
Looker dashboard	error visible immediately to the user
Hightouch sync	last successful tag persists; alert fires only if observability is set up

Worked example — when reverse ETL is the wrong tool

Input.

Requirement	Latency target	Architecture
Sales Salesforce score	30 minutes	reverse ETL
Marketing Intercom tag	6 hours	reverse ETL
Ad audience refresh	24 hours	reverse ETL
Homepage personalisation	< 500 ms	event stream
Fraud block at checkout	< 200 ms	online ML feature store

Code.

Decision rubric (pseudo-code):

if latency_target >= 5_minutes:
    use reverse_etl (Hightouch / Census / RudderStack)

elif latency_target >= 30_seconds:
    use event_stream_reverse_etl (RudderStack event stream)

elif latency_target >= 100_ms:
    use online_feature_store + low_latency_api (Tecton, Feast, custom)

else:
    use in_request_compute (edge function, cached cache lookup)

Step-by-step explanation.

The latency floor for batch reverse ETL is a function of warehouse query time + diff computation + destination API throughput. On a shared tenant in Hightouch this typically lands at 5–15 minutes.
RudderStack's event-stream reverse ETL closes the loop in seconds for individual event triggers but still cannot serve a single-millisecond synchronous API call.
Online ML feature stores (Tecton, Feast) maintain a serving layer separate from the warehouse precisely for sub-100ms reads. Reverse ETL pre-materialises features into that layer on a slower cadence.
The rubric ranks tools by the actual latency budget the use case requires. Picking the wrong tier wastes either money (using a feature store for a daily ad audience) or signal (using reverse ETL for sub-second personalisation).

Output.

Use case	Tool	Why
Lead score in Salesforce	Hightouch upsert	30-min cadence, batch fine
Churn risk tag in Intercom	Census sync	6h cadence, batch fine
Homepage banner	Edge feature read	sub-500ms, batch insufficient
Fraud rule at checkout	Online feature store	sub-200ms, must be pre-materialised

Reverse ETL interview question on the lift-up from forward ETL

Solution Using the data activation contract

The data team takes on three new responsibilities the day reverse ETL ships:

1. Model stability is now an operational SLA.
   - Every sync model needs a stable primary key (renaming it
     breaks identity resolution downstream).
   - Column renames now break SaaS-tool fields that humans rely on.
   - Type changes can silently corrupt destination fields.
   - Solution: dbt contract tests + dbt exposures + protected branch
     for any model with downstream syncs.

2. Freshness is now a destination-level SLA.
   - Warehouse "fresh as of midnight" is no longer enough.
   - Each destination has its own freshness contract (Salesforce: 30m,
     Intercom: 6h, Facebook ads: 24h).
   - Solution: per-sync alerting, last_synced_at columns, freshness
     dashboards.

3. Governance now spans warehouse + SaaS tools.
   - PII synced to Marketo is now subject to Marketo's retention.
   - GDPR delete must propagate to every destination.
   - Solution: PII tags on every column, per-destination policy,
     destination-side deletes.

Step-by-step trace.

Responsibility	Before reverse ETL	After reverse ETL
Model PK stability	nice-to-have	hard contract
Column rename	dashboard fix	downstream sync break
Freshness	warehouse-wide	per-destination
PII	warehouse policy	propagated to N SaaS tools
Lineage	dbt + BI	dbt + BI + syncs

Output:

Practice	New requirement once reverse ETL is in the stack
dbt contracts	required on every sync model
dbt exposures	every sync surfaced in lineage
PII tagging	per-column tags propagated to destination policy
Alerting	per-sync row-error rate and freshness SLA
On-call	one person owns sync health

Why this works — concept by concept:

Models become contracts — the (primary_key, columns, types) tuple is now a stable API. Any change is a versioned migration with downstream blast-radius assessment.
Freshness becomes per-destination — the warehouse SLA is the upper bound; each sync has its own, often tighter, freshness contract because downstream SaaS automation acts on it.
PII becomes propagated — a column tagged "email PII" in the warehouse must inherit the same handling everywhere it lands. GDPR delete is the canonical stress test.
Lineage becomes end-to-end — dbt exposures are the standard way to surface "this model is consumed by this Hightouch sync" inside the dbt docs and the data catalog.
On-call gets a new pager — the day a sync fails silently is the day the data team learns operational analytics needs operational ownership. One person owns sync health, full stop.
Cost — the new responsibilities are mostly process; the dbt features (contracts, exposures, tags) ship out of the box. Marginal infrastructure cost is the reverse ETL vendor subscription itself.

ETL
Topic — etl
ETL pipeline problems (data engineering)

Practice →

2. The reverse ETL data model — models, audiences, syncs

Every reverse ETL platform organises around four nouns: model, audience, sync, destination — learn them once and every vendor feels the same

The four-noun glossary.

Model. A SQL query (or dbt model reference) that returns rows of a single entity — one_row_per_user, one_row_per_account, one_row_per_subscription. The model has a primary key column and a set of attribute columns.
Audience. A filter expression layered on top of a model — WHERE plan = 'pro' AND last_seen_at < CURRENT_DATE - INTERVAL '30 days'. Audiences are reusable across syncs and across destinations.
Sync. The full specification: which model (or audience), which destination, which field mappings, which sync mode, which schedule. A sync is the deployable unit.
Destination. The SaaS tool credentials + the destination object (Salesforce Contact, HubSpot Company, Intercom User, Marketo Lead, Facebook Custom Audience).

Sync modes you will encounter.

Insert. New rows are inserted into the destination; existing rows are untouched. Used for append-only destinations like logging or analytics events.
Update. Existing rows are updated; new rows are not inserted. Used when the destination owns identity creation (e.g. only update Salesforce contacts that already exist via lead capture).
Upsert. Insert new rows, update existing rows. The most common mode for customer attribute syncs.
Mirror. Make the destination match the model exactly — insert new, update changed, delete rows no longer in the model. The most powerful and the most dangerous; usually scoped to audiences (e.g. "the at-risk audience").
Delete only. Remove rows from the destination based on a "tombstone" model. Often used for GDPR delete propagation.

Identity resolution at the sync boundary.

External ID matching. The most common pattern: the warehouse primary key (salesforce_contact_id, hubspot_vid) is the same as the destination's primary key. The sync upserts on that key.
Email / phone matching. When the warehouse and the destination both store contact PII, syncs can match on email or phone. Brittle to changes (a user's email change creates a "new" record) but works for greenfield setups.
Custom external_id field. Hightouch and Census both support designating a custom external ID field in the destination (e.g. Marketo's external_id_c). The sync writes the warehouse PK there once, then matches on it forever.
Composite key matching. Some destinations (Salesforce, Marketo) support compound external IDs (e.g. account_id + region). Rarely used; useful when the same person lives in multiple tenants.

Idempotency — the contract that saves the team.

Stable primary key on every model. If the warehouse PK can change, the sync will double-write or fail to dedupe — every reverse ETL platform assumes the model PK is stable across runs.
Idempotent upserts. A retry on the same row must produce the same destination state. Most SaaS APIs support id based upsert; some require a "create-or-update" two-step.
Diff-only by default. Sync only the rows that changed since the last successful run. Saves API quota, reduces destination clutter, simplifies observability ("zero diffs is a healthy sync, not a broken one").

Change detection — three strategies.

Full refresh. Read the entire model every run, ship every row. Simple, expensive, almost never the right answer above 100k rows.
Diff-only (snapshot). Store a hash of every (PK, attribute) tuple on each successful run. On the next run, compare hashes and only ship the diffs.
CDC mirror. Subscribe to the warehouse's change-data-capture stream (Snowflake streams, BigQuery change streams, Databricks CDC) and apply diffs incrementally. The lowest-latency option; vendor support varies.

Worked example — defining a model with a stable PK and clean attributes

Question. Given a fact_orders table and a dim_customers table, write the right dbt model for a "current customer state" reverse ETL sync into Salesforce.

Input — fact_orders.

order_id	customer_id	amount	order_date
1	C1	50	2026-06-01
2	C1	30	2026-06-10
3	C2	200	2026-05-20

Input — dim_customers.

customer_id	name	salesforce_contact_id
C1	Alice	003A1
C2	Bob	003A2

Code.

-- WRONG — multiple rows per customer; will fail upsert.
SELECT
    c.salesforce_contact_id,
    o.amount,
    o.order_date
FROM dim_customers c
JOIN fact_orders o ON o.customer_id = c.customer_id;

-- RIGHT — one row per customer with aggregated attributes.
-- File: models/marts/reverse_etl_customer_state.sql
SELECT
    c.salesforce_contact_id,
    c.name,
    COUNT(o.order_id)                              AS lifetime_orders,
    COALESCE(SUM(o.amount), 0)                     AS lifetime_revenue,
    MAX(o.order_date)                              AS last_order_at,
    COUNT(o.order_id) FILTER
        (WHERE o.order_date >= CURRENT_DATE - 30)  AS orders_last_30d
FROM dim_customers c
LEFT JOIN fact_orders o ON o.customer_id = c.customer_id
GROUP BY c.salesforce_contact_id, c.name;

Step-by-step explanation.

The wrong model emits two rows for C1 (one per order). The Hightouch sync sees two rows with the same salesforce_contact_id, fails the "unique PK" assertion, and either rejects the sync or upserts the last row arbitrarily.
The right model wraps fact_orders in a GROUP BY on customer_id, collapsing every customer to one row. Attributes are aggregated: COUNT for orders, SUM for revenue, MAX for last order date.
LEFT JOIN preserves customers with zero orders. COALESCE(SUM(...), 0) turns the NULL sum into a clean 0 for downstream Salesforce automations.
COUNT(...) FILTER (WHERE ...) produces the "last 30 days" attribute without a separate subquery. Postgres / Snowflake / BigQuery support FILTER; SQL Server uses COUNT(CASE WHEN ... THEN 1 END).

Output (the reverse ETL model).

salesforce_contact_id	name	lifetime_orders	lifetime_revenue	last_order_at	orders_last_30d
003A1	Alice	2	80	2026-06-10	2
003A2	Bob	1	200	2026-05-20	0

Worked example — defining an audience from a model

Input — marts.reverse_etl_customer_state.

salesforce_contact_id	name	plan	lifetime_revenue	churn_risk	trial_ends_at
003A1	Alice	pro	8000	0.05	NULL
003A2	Bob	trial	0	NULL	2026-06-18
003A3	Cara	pro	1200	0.82	NULL
003A4	Dan	trial	0	NULL	2026-06-30

Code.

-- Audience: at_risk
SELECT * FROM {{ ref('marts.reverse_etl_customer_state') }}
WHERE churn_risk > 0.7;

-- Audience: high_value
SELECT * FROM {{ ref('marts.reverse_etl_customer_state') }}
WHERE lifetime_revenue > 5000;

-- Audience: active_trial
SELECT * FROM {{ ref('marts.reverse_etl_customer_state') }}
WHERE plan = 'trial'
  AND trial_ends_at IS NOT NULL
  AND trial_ends_at - CURRENT_DATE BETWEEN 0 AND 7;

# Three syncs, one model, three destinations.
- name: at_risk_to_intercom
  audience: at_risk
  destination: intercom
  sync_mode: mirror
  mappings:
    - source: churn_risk         -> Company.churn_risk_attr
    - source: lifetime_revenue   -> Company.ltv_attr

- name: high_value_to_facebook_ads
  audience: high_value
  destination: facebook_ads
  sync_mode: mirror
  mappings:
    - source: email              -> custom_audience.email_hash

- name: active_trial_to_iterable
  audience: active_trial
  destination: iterable
  sync_mode: mirror
  mappings:
    - source: trial_ends_at      -> User.trial_end_date
    - source: name               -> User.first_name

Step-by-step explanation.

The single underlying model marts.reverse_etl_customer_state is the source of truth. Every audience is a filter on top of it.
Audience at_risk mirrors to Intercom for CS alerting. The sync ships only the matching subset and removes the tag when a customer drops out of the audience (mirror mode).
Audience high_value mirrors hashed emails to a Facebook custom audience. Add/remove behaviour follows audience membership automatically.
Audience active_trial syncs to Iterable for an automated email sequence. The mirror mode adds users when they enter the trial window and removes them when the trial ends.
Each sync inherits the same model contract — change the column, every audience and sync notices on the next run.

Output.

Audience	Members	Destination	Effect
at_risk	003A3 (Cara)	Intercom	tagged as at_risk
high_value	003A1 (Alice)	Facebook	added to custom audience
active_trial	003A2 (Bob)	Iterable	trial-end sequence triggered

Worked example — change detection: snapshot diff vs full refresh

Input — assumptions.

Variable	Value
Total model rows	1,000,000
Rows changed per run	2,000 (0.2%)
Destination batch size	200 rows
Syncs per day	24
Destination API call cost	$0.001

Code.

Full refresh per run:
    api_calls    = ceil(1_000_000 / 200) = 5_000
    runs_per_day = 24
    daily_cost   = 5_000 * 24 * $0.001 = $120

Diff-only per run:
    api_calls    = ceil(2_000 / 200) = 10
    runs_per_day = 24
    daily_cost   = 10 * 24 * $0.001 = $0.24

Cost ratio: 500x cheaper with diff-only.
Time ratio: same — typical API latency dominated by call count, not payload size.

Step-by-step explanation.

Full refresh ships every row on every run. With 1M rows and 200/batch, the platform issues 5,000 API calls per run. 24 runs/day is 120,000 calls/day.
Diff-only ships only the 0.2% changed rows. 2,000 rows / 200 per batch = 10 API calls per run. 24 runs/day is 240 calls/day.
The math is independent of vendor — every reverse ETL platform that supports diff-only will produce this savings on a typical attribute-update workload.
Diff-only does require the platform to maintain the previous snapshot. The snapshot is typically stored in the reverse ETL platform's own metadata DB (Hightouch) or as a hidden audit table in the source warehouse (Census's "tracking table" pattern).

Output.

Strategy	API calls / day	Cost / day	Quota risk
Full refresh	120,000	$120	high (Salesforce 15k cap)
Diff-only	240	$0.24	very low

Reverse ETL interview question on idempotency

Solution Using the idempotent upsert contract

Idempotency is guaranteed if and only if:

1. Every model row has a stable primary key.
   - The PK is the natural identity (salesforce_contact_id),
     not a row number or a hash that changes between runs.
   - dbt test: unique + not_null on the PK column.

2. The sync mode is upsert (not insert) on a destination-side
   external ID field.
   - Salesforce: Upsert /sobjects/Contact/extId/{externalId}
   - HubSpot: Upsert /contacts/v1/contact/createOrUpdate/email/{email}
   - Marketo: leads/createOrUpdate with lookupField=externalId

3. The destination accepts a duplicate row as a no-op when
   nothing has actually changed.
   - Hightouch: built-in "skip unchanged rows" toggle.
   - Census: built-in idempotency cache.
   - RudderStack: ETag / If-Match conditional updates.

4. Retries on transient errors (5xx, network timeout) are
   safe because step 2 guarantees the second call lands the
   same destination state as the first.

5. Permanent errors (4xx) go to a dead-letter queue for
   manual inspection, NOT into the auto-retry loop.

Step-by-step trace.

Run	Outcome	Destination state	Idempotent?
Run 1 (initial)	success	1000 rows in Salesforce	n/a
Run 2 (no diff)	success	1000 rows unchanged	yes — zero API calls
Run 3 (1 row change)	success	1000 rows, 1 updated	yes — 1 API call
Run 4 (mid-run network blip)	partial fail at row 500	500 of 999 deltas applied	next run resumes
Run 4 retry	success	all deltas applied	yes — final state matches success-on-first-try

The fourth row shows the key behaviour: a half-applied sync is safe because each row's upsert is idempotent. The retry picks up the unfinished deltas without re-applying the already-applied ones.

Output:

Property	Behaviour with idempotency contract
Network blip	safe — retry resumes
Same model, two runs back-to-back	second run is a no-op
Schema change downstream	sync fails loudly, no half-update
Concurrent runs	platform locks the sync to one instance
Duplicate row in model	dbt test fails before sync starts

Why this works — concept by concept:

Stable PK — the primary key is the bridge between warehouse identity and destination identity. The whole upsert mechanism depends on it being stable across runs.
External ID upsert — every modern SaaS API offers an upsert primitive keyed on a custom external ID. Use it. Two-step "search-then-create-or-update" patterns are error-prone and not idempotent under concurrency.
Diff-only + skip-unchanged — short-circuits the destination call entirely when nothing has changed. A healthy sync run can legitimately make zero API calls.
Dead-letter queue — permanent errors (validation failure, missing required field) are not retried in a tight loop; they go to an inspect-and-fix queue. The retry loop is only for transient errors.
Concurrent-run lock — every reverse ETL platform single-instances each sync. Two parallel runs of the same sync would race on the diff snapshot and corrupt the next-run baseline.
Cost — idempotency is essentially free once the contract is in place. The cost is the up-front discipline of designing models with stable PKs and configuring destination external IDs.

Data
Topic — dimensional-modeling
Dimensional modelling problems (data engineering)

Practice →

3. Hightouch vs Census vs RudderStack — vendor comparison

Each vendor optimises for a different team shape — pick by who owns syncs and how dbt-native your stack is

The vendor matrix in one table.

Capability	Hightouch	Census	RudderStack
Destinations (2026)	200+	180+	200+ (events + reverse ETL)
dbt integration	strong (model picker, exposures)	strongest (dbt exposures native, "data-team first")	adequate
Audience builder	first-class visual UI	SQL-first, basic UI builder	basic
Sequences / journeys	yes (Hightouch sequences)	yes (Census audiences with priority)	partial
Identity resolution	strong (configurable matching)	strong (entity model)	event-stream-first
Self-hosted option	no (managed only)	no (managed only)	yes (RudderStack OSS + BYOC)
Combined CDP + reverse ETL	no	no	yes (event stream + reverse ETL)
Observability	strong (per-row, per-sync)	strong (sync alerts)	adequate
Pricing model	per-destination + MTU	per-row synced	per-MTU + events

Hightouch — audience-builder first, GTM-team first.

Strengths. Best-in-class audience builder UI (drag-and-drop filters, custom calculations); broadest destination catalogue; "Hightouch sequences" let marketing build journeys without leaving the tool; deep observability with row-level error inspection.
Best fit. Teams where the audience definitions live half in SQL and half in marketing's head; companies with 5+ destinations across CRM + marketing + ads.
Trade-offs. Managed-only (no self-host); MTU-based pricing surprises mid-market companies as their user count grows; Hightouch's UI-first audience editor can drift from the dbt definition of an entity if not policed.

Census — data-team first, dbt-native.

Strengths. Tightest dbt integration of the three — Census reads dbt_project.yml, recognises exposures, and surfaces sync metadata back into the dbt docs; "entity" model is a first-class concept; sync alerting is mature.
Best fit. Data teams that already live in dbt and want the warehouse-to-SaaS contract owned by analytics engineers, not marketing ops.
Trade-offs. Audience-builder UI is intentionally minimal (SQL is the way); fewer "GTM goodies" like multi-channel journeys; managed-only; per-row pricing means batch refreshes can sting.

RudderStack — open-source CDP + reverse ETL combined.

Strengths. Open-source under AGPLv3 with a managed plan; combines event streaming (Segment-style) with reverse ETL in one tool; self-hostable for BYOC / on-prem / compliance-driven shops; the only one of the three that can serve sub-30-second event reverse ETL.
Best fit. Companies that need both event collection and reverse ETL but want to avoid SaaS sprawl; compliance / BYOC use cases; engineering-heavy teams comfortable running infra.
Trade-offs. UI is less polished than Hightouch / Census; destination catalogue runs slightly behind on long-tail SaaS tools; the self-hosted operational cost is real (operate Postgres, Kubernetes, observability).

Pricing dimensions to model before procurement.

MTU (Monthly Tracked Users). Most platforms charge per unique entity synced per month. The metric grows roughly with total customer base.
Per-row synced. Census's primary metric. Drives a "diff-only is required" discipline because full refresh becomes ruinously expensive.
Per-destination. Hightouch's standard plans cap the number of destinations on lower tiers. Multi-channel companies feel this fast.
Per-seat. Both Hightouch and Census charge per audience-builder seat above a baseline.
Events (RudderStack). Event-stream pricing is per event, not per unique user. Plan for both axes.

Self-hosted (RudderStack OSS) vs managed trade-off.

Self-hosted wins for. BYOC compliance, data-residency, "all data must stay in our VPC," low cost at very large MTU counts (>1M).
Managed wins for. Speed (live in a day vs a quarter), no infra ops burden, faster destination roll-outs, no upgrade cycles.
Hybrid pattern. Many shops run RudderStack OSS for event collection (zero per-event vendor cost) and Hightouch managed for reverse ETL (fastest catalogue + audience UI).

Worked example — picking Hightouch when GTM owns audiences

Input — the company profile.

Property	Value
Audience owners	Revenue ops (non-SQL)
Destinations	Salesforce, HubSpot, Marketo, Outreach, FB Ads, LinkedIn Ads
Warehouse	Snowflake + dbt
Sync latency	30 minutes acceptable
Self-host requirement	none

Code.

Decision matrix:

| Need                       | Hightouch | Census | RudderStack |
|----------------------------|-----------|--------|-------------|
| Drag-drop audience builder | strong    | basic  | basic       |
| 6+ destinations            | yes       | yes    | yes         |
| dbt exposure surfacing     | yes       | best   | adequate    |
| Multi-channel sequences    | yes       | partial| partial     |
| No-SQL revenue ops users   | strong    | weak   | weak        |

Decision: Hightouch wins on (1) audience builder, (4) sequences,
(5) non-SQL audience editors. Census's dbt-first stance is a real
strength but the GTM team owns audiences in this org.

Step-by-step explanation.

The team's bottleneck is "GTM ops cannot self-serve audiences." Hightouch's audience builder is the only one of the three optimised for that exact persona.
Census's strength (dbt-native) does not help when the audience layer is owned outside the data team. The model is still in dbt; the audience-on-top-of-model is what's UI-driven.
RudderStack's event-stream story is not relevant — this team is not building real-time personalisation, just attribute syncs at 30-minute cadence.
The decisive feature is the audience builder UI, with Hightouch sequences as a bonus for multi-step marketing journeys.

Output.

Decision	Hightouch
Why	audience builder + sequences + destination catalogue
Estimated MTU cost	$$ (mid-market plan)
Implementation timeline	4 weeks to first sync

Worked example — picking Census when dbt is the source of truth

Question. Given the company profile (dbt-first, analytics engineering owns audiences, strict change management), justify Census over Hightouch. List the decisive dbt integration features.

Input — the company profile.

Property	Value
Audience owners	Analytics engineering (SQL-fluent)
Source of truth	dbt models, branch-protected
Destinations	Salesforce, Iterable, Customer.io
Sync latency	1 hour acceptable
Compliance	strict — every change reviewed

Code.

Census dbt-native features that decided it:

1. dbt project sync — Census reads dbt_project.yml directly.
   Models appear in Census with the same name as in dbt.

2. dbt exposures — every Census sync is automatically surfaced
   as a dbt exposure. Lineage in dbt docs shows the destination.

3. Git-backed sync definitions — sync YAML lives in the dbt
   repo, change-managed via PR.

4. dbt tests propagate — failing dbt tests block the sync.
   Census never ships a failing-test row to a destination.

5. Entity model — Census's "entity" concept is the equivalent
   of a dbt model with documented PK + columns. Discoverable
   across the team.

Step-by-step explanation.

The data team's discipline is "everything ships via PR." Census's git-backed sync definitions extend that discipline to the reverse ETL layer.
Hightouch supports a Terraform provider for sync-as-code, but the UI-first culture pulls non-engineers off the git workflow. Census's SQL-first culture matches the team.
dbt exposures inside Census are decisive — every destination becomes a known consumer in the lineage graph. Census surfaces "this sync depends on this model" automatically.
Failing dbt tests blocking the sync is the killer feature for compliance — it means a regression in the model never silently corrupts a downstream SaaS field.

Output.

Decision	Census
Why	dbt-native + git-backed syncs + exposures + test-gating
Estimated cost	$$ (per-row pricing acceptable at this volume)
Implementation timeline	6 weeks to production sync

Worked example — picking RudderStack OSS for BYOC compliance

Question. Given the company profile (PII must stay in VPC, single tool preferred for events + syncs), justify RudderStack OSS over the managed options.

Input — the company profile.

Property	Value
Compliance	PII must stay in customer VPC
Existing event tool	considering Segment replacement
Destinations	Salesforce Health Cloud, HubSpot, internal API
Sync latency	5 minutes for high-priority
Team	engineering-heavy, comfortable running infra

Code.

Why RudderStack OSS wins on this profile:

1. Self-hosted in customer VPC.
   - No PII leaves the customer's cloud account.
   - Audit trail end-to-end within customer-owned storage.

2. Combined event stream + reverse ETL.
   - Single tool covers Segment-like event collection AND
     Hightouch-like warehouse reverse ETL.
   - One destinations catalogue, one UI, one set of credentials.

3. Event-stream reverse ETL.
   - Sub-30-second latency on high-priority warehouse changes
     via the event-stream path (not the batch path).

4. AGPLv3 source-available.
   - Customer can patch, audit, and extend.
   - No vendor lock-in for compliance-critical features.

Trade-offs accepted:
- Operate Postgres, Redis, K8s yourself.
- Destination catalogue runs slightly behind Hightouch on
  long-tail tools.
- UI is less polished — engineers, not marketers, configure syncs.

Step-by-step explanation.

The PII-in-VPC requirement removes Hightouch and Census from contention immediately — both are managed-only.
The combined event-stream + reverse ETL story removes Segment from the picture and consolidates spend.
RudderStack OSS's event-stream reverse ETL path is the only sub-30-second option in this comparison — relevant for the "high-priority sync" use case.
The trade-off is operational burden. The team must own the Postgres metadata DB, Redis broker, and Kubernetes orchestration. An engineering-heavy org accepts this.

Output.

Decision	RudderStack OSS
Why	self-hosted compliance + combined CDP + sub-30s reverse ETL
Estimated cost	infrastructure + 0.5 SRE FTE
Implementation timeline	8 weeks to production

Reverse ETL interview question on the buy-vs-build decision

A senior interviewer often frames it as: "Your CTO is asking whether we can just build reverse ETL in-house with Airflow + Python + the destination SDKs. Walk me through the buy-vs-build decision."

Solution Using the operational-burden lens

The build-it-yourself stack:

1. Airflow / Dagster orchestration.
2. Custom Python writers for each destination API.
3. Snapshot diff engine (you build it).
4. Queue + worker pool with retry semantics (you build it).
5. Dead-letter queue + inspection UI (you build it).
6. Per-row error logging (you build it).
7. Schema-change detection (you build it).
8. Audit log + lineage (you build it).
9. Audience builder UI for non-engineers (... you build it).
10. PII tagging + governance UI (you build it).

The buy stack:

1. Hightouch / Census / RudderStack subscription.
2. Sync configuration (a week of work).

The break-even calculation:

- Year 1 build cost: 2 senior engineers × 6 months = ~$300k.
- Year 1 buy cost:   ~$30k–$80k subscription, depending on MTU.
- Year 2 build cost: 1 engineer × full year maintenance = ~$200k.
- Year 2 buy cost:   ~$50k–$120k subscription.

Buy wins decisively unless:
- You have a destination not on the vendor catalogue (rare).
- You have a sub-second latency requirement (use a feature store).
- You have a compliance constraint requiring on-prem (use OSS).

Step-by-step trace.

Component	Buy time	Build time
Destination connectors	1 day per destination (config)	2 weeks per destination (code + tests)
Diff engine	included	4 weeks
Queue + retry	included	6 weeks
Dead-letter inspection	included	2 weeks
Audience builder UI	included	12+ weeks (and your data team has to maintain it)
Schema-change detection	included	4 weeks

The "build everything" path lands at 6–9 months for a v1 covering 5 destinations with no UI. The "buy" path lands at 4–6 weeks for the same scope plus an audience UI.

Output:

Year	Build cost	Buy cost
1	~$300k	~$50k
2	~$200k	~$80k
3	~$200k	~$100k

Why this works — concept by concept:

Connector breadth — vendors maintain hundreds of destination integrations as their full-time job. A 2-engineer team building from scratch will cover 5–10 destinations at best in year 1.
Diff engine is the moat — every reverse ETL platform's secret sauce is the diff/snapshot/incremental detection logic. Building a reliable one is a 6-month research project, not a weekend hack.
Audience UI — the moment a non-engineer needs to ship an audience, you need a UI. Building that internally is a years-long product investment that has nothing to do with your company's actual product.
Observability — per-row error tracking, dead-letter queues, sync success ring charts — all included in the vendor stack. Building them stalls your data team for months.
Compliance escape hatch — RudderStack OSS exists precisely for the rare cases where vendor managed cannot work. Use OSS, not in-house build.
Cost — over a 3-year window the buy path is 3–5× cheaper and ships in 1/10 the time. The only counter-arguments are scale (>10M MTU and you renegotiate hard) or compliance (and OSS solves that).

ETL
Topic — api-integration
API integration problems (data engineering)

Practice →

4. Sync architecture — incremental detection, queues, rate limits

A sync is a diff engine plus a queue plus a worker pool plus a rate-limited destination API — every reverse ETL platform implements the same four-stage pipeline

Stage 1 — warehouse query and snapshot detection.

Query. The model SQL (or audience-filtered model SQL) runs against the warehouse. Result is materialised either into a temp table or streamed.
Snapshot store. The previous run's (pk, hash(attributes)) set lives somewhere — a hidden table in the warehouse, a Postgres metadata DB in the vendor's infra, or a CDC stream offset.
Diff classification. For each current row: if PK absent in snapshot → INSERT; if PK present and hash differs → UPDATE; for each snapshot PK absent in current → DELETE (or "tombstone").

Stage 2 — staging / queue.

Per-sync queue. Each sync gets its own queue, single-instanced. No parallel runs of the same sync.
Back-pressure absorption. When the destination's API is slow, the queue grows; workers pull at the destination's pace, not the warehouse's pace.
Persistence. Queues persist to disk so a vendor restart does not lose in-flight rows.

Stage 3 — worker pool.

Worker concurrency. Configured per destination; usually 1–8 parallel workers per sync.
Batch packing. Workers pack queue rows into destination-specific batches (Salesforce: 200/batch, HubSpot: 100/batch, Marketo: 300/batch).
Token-bucket rate limiter. Each worker checks the destination's quota before issuing the call.

Stage 4 — destination API.

Auth. OAuth, API key, service account — refreshed automatically by the platform.
Rate limit response. 429 (Too Many Requests) triggers exponential backoff and a slowdown of the worker pool.
Per-row error response. 4xx errors on specific rows are recorded as row-level failures, surfaced in the sync log, and either retried (transient) or dead-lettered (permanent).

Destination rate limits in the wild (2026 baselines).

Destination	Limit	Notes
Salesforce	15,000 / 24h (standard)	per-org, all APIs share
HubSpot	100 / 10s + 250k / day	per-portal
Marketo	100 / 20s + 50k / day	per-instance
Intercom	1,000 / minute	per-app
Iterable	4 / second list endpoints	varies by endpoint
Facebook Custom Audience	200,000 users / API call	batched mode
Slack	1 / second per webhook	basic tier

Retry semantics.

Transient (5xx, 429, network timeout) — retry with exponential backoff. Typical: 1s → 2s → 4s → 8s → 16s → 32s, then dead-letter.
Permanent (4xx with validation error) — log and dead-letter immediately. Retrying will not help.
Auth (401, token expired) — refresh the token and retry once, then alert.
Quota exhausted (429 with daily-cap header) — pause the sync until the quota window resets; alert if the window is >12 hours.

Latency tiers.

Hourly batches. Default for most syncs. 5–60 minutes end-to-end.
Sub-minute batches. Census + small models. 30 seconds–5 minutes.
CDC mirror. Continuous; reflects warehouse changes in seconds.
Event-stream reverse ETL. RudderStack's path; reflects in 1–30 seconds.

Worked example — the diff engine in pseudo-code

Question. Write a pseudo-code sketch of a diff engine that takes (current_rows, previous_snapshot) and emits classified events. Explain how it handles deletes.

Input.

previous_snapshot:
  C1 -> hash("Alice|pro|0.05")
  C2 -> hash("Bob|trial|null")
  C3 -> hash("Cara|pro|0.40")

current_rows:
  C1 -> ("Alice", "pro", 0.05)         # unchanged
  C2 -> ("Bob",   "pro", 0.10)         # changed (trial -> pro)
  C4 -> ("Dan",   "trial", null)       # new
  # C3 missing -> deleted

Code.

def diff_engine(current_rows, previous_snapshot):
    """Yield classified change events."""
    current_keys  = set(current_rows.keys())
    previous_keys = set(previous_snapshot.keys())

    # INSERTs — PKs in current but not previous.
    for pk in current_keys - previous_keys:
        yield ("INSERT", pk, current_rows[pk])

    # UPDATEs — PKs in both, hash differs.
    for pk in current_keys & previous_keys:
        new_hash = row_hash(current_rows[pk])
        if new_hash != previous_snapshot[pk]:
            yield ("UPDATE", pk, current_rows[pk])
        # else: unchanged, emit nothing (this is the big saving).

    # DELETEs — PKs in previous but not current.
    # Only if sync_mode == "mirror"; otherwise skip deletes.
    for pk in previous_keys - current_keys:
        yield ("DELETE", pk, None)

    # Persist new snapshot for next run.
    new_snapshot = {pk: row_hash(row) for pk, row in current_rows.items()}
    save_snapshot(new_snapshot)

Step-by-step explanation.

The set difference current - previous yields rows present this run but not last run — INSERTs.
The set intersection plus hash comparison yields rows present in both runs whose attributes changed — UPDATEs. Unchanged rows are skipped silently (zero API calls).
The set difference previous - current yields rows present last run but absent this run — DELETEs. Only emitted in mirror sync mode; upsert mode ignores them.
The new snapshot is persisted at the end. If the run crashes before this point, the next run sees the same previous snapshot and re-classifies the same diffs (idempotent recovery).
The row hash function is typically MD5 / xxHash over the JSON serialisation of attributes in a canonical column order. Hash collisions are theoretically possible; in practice the rate is negligible at billion-row scale.

Output.

Event	PK	Attributes
INSERT	C4	(Dan, trial, NULL)
UPDATE	C2	(Bob, pro, 0.10)
DELETE	C3	—

Worked example — the rate limiter and the 429 backoff loop

Question. Sketch a worker loop that drains a queue of upsert events into a Salesforce-like API with a 15k/24h limit, handles 429 responses, and emits to dead-letter on permanent errors.

Input.

Queue items:
  - upsert C1 with payload P1
  - upsert C2 with payload P2
  - upsert C3 with payload P3 (will return 400 — invalid email)
  - upsert C4 with payload P4

Destination state:
  - quota_remaining = 14_998
  - quota_resets_at = 24h from now

Code.

def worker(queue, destination, rate_limiter, dead_letter):
    while True:
        event = queue.pop()
        if event is None:
            break

        # 1. Respect the destination's rate limit.
        rate_limiter.acquire(destination)

        # 2. Make the API call.
        backoff = 1
        for attempt in range(7):
            try:
                response = destination.upsert(event.pk, event.payload)
                if response.status == 200:
                    break
                if response.status == 429:
                    # Rate limited — exponential backoff.
                    sleep(backoff)
                    backoff = min(backoff * 2, 60)
                    continue
                if 400 <= response.status < 500:
                    # Permanent error — dead letter.
                    dead_letter.push(event, response.body)
                    break
                if 500 <= response.status:
                    # Transient server error — retry.
                    sleep(backoff)
                    backoff = min(backoff * 2, 60)
                    continue
            except NetworkTimeout:
                sleep(backoff)
                backoff = min(backoff * 2, 60)
                continue
        else:
            # Out of attempts — dead letter.
            dead_letter.push(event, "max_retries_exceeded")

Step-by-step explanation.

rate_limiter.acquire blocks the worker until the token bucket has a slot. Implementation is typically a Redis script that decrements a per-destination counter and refills it at the destination's rate.
The retry loop runs up to 7 attempts. On 429, the worker sleeps and retries (backoff 1s → 2s → 4s → ... capped at 60s).
On 5xx transient server errors, the worker also retries — server-side issues are usually self-healing within seconds.
On 4xx permanent errors (validation failure, malformed payload, missing required field), the worker stops retrying and pushes the event to the dead-letter queue for human inspection.
Network timeouts (no response) are treated as transient — the worker retries with backoff.
If all 7 attempts fail, the event is dead-lettered with max_retries_exceeded so on-call has visibility.

Output (events that reach the destination vs dead-letter).

Event	Destination state	Dead-letter?
C1	upserted	no
C2	upserted	no
C3	rejected (400 invalid email)	yes
C4	upserted	no

Worked example — back-pressure from a slow destination

Input.

Variable	Value
Warehouse output rate	10,000 rows/min
Destination accept rate	100 rows/min
Initial queue depth	0
Alert threshold	50,000 rows

Code.

def queue_growth(warehouse_rate, destination_rate, minutes):
    depth = 0
    growth_per_min = warehouse_rate - destination_rate
    log = []
    for minute in range(1, minutes + 1):
        depth += growth_per_min
        log.append((minute, depth))
    return log

# Compute for one hour:
growth = queue_growth(10_000, 100, minutes=60)
# Alert fires when depth crosses 50_000.
alert_minute = next(m for m, d in growth if d > 50_000)
print(f"Queue depth alert at minute {alert_minute}")
# -> Queue depth alert at minute 6

Step-by-step explanation.

Net growth per minute = warehouse output - destination accept = 10000 - 100 = 9900 rows/min.
After 1 min: 9,900 rows queued. After 5 min: 49,500 queued. After 6 min: 59,400 — crosses the 50k alert threshold.
The alert at minute 6 gives on-call 50 minutes of headroom before the queue passes a typical "platform refuses to enqueue" limit of ~500k rows.
The right remediation depends on the cause: (a) destination is rate-limited — wait for the quota to reset and accept the lag; (b) destination is genuinely broken — pause the sync until the destination is healthy; (c) warehouse is producing duplicates — fix the model.
Without the queue-depth alert the team only learns about the problem when the platform errors out at 500k+ — too late, downstream is already stale by hours.

Output.

Minute	Queue depth	Alert?
1	9,900	no
3	29,700	no
5	49,500	no
6	59,400	yes
60	594,000	platform errors

Reverse ETL interview question on rate-limit-aware design

A senior interviewer often asks: "Salesforce has a 15k API calls per day quota and our customer state model has 200k rows. How do you design a sync that fits inside the quota?"

Solution Using batching + diff-only + audience filtering

The math first:

  raw rows                            = 200_000
  Salesforce upsert batch size        = 200 rows / call
  full refresh calls                  = 1_000 calls / run
  diff-only on 0.5% changed rows      = 1_000 changed rows
  diff-only batch calls               = ceil(1_000 / 200) = 5 calls / run
  hourly cadence                      = 24 runs / day
  daily API calls                     = 5 * 24 = 120 calls / day

  Headroom under the 15k quota: 124x.

The design:

1. Composite Tooling API batching.
   - Use Salesforce's Composite/sObject Collections API:
     200 records per call vs 1 record per Standard upsert.

2. Diff-only sync mode (no full refresh).
   - Reverse ETL platform stores last-run snapshot.
   - Ship only rows whose attribute hash changed.

3. Audience scoping.
   - Many syncs only need the "active" subset of customers.
   - Filter at the audience layer (plan != 'churned')
     so the diff engine compares smaller sets.

4. Cadence sized to business need.
   - Sales routing: every 30 minutes.
   - Account health: every 6 hours.
   - LTV refresh: every 24 hours.
   - Do not over-spec freshness; quota is finite.

5. Per-sync quota guard.
   - Configure the reverse ETL platform's "max API calls per
     window" knob to a sub-quota share per sync.
   - Hightouch and Census both expose this; RudderStack via config.

Step-by-step trace.

Design choice	Effect on quota
Full refresh	1,000 calls/run × 24 = 24,000/day — over quota
Diff-only	5 calls/run × 24 = 120/day
Audience scoping	reduces diff size further
Per-sync quota guard	prevents any one sync from monopolising quota
Hourly vs 30-min cadence	doubles or halves daily API calls

The combination of (2) and (4) is decisive. Diff-only converts the metric from "rows in the model" to "rows that changed," which on most attribute syncs is 0.1–2% of the model.

Output:

Strategy	Calls / day	Inside 15k quota?
Full refresh hourly	24,000	no
Diff-only hourly	120	yes
Diff-only every 30m	240	yes
Diff-only every 5m	1,440	yes
Full refresh every 5m	288,000	catastrophic

Why this works — concept by concept:

Batched upsert — Salesforce's composite endpoint is the single biggest lever. Going from 1 row per call to 200 rows per call drops the call count by 200×.
Diff-only sync — the second biggest lever. Only ship rows that actually changed. Drops the call count by 50–500× on typical attribute workloads.
Audience filtering — shrinks the model to the rows that matter. Skipping churned customers saves both diff computation and quota.
Cadence sizing — the third lever. Match the sync frequency to the actual business cadence; "fresh every 5 minutes" is rarely needed for a CRM attribute.
Per-sync quota guard — defensive design. Even if one sync misbehaves (e.g. a model bug emits 200k diffs), the guard prevents it from burning the org-wide quota and breaking unrelated syncs.
Cost — the design is essentially free. All the levers are configuration, not code. The cost is the discipline to model the math up-front for each new sync.

ETL
Topic — etl
ETL design problems (data engineering)

Practice →

5. Governance, observability, and failure modes

A sync that has no governance, no observability, and no defined failure modes is not a data product — it is a time bomb

Governance — five non-negotiables.

Field-level PII tagging. Every column tagged pii=email | phone | address | name | ssn. Tags propagate to the sync layer so destinations can enforce per-tag policy.
Per-destination policy. "Email PII can sync to Marketo; SSN PII cannot sync to anything." Hightouch and Census both support sync-level allow/deny rules.
Audience approval. New audiences > 10k members require analytics-engineering sign-off. Catches "I just synced 200k users to Facebook by accident."
GDPR delete propagation. A user's right-to-delete must reach every destination. The platform must support a "delete pipeline" sync (model = users_to_delete, mode = delete-only, fanned out to every destination).
Audit log. Every sync edit, schedule change, and credential rotation is logged with actor + timestamp.

Observability — six SLIs to track.

Sync success rate. Percent of runs that finished without a top-level error. Target: >99.5%.
Row-error rate. Percent of rows in a successful run that failed (typically destination 4xx validation). Target: <1%.
Freshness lag. Time since last successful run vs the scheduled cadence. Target: <2× cadence.
Queue depth. Pending rows waiting for the worker pool. Leading indicator of destination slowness.
Rejected payload sample. Stratified sample of dead-letter events for human inspection.
Latency p50 / p99. Wall-clock time from model row produced to destination row accepted.

Failure modes — the four most common.

Mapping drift. Warehouse column renamed; destination field still expects the old name; sync silently writes NULL or fails.
Schema drift. Column type changed (INT → BIGINT, VARCHAR(50) → VARCHAR(500)); destination rejects with type-mismatch error.
Row-cap breach. Audience suddenly grows from 5k to 200k members because a filter became overly permissive; destination quota burns out.
Credential expiry. OAuth refresh token expires; sync fails with 401; team finds out hours later when freshness lag alert fires.

Catalog + lineage — surfacing syncs as dbt exposures.

Every sync is a known consumer of one or more dbt models. The standard surface is a dbt exposure:

# models/marts/exposures.yml
exposures:
  - name: salesforce_lead_score_sync
    type: application
    owner:
      name: Analytics Engineering
      email: ae@example.com
    depends_on:
      - ref('reverse_etl_customer_state')
    description: |
      Hightouch sync into Salesforce.Contact.lead_score__c.
      Cadence: every 30 minutes.
      On-call: data-team rotation.

Cost guardrails.

Per-sync row caps. "This sync will never ship more than 50k rows per run; abort if it tries."
Audience size caps. "This audience will never include more than 100k members; alert if it does."
Quota share caps. "This sync will use no more than 30% of the destination's daily API quota."
Frequency caps. "Even if scheduled hourly, no more than 24 runs per day."

Worked example — propagating PII tags from dbt to the sync layer

Input.

# dbt model schema.yml
version: 2

models:
  - name: dim_users
    columns:
      - name: user_id
        tests: [unique, not_null]
      - name: email
        meta:
          pii: email
          contains_pii: true
      - name: ssn
        meta:
          pii: ssn
          contains_pii: true

Code.

# Census destination policy.
destinations:
  marketo:
    allowed_pii_tags: [email, name]
    blocked_pii_tags: [ssn, phone, address]

  experimentation_tool:
    allowed_pii_tags: [name]
    blocked_pii_tags: [email, ssn, phone, address]
    # Note: email is blocked here.

# Census sync definition.
syncs:
  - name: users_to_marketo
    source: dim_users
    destination: marketo
    mappings:
      - source: email   -> Lead.Email          # OK — email allowed in Marketo
      - source: name    -> Lead.Name           # OK

  - name: users_to_experimentation
    source: dim_users
    destination: experimentation_tool
    mappings:
      - source: name    -> User.display_name   # OK
      - source: email   -> User.identifier     # BLOCKED — sync refuses to compile

Step-by-step explanation.

The dbt meta block tags the column with structured PII metadata. Census's dbt project reader picks up the tag automatically — no second source of truth.
The destination policy lists allowed and blocked PII categories per destination. Marketo accepts email + name; the experimentation tool accepts only name.
When the sync to Marketo compiles, every mapping is checked against the policy. Email → Lead.Email is allowed; the sync ships.
When the sync to the experimentation tool compiles, the email mapping triggers a policy violation. Census refuses to compile the sync; the engineer sees a clear error and either removes the mapping or escalates for an exception approval.
The policy is enforced at compile time, before any row hits a network. A misconfigured sync never reaches the destination.

Output.

Sync	Policy decision
users_to_marketo	compiles + ships
users_to_experimentation	refused to compile (email blocked)

Worked example — the freshness SLA alert

Question. Configure a freshness alert for the salesforce_lead_score_sync (cadence 30 min, SLA 2h) and walk through the on-call response when it fires.

Input.

Sync	Cadence	SLA	Freshness now
salesforce_lead_score_sync	30 min	2h	3h 15m ago

Code.

# Census alert definition (illustrative).
alerts:
  - name: lead_score_sync_freshness
    sync: salesforce_lead_score_sync
    condition: minutes_since_last_success > 120  # 2h SLA
    severity: page
    notify:
      - pagerduty: data-team-oncall
      - slack: "#data-alerts"
    runbook: |
      Sync has not succeeded in over 2 hours.
      Steps:
        1. Check Census dashboard for recent error.
        2. If 401 — refresh OAuth credential.
        3. If 429 — wait for quota reset; backfill afterwards.
        4. If model SQL error — open dbt repo, fix, redeploy.
        5. If destination outage — pause sync, monitor status page.

Step-by-step explanation.

The alert condition minutes_since_last_success > 120 measures actual freshness against the 2h SLA. The 30-minute cadence is the target; the SLA is the deadline.
When the alert fires, PagerDuty pages the on-call data engineer and posts to the Slack channel. The runbook is in the alert body, not in a separate wiki.
The on-call reads the Census dashboard, identifies the failure category (auth, quota, model error, destination outage), and applies the matching runbook step.
The runbook covers the four most-common failure modes. Steps 1–3 are operational; step 4 escalates to the model owner; step 5 escalates to the destination vendor.

Output (timeline of the on-call response).

Time	Event
03:00	Last successful run.
03:30	Scheduled run fails — 401 (token expired).
04:00	Second scheduled run fails — 401.
04:30	Third scheduled run fails — 401.
05:00	Freshness alert fires (2h SLA breached). PagerDuty pages on-call.
05:05	On-call reads runbook, refreshes OAuth credential.
05:10	Sync retries successfully. Freshness lag drops to 10 minutes.

Worked example — schema drift catches before deploy

Input.

# dbt contract on the model.
models:
  - name: reverse_etl_customer_state
    config:
      contract:
        enforced: true
    columns:
      - name: salesforce_contact_id
        data_type: varchar
        tests: [unique, not_null]
      - name: name
        data_type: varchar
      - name: lifetime_orders
        data_type: integer
      - name: lifetime_revenue
        data_type: numeric
      - name: last_order_at
        data_type: timestamp

Code.

-- Developer's PR — renames lifetime_revenue.
-- File: models/marts/reverse_etl_customer_state.sql
SELECT
    c.salesforce_contact_id,
    c.name,
    COUNT(o.order_id)                AS lifetime_orders,
    COALESCE(SUM(o.amount), 0)       AS lifetime_value,   -- renamed!
    MAX(o.order_date)                AS last_order_at
FROM dim_customers c
LEFT JOIN fact_orders o ON o.customer_id = c.customer_id
GROUP BY c.salesforce_contact_id, c.name;

Step-by-step explanation.

The developer renames lifetime_revenue to lifetime_value in the SELECT clause.
dbt CI runs dbt build. The contract check inspects the actual output schema against the declared columns: list.
The output column lifetime_value does not match the declared lifetime_revenue. dbt fails the build with a clear error: "column lifetime_revenue not produced; column lifetime_value produced unexpectedly."
The CI failure blocks the merge. The developer either reverts the rename or files a coordinated migration (rename in dbt + rename mapping in sync + cutover plan).
Without the contract, the rename would merge, the next sync run would silently ship NULL for lifetime_revenue (Salesforce field overwritten with NULL), and the marketing team would discover the bug three days later when their nurture sequence fires for everyone.

Output.

Stage	Outcome
Pre-contract	rename merges, sync silently writes NULL, downstream stale
With contract	rename blocked in CI, coordinated migration required

Reverse ETL interview question on the sync as a data product

A senior interviewer often asks: "How do you turn a one-off sync from a side-project into a production data product? What does the full lifecycle look like?"

Solution Using the data-product lifecycle

The data-product lifecycle for a reverse ETL sync:

1. INTAKE
   - Consumer team files a sync request.
   - Required fields: model, destination, fields, cadence,
     SLA, on-call owner.

2. DESIGN
   - Analytics engineer reviews the model PK + idempotency.
   - PII tags audited; destination policy verified.
   - Audience defined if filtering required.
   - dbt contract on the source model.
   - Cost estimate (quota + MTU).

3. BUILD
   - Sync YAML / config committed to git.
   - CI runs dbt build + sync linting.
   - PR review by analytics engineering.

4. DEPLOY
   - Sync deployed to staging destination first.
   - Manual QA on 10 sample rows.
   - Cut over to production destination.

5. MONITOR
   - dbt exposure surfaced in catalog.
   - Freshness alert + row-error alert configured.
   - Queue-depth alert configured.
   - On-call runbook attached.

6. ITERATE
   - Quarterly review of sync health metrics.
   - Audience drift review (size still in expected range?).
   - Destination policy review (PII still compliant?).
   - Cost review (still inside quota envelope?).

7. RETIRE
   - When the consumer no longer needs it: archive the sync,
     drop the dbt exposure, document the deprecation.

Step-by-step trace.

Phase	Owner	Output
Intake	Consumer team + AE	sync request ticket
Design	Analytics engineering	sync design doc
Build	Analytics engineering	sync YAML + PR
Deploy	Analytics engineering	staging then prod
Monitor	Data on-call	dashboards + alerts
Iterate	Analytics engineering	quarterly review notes
Retire	Analytics engineering	deprecation note

The discipline is the same as any backend service. The vocabulary borrows from product management (intake, MVP, monitoring, deprecation) more than from data engineering (model, refresh, materialise).

Output:

Artifact	Where it lives
Sync config	dbt repo / sync YAML
dbt contract	model schema.yml
dbt exposure	exposures.yml
Alerts	observability platform
Runbook	alert body + wiki
Cost budget	per-sync row cap + quota share
On-call rota	PagerDuty schedule

Why this works — concept by concept:

Intake gates entry — not every "we want a sync" idea becomes a sync. The intake form forces the consumer to articulate model, destination, SLA, and ownership before any engineering time is spent.
dbt contracts gate change — every sync model has an enforced contract. Drift is caught at PR time, not at production-failure time.
Exposures surface lineage — the data catalog knows every sync. When a model changes, the catalog shows every downstream sync that will be affected.
Alerts surface failure — freshness lag, row-error rate, and queue depth are the three SLIs. Every sync has them; on-call wakes up to them.
Quarterly review surfaces drift — audiences grow, costs shift, PII policy evolves. Quarterly review catches slow drift before it becomes an incident.
Retirement is explicit — syncs are retired explicitly, not abandoned. A retired sync is archived in git and removed from exposures so the catalog stays accurate.
Cost — the discipline is overhead. For a low-stakes internal sync, the full lifecycle is overkill. For any sync touching customer-facing automation, the lifecycle is the floor.

Data
Topic — data-transformation
Data transformation problems (data engineering)

Practice →

Cheat sheet — reverse ETL recipes

Lead score → Salesforce. Model fct_lead_score (one row per Salesforce contact) → audience "lead_score >= 80" → upsert into Contact.lead_score__c. Cadence: 30 minutes. Use composite API batching for 200 rows/call.
Account churn risk → Intercom. Model dim_accounts with churn_risk → audience "churn_risk > 0.7" → mirror sync sets Company.churn_risk_tag = at_risk and clears the tag when the account drops out of the audience.
High-value users → Facebook custom audience. Model dim_users joined to fct_user_revenue → audience "ltv_usd > 5000" → mirror sync hashes emails and pushes to a Meta custom audience. Reflects add/remove automatically on each run.
Slack high-value signup alert. Model fct_signups filtered to "plan = pro AND first_seen_at >= today" → RudderStack event sync → Slack webhook posts to #sales-alerts with the new account name + plan + region.
Marketing suppression list. Model dim_users filtered to "opted_out = true OR gdpr_deleted = true" → mirror sync to every marketing destination's suppression list (Marketo, Iterable, Customer.io, Mailchimp).
Reverse ETL → product analytics. Model marts.user_cohorts with (user_id, cohort_label) → upsert into Amplitude's cohorts API, mirrored to Mixpanel's cohort endpoint. Lets PMs filter funnels by warehouse-defined cohorts.
GDPR delete pipeline. Model users_to_delete (one row per requested deletion) → delete-only sync fanned out to Salesforce, HubSpot, Marketo, Intercom, Iterable, Facebook. Idempotent: a row deleted twice is a no-op.
Trial-ending sequence trigger. Model dim_users filtered to "plan = trial AND trial_ends_at BETWEEN today AND today + 7" → mirror sync to Iterable user property trial_end_date. Iterable workflow fires the in-app + email sequence.
Customer attribute fan-out. Single model marts.customer_attributes (one row per customer) → multiple syncs to Salesforce, HubSpot, Intercom, Iterable each picking the columns they need. One source, many destinations.
Sales territory routing. Model dim_accounts with territory_code → upsert into Salesforce Account.RoutingTerritory__c. Pairs with a Salesforce assignment rule that reads the field at lead creation.
NPS score sync. Model marts.nps (one row per account with rolling NPS) → upsert into Salesforce Account.nps_rolling__c. Customer success team filters Salesforce dashboards by NPS bucket.
Webhook fan-out. Model fct_account_events (one row per significant account event) → RudderStack event sync → internal API webhook, Slack channel, and Salesforce task creation in parallel.

Frequently asked questions

Is reverse ETL the same as a CDP?

Do I need a customer data warehouse before reverse ETL?

How is Hightouch different from Census?

Can I build reverse ETL myself with Airflow + APIs?

What latency can reverse ETL realistically deliver?

How do I handle GDPR deletes through reverse ETL?

Practice on PipeCode

Drill the ETL practice library → for the warehouse-to-destination data movement patterns that reverse ETL formalises.
Layer in API integration drills → for the rate-limit + retry + idempotency primitives every sync depends on.
Stack the dimensional modelling library → so your reverse ETL models are one-row-per-entity by default.
Sharpen the data transformation library → for the aggregation patterns that turn fact tables into reverse ETL models.
Practise streaming problems → for the event-stream reverse ETL path RudderStack and modern Hightouch / Census tiers ship.
For the broader interview surface, read top data engineering interview questions →.
Stack the prerequisites with the only 5 skills you need to become a data engineer →.
Sharpen the system-design axis with the ETL system design course →.
For long-form data modelling craft, work through data modelling for DE interviews →.

Practice ETL now → API integration drills →

Semantic Layer Showdown: Cube vs dbt Semantic Layer vs Looker LookML

Gowtham Potureddi — Tue, 16 Jun 2026 12:36:58 +0000

A semantic layer is the part of the modern data stack that decides what "active user" means — once — so every dashboard, notebook, embedded chart, and LLM agent that asks the question receives the same answer. Skip it and every BI tool ships its own definition; ship it and the warehouse becomes the canonical source of metric truth instead of the source of metric disagreement.

This guide compares the three engines analytics engineers actually shortlist in 2026: cube.dev as the standalone open-source headless-BI engine, the dbt semantic layer powered by MetricFlow, and lookml as the original semantic model inside Looker. We walk through where each one sits between the warehouse and the consumer surface, how the data models map onto each other, how to define the same Weekly Active Users metric three ways, and how the metrics layer routes queries from Tableau, Power BI, Hex, Mode, embedded apps, and LLM agents.

When you want hands-on reps the moment you finish reading, drill the aggregation practice library → for the measure / metric foundations, rehearse on joins problems → for the entity-resolution patterns that semantic layers automate, and stack the group-by drills → for the granularity reasoning every measure definition leans on.

On this page

What a semantic layer actually solves
The role of the semantic layer in a modern stack
The three platforms compared
Defining a metric in each platform
Consumer fan-out — who queries the semantic layer
Cheat sheet — semantic layer recipes
Frequently asked questions
Practice on PipeCode

1. What a semantic layer actually solves

The "every dashboard redefines active user" problem — and why a governed metric layer is the antidote

The mental model in one line: a semantic layer is a single place where business metrics, dimensions, and joins are defined as objects — the warehouse stays the storage, the BI tool stays the surface, and the metric definition lives in between as code that every consumer reads from. Without it, the same metric is re-invented in every dashboard, every SQL snippet, and every notebook — and the numbers diverge.

The four symptoms of a missing semantic layer.

Five definitions of "active user." Marketing counts a session-open; product counts a feature-event; finance counts a paid event; data science counts a 30-day retention bucket; the CEO sees a different number on every dashboard.
Joins re-written by every analyst. The orders to customers to regions join chain lives in a Tableau workbook, a Looker explore, a Hex notebook, and a one-off SQL snippet — four copies, four chances to drift.
Filters re-implemented per surface. "Exclude internal users" lives as a WHERE email NOT LIKE '%@acme.internal' in some places and a WHERE is_internal = FALSE in others. Numbers drift the day a new internal domain appears.
No diffable governance. When the metric definition lives in a BI workbook's hidden calculated field, you cannot review it in a pull request — and you cannot tell which definition matches the "official" one.

One metric definition, many consumers — the headless-BI premise.

A semantic layer publishes metric definitions as code. Cube ships them as YAML / JS cubes; dbt SL ships them as YAML semantic_models + metrics; LookML ships them as view / explore files.
The same definition resolves into a query when a consumer asks for it. The consumer never writes the join, the GROUP BY granularity, or the filter — the layer does.
Every consumer — Tableau, Power BI, Hex, Mode, embedded apps, and now LLM agents — sees the same number because they all read from the same definition.
This is the headless BI premise: a layer that is a query API but is not a visualisation tool. The viz layer becomes interchangeable.

Where a semantic layer sits.

Below it: the warehouse (Snowflake, BigQuery, Databricks, Redshift, Postgres) holds the marts produced by dbt or any ELT tool.
The semantic layer itself: Cube / dbt SL / LookML translate metric requests into warehouse SQL and (often) cache the results.
Above it: the BI / notebook / embedded / LLM surfaces fan out, each one calling a SQL, REST, or GraphQL endpoint that the layer exposes.

Dimensions, measures, metrics, and joins as first-class objects.

Dimensions are the columns you group by — region, signup_month, device_type.
Measures are the aggregations on a single table — count(distinct user_id), sum(amount). Measures are the LEGO bricks.
Metrics are the named, business-level expressions built from measures — WAU = count_distinct_users filtered to the last 7 days. Metrics are what the dashboard asks for.
Joins are declared once and re-used by every metric. The consumer never has to know that orders.region_id joins to regions.id.

Why this didn't take off until dbt + Cube made it cheap.

The pre-2020 attempts (Looker, Power BI Datasets, Microstrategy schemas) were bundled into a single BI tool — adopting them locked you into that vendor's viz layer.
Cube.dev (2019) decoupled the layer from the viz tool by exposing REST / GraphQL / SQL APIs — every BI tool, notebook, and embedded app could now consume the same definitions.
The dbt Semantic Layer (powered by MetricFlow, 2023) put metric definitions next to dbt models — the same git repo, the same review process, the same CI.
LLM agents (2024–2026) finally made the governance argument concrete: a model that hallucinates a join or a filter ships a confident wrong answer; one that calls the semantic layer ships a verifiable one.

Worked example — the "five definitions of active user" failure mode

Detailed explanation. A consumer-app company has five dashboards labelled "active users." Each was built independently by a different team using a different SQL definition. The CEO opens all five in the Monday meeting and sees five different numbers. This is the canonical symptom that pushes a team to adopt a semantic layer.

Question. Given a shared events table, write the five drifted definitions of "weekly active users" and show how a single semantic-layer definition collapses them into one number. Trace why each ad-hoc version returns a different count.

Input — events.

event_id	user_id	event_name	event_ts	platform
1	100	login	2026-06-08 10:00	web
2	100	search	2026-06-08 10:01	web
3	200	login	2026-06-09 12:00	mobile
4	300	login	2026-06-09 13:00	web
5	300	purchase	2026-06-09 13:05	web
6	400	login	2026-05-30 09:00	web
7	100	login	2026-06-09 09:00	mobile

Code.

-- Definition 1: marketing — anyone with any event in last 7 days
SELECT COUNT(DISTINCT user_id) AS wau_marketing
FROM events
WHERE event_ts >= CURRENT_DATE - INTERVAL '7 days';

-- Definition 2: product — logged in at least once
SELECT COUNT(DISTINCT user_id) AS wau_product
FROM events
WHERE event_name = 'login'
  AND event_ts >= CURRENT_DATE - INTERVAL '7 days';

-- Definition 3: finance — purchased in last 7 days
SELECT COUNT(DISTINCT user_id) AS wau_finance
FROM events
WHERE event_name = 'purchase'
  AND event_ts >= CURRENT_DATE - INTERVAL '7 days';

-- Definition 4: data science — at least 2 distinct event days
SELECT COUNT(DISTINCT user_id) AS wau_ds
FROM (
    SELECT user_id, DATE(event_ts) AS d
    FROM events
    WHERE event_ts >= CURRENT_DATE - INTERVAL '7 days'
    GROUP BY user_id, DATE(event_ts)
) t
GROUP BY user_id
HAVING COUNT(*) >= 2;

-- Definition 5: mobile-only WAU
SELECT COUNT(DISTINCT user_id) AS wau_mobile
FROM events
WHERE event_ts >= CURRENT_DATE - INTERVAL '7 days'
  AND platform = 'mobile';

Step-by-step explanation.

Each query asks "weekly active users" but folds in a different qualifying event. None is wrong — they answer different business questions — but they are all labelled identically on five dashboards.
Marketing counts every distinct user with any event = users 100, 200, 300 → 3.
Product counts users with a login = users 100, 200, 300 → 3 (coincidentally the same here, drifts on other weeks).
Finance counts users with a purchase = user 300 → 1.
Data-science counts users active on two or more days = user 100 (active 2026-06-08 and 2026-06-09) → 1.
Mobile-only WAU restricts to platform = 'mobile' → users 100 and 200 → 2.
A semantic layer collapses this by publishing one named metric per business question: weekly_active_users (default — any event), weekly_logged_in_users, weekly_purchasers, weekly_engaged_users (2+ days), weekly_active_mobile. Each dashboard asks for the named metric, not for an ad-hoc SQL string.

Output.

Metric label	Count
wau_marketing (any event)	3
wau_product (login)	3
wau_finance (purchase)	1
wau_ds (2+ days active)	1
wau_mobile (mobile-only)	2

Rule of thumb. If five dashboards labelled the same metric show five different numbers, the fix is not "agree on the definition once." It is "move the definition into a semantic layer so the next dashboard reads from the same file." Without the layer the next disagreement is one PR away.

Worked example — joins re-written by every analyst

Detailed explanation. A multi-table join chain (orders → customers → regions → tax tables) lives as boilerplate at the top of every Looker workbook, every Hex notebook, and every ad-hoc SQL snippet. The chain is re-typed every time. A semantic layer declares the joins once and lets every metric reference them by entity name.

Question. Given orders, customers, and regions, show the boilerplate every analyst types, then show how a semantic-layer entity declaration removes the join from the query surface entirely.

Input — schema.

table	columns
orders	order_id, customer_id, amount, order_date
customers	customer_id, name, region_id
regions	region_id, region_name, country_code

Code.

-- Every analyst types this join chain — by hand — once per query
SELECT r.region_name,
       SUM(o.amount) AS total_revenue
FROM orders o
INNER JOIN customers c ON o.customer_id = c.customer_id
INNER JOIN regions   r ON c.region_id   = r.region_id
WHERE o.order_date >= CURRENT_DATE - INTERVAL '30 days'
GROUP BY r.region_name;

# Semantic-layer style — declare the joins once, query the metric
# Cube.dev model (simplified)
cubes:
  - name: Orders
    sql_table: orders
    joins:
      - name: Customers
        relationship: many_to_one
        sql: "{Orders}.customer_id = {Customers}.customer_id"
    measures:
      total_revenue:
        sql: amount
        type: sum
    dimensions:
      order_date:
        sql: order_date
        type: time
  - name: Customers
    sql_table: customers
    joins:
      - name: Regions
        relationship: many_to_one
        sql: "{Customers}.region_id = {Regions}.region_id"
  - name: Regions
    sql_table: regions
    dimensions:
      region_name:
        sql: region_name
        type: string

Step-by-step explanation.

The ad-hoc SQL re-types INNER JOIN customers ... INNER JOIN regions ... every time. Five analysts, five copies. The day a new tax-region table is added, all five copies have to be edited.
The Cube schema declares the same joins as relationship: many_to_one lines, one time, per cube. The semantic layer now knows how to traverse from Orders to Regions whenever a query mentions a column from both cubes.
The consumer query becomes: "give me Orders.total_revenue grouped by Regions.region_name, filtered to the last 30 days." Cube generates the join chain on the fly — and it is the same chain every time.
The dbt SL equivalent uses entities and relationships on each semantic_model; LookML uses joins declared inside each explore. The shape is the same: declare once, traverse forever.

Output.

region_name	total_revenue (semantic)	total_revenue (ad-hoc, copy 1)	total_revenue (ad-hoc, copy 2)
EU	12,400	12,400	12,400
US	9,800	9,800	9,800
APAC	6,200	6,200	6,200

The numbers match — but only because every ad-hoc copy happens to be in sync today. The semantic-layer version is in sync by construction.

Rule of thumb. Any join chain that appears in three or more queries should live as a declared entity / join in the semantic layer. Re-typing the same INNER JOIN ... ON ... block is the analytics-engineering equivalent of cargo-cult code copy-paste.

Worked example — the headless-BI contract

Detailed explanation. "Headless BI" is the marketing term; the engineering contract behind it is "expose a query API that any front end can consume." A semantic layer that ships REST, GraphQL, and SQL endpoints can serve a Tableau dashboard, a React embedded chart, and a Slack-bot LLM agent from the same metric file. The metric author writes once; the surfaces fan out.

Question. Show how the same Weekly Active Users metric is requested over (1) a SQL endpoint, (2) a REST endpoint, and (3) a GraphQL endpoint — and explain why the output is byte-for-byte the same number on every surface.

Input. A semantic layer that has published a metric named weekly_active_users over the Events cube.

Code.

-- 1. SQL endpoint (Cube SQL API, dbt SL JDBC, Looker SQL Runner)
SELECT MEASURE(weekly_active_users)
FROM events
WHERE event_date >= DATE_TRUNC('week', CURRENT_DATE);

# 2. REST endpoint (Cube REST, Looker API, dbt Cloud Semantic Layer API)
curl -s -H "Authorization: $TOKEN" \
  "https://semantic.example.com/cubejs-api/v1/load" \
  --data '{
    "query": {
      "measures": ["Events.weekly_active_users"],
      "timeDimensions": [{
        "dimension": "Events.event_date",
        "granularity": "week",
        "dateRange": "this week"
      }]
    }
  }'

# 3. GraphQL endpoint (Cube GraphQL API)
query {
  cube(where: { eventDate: { inDateRange: "this week" } }) {
    events {
      weeklyActiveUsers
    }
  }
}

Step-by-step explanation.

Each endpoint translates the request into the same underlying warehouse SQL. The SQL endpoint is the most direct; REST and GraphQL wrap the request in JSON.
The metric definition for weekly_active_users lives in one file — events.yml (Cube) or events.sql + metrics.yml (dbt SL) or events.view.lkml + events.model.lkml (LookML). Every endpoint reads from that file.
The semantic layer caches the result of the compiled SQL. If three surfaces ask the same question within the cache window, the warehouse runs the query once and the layer serves the cached answer thrice.
Authorization is enforced at the layer — the user's JWT or session is the same across endpoints, and row-level-security rules in the layer rewrite the SQL before it hits the warehouse.

Output (one number, three surfaces).

Surface	Endpoint	Response field	Value
Tableau dashboard	SQL	`MEASURE(weekly_active_users)`	18,432
React embedded chart	REST	`data[0].Events.weekly_active_users`	18,432
Slack LLM agent	GraphQL	`cube[0].events.weeklyActiveUsers`	18,432

Rule of thumb. If your team needs the same metric on more than one surface, the cheapest path to consistency is the semantic layer — not a metrics_macros.sql file shared across BI tools, not a "single source of truth" doc, not a Slack thread. Code, one file, three endpoints.

Semantic layer interview question on metric governance

A senior analytics-engineering interviewer often opens with: "Walk me through how you'd give a CEO confidence that the 'active users' number on the executive dashboard is the same one the ML team trains on, the embedded customer-portal chart renders, and the Slack LLM agent quotes back when asked."

Solution Using a layered semantic-layer governance pattern

# 1. A single metric definition (Cube / dbt SL / LookML pseudocode)
metric: weekly_active_users
description: Distinct users with any event in the trailing 7 days.
type: count_distinct
target_field: user_id
filter: event_ts >= dateadd('day', -7, current_date)
owner: data-platform-team
review: PR-required
sla: <30s p95
cache_window: 1h

# 2. The metric is exposed on three endpoints (SQL / REST / GraphQL).
# 3. Three consumers register subscriptions to the metric:
consumers:
  - executive_dashboard (Looker / Tableau, SQL endpoint)
  - embedded_customer_portal (REST endpoint, RLS on tenant_id)
  - slack_llm_agent (GraphQL endpoint, RLS on slack_user.email)

Step-by-step trace.

Step	Action	Outcome
1	Analytics engineer opens PR editing `weekly_active_users.yml`	CI runs metric tests + freshness check
2	Reviewer approves; merge to main	Semantic layer redeploys metric definition
3	Executive dashboard auto-reloads (cache invalidated)	New number visible in <60s
4	Embedded chart polls REST endpoint	Same new number, same cache key
5	LLM agent grounds prompt with semantic-layer schema	Returns the new definition + new number, citing the layer
6	Auditor diffs `git log` for `weekly_active_users.yml`	One source of truth, one commit history

Output:

Surface	Definition version	Value	Cache state
Executive dashboard	v17 (post-merge)	18,432	hot
Embedded portal	v17	18,432	hot
LLM Slack agent	v17	18,432	hot
Auditor PR-log	v1 → v17	full diff visible	n/a

Why this works — concept by concept:

Single source of truth — the metric file in version control is the only place the definition exists. No duplicate calculated fields, no hidden Tableau formulas, no LLM hallucinations of joins.
PR-required review — metric changes flow through the same code review as any other code change. The "five-dashboards-disagree" failure mode cannot recur because there is no other place to edit the definition.
Cache invalidation on deploy — the semantic layer flushes its cache for the affected metric the moment the new definition lands. Surfaces converge to the new number within the cache-refresh window.
RLS at the layer — row-level-security predicates live in the metric definition. The embedded portal automatically scopes to the tenant; the Slack agent automatically scopes to the asking user. The consumer code carries no security logic.
LLM grounding — the agent calls the semantic layer instead of generating SQL from scratch. Hallucinated joins become impossible because the layer publishes the schema (cubes, measures, dimensions, joins) as the agent's tool surface.
Cost — one warehouse query per cache window (often 1 hour), regardless of how many surfaces poll. The semantic layer is a strict cost reducer relative to N independent BI tools each running their own SQL.

SQL
Topic — aggregation
Measure and metric aggregation problems (SQL)

Practice →

2. The role of the semantic layer in a modern stack

Below the warehouse, beside the BI tool, above the consumer — and right where LLM agents finally need it

The mental model in one line: the semantic layer sits between the warehouse marts and every consumer (BI tool, notebook, embedded app, or LLM agent), translating a metric request into governed warehouse SQL with caching, row-level security, and access control along the way. Get the placement right and every downstream surface becomes interchangeable.

The three tiers in one paragraph.

Bottom — Warehouse + dbt marts. Snowflake, BigQuery, Databricks, Redshift, or Postgres holding the cleaned, tested, joined tables. dbt models do the row-level transforms — staging → marts. The semantic layer reads from the marts; it does not own them.
Middle — Semantic layer. Cube / dbt SL / LookML hold the metric definitions, dimension hierarchies, joins, and security rules. The layer compiles requests to SQL and (often) caches results.
Top — Consumers. Tableau, Power BI, Hex, Mode, Sigma, Streamlit, custom React dashboards, embedded analytics, and now LLM agents. Each consumer speaks SQL, REST, or GraphQL — and gets the same metric value.

Caching, query optimisation, and query rewriting.

Pre-aggregations (Cube). Cube can materialise a roll-up table — e.g. "daily active users by region by platform" — and route the incoming request to the pre-agg when the granularity matches. Sub-second queries against trillion-row fact tables.
dbt SL caching. The dbt Semantic Layer (dbt Cloud) ships a query cache keyed by metric + filters + granularity. Repeat requests within the TTL hit the cache, not the warehouse.
Looker PDTs (persistent derived tables). LookML's derived_table blocks can be persisted on a schedule, turning expensive transforms into a pre-computed warehouse table. The explore reads from the PDT, not from the live mart.
Query rewriting. Modern semantic layers detect when a roll-up table can answer the query and rewrite the SQL transparently — the consumer never knows the query plan changed.

Governed metrics vs ad-hoc SQL — the contract boundary.

Inside the layer: governed, named metrics with descriptions, owners, SLAs, and PR-review history.
Outside the layer: ad-hoc SQL still works (and is sometimes the right answer for exploratory analysis), but it is not the dashboard's source.
The boundary is enforced operationally: BI tools and embedded apps point only at the semantic layer's endpoint. Ad-hoc SQL is a separate Snowflake / BigQuery role that does not feed any production dashboard.

Multi-tenant security and row-level access.

Row-level security (RLS). A B2B SaaS company has 5,000 tenants. The semantic layer rewrites every query to add WHERE tenant_id = :current_tenant. Consumers cannot bypass it because they never write the SQL.
Column masking. Salary or PII columns can be selectively masked at the layer based on the caller's role.
Tenant isolation across consumers. The same metric file works for the internal dashboard and the customer-facing embedded chart, because the security predicate is bound to the request context, not the query.
Auth pass-through. The layer accepts a JWT (or OAuth token) and resolves the user's permissions to row predicates at query time. No "service account that sees everything" pattern.

Why LLM agents finally make semantic layers strategic.

A general-purpose LLM that sees a raw warehouse schema guesses joins and filters — and ships confident wrong answers. A semantic layer publishes the governed schema (cubes, measures, dimensions, joins, allowed filters) as the agent's tool surface.
Grounded queries become deterministic — the same prompt yields the same SQL because the agent is constrained to the cubes that exist.
Audit trails become possible — every agent call resolves to a named metric, not to an opaque generated SQL string. Compliance and finance teams can review what the agent asked for.
The semantic layer is now the interface the LLM agent uses, not a competing surface. This is the inversion that pushed every major BI vendor to ship a semantic-layer story in 2024–2026.

Worked example — without vs with a semantic layer, side by side

Detailed explanation. Two stacks ship the same Weekly Active Users number to a Tableau dashboard. The "without" stack has Tableau pointing directly at the Snowflake mart with a workbook-local calculated field. The "with" stack has Tableau pointing at the semantic layer's SQL endpoint and asking for the named metric.

Question. Compare the two architectures end-to-end. Show where the metric definition lives, who can edit it, and what happens when a second consumer (a Hex notebook) is added.

Input — comparison table.

Concern	Without semantic layer	With semantic layer
Where metric SQL lives	Tableau workbook calculated field	`weekly_active_users.yml` in git
Who can edit it	Anyone with Tableau workbook access	PR-required code review
How a second consumer adopts it	Copy-paste the SQL into Hex	Point Hex at the semantic-layer SQL endpoint
What happens when underlying schema changes	Tableau workbook breaks silently	CI catches the break in the next PR
RLS / multi-tenant security	Per-workbook plumbing	Declared once at the layer
LLM agent integration	Hallucinated SQL	Tool calls to named metrics

Code (without).

-- Tableau workbook calculated field — not in version control
COUNT_DISTINCT(
  IIF(
    DATE(event_ts) >= TODAY() - 7
    AND event_name IN ('login','search','purchase'),
    user_id,
    NULL
  )
)

Code (with).

# weekly_active_users.yml — single source of truth, in git
- name: weekly_active_users
  description: Distinct users with any event in the trailing 7 days.
  type: count_distinct
  target_field: user_id
  semantic_model: events
  filters:
    - "{{ Dimension('events__event_ts') }} >= dateadd('day', -7, current_date)"

Step-by-step explanation.

In the "without" world, the metric is locked inside a Tableau workbook. The Hex team copies the SQL because there is no API to call. Now two definitions exist; they drift the day someone changes the filter on one side.
In the "with" world, the metric lives in a YAML file. Tableau queries the semantic layer's SQL endpoint with SELECT MEASURE(weekly_active_users) FROM events. Hex points at the same endpoint. Both surfaces see the same number by construction.
When the schema changes (e.g. event_ts renamed to event_timestamp), the dbt CI / Cube CI breaks on the next PR — the change is caught before any dashboard sees a stale number.
RLS in the "without" world is a per-workbook setting; in the "with" world it is a layer-level rule that applies uniformly.

Output. Two stacks ship the same number today — but the "with" stack ships the same number on Tuesday at 4pm when a new consumer onboards. The "without" stack ships a divergent number the same week.

Rule of thumb. Adopt the semantic layer the moment you have a second consumer of the same metric. One dashboard can live with an ad-hoc definition; two dashboards cannot — they will drift, and the cost of drift is one quarterly business review with conflicting numbers.

Worked example — caching: pre-aggregations, dbt SL cache, and Looker PDTs

Detailed explanation. Each platform's caching story differs. Cube's pre-aggregations roll up to a smaller table; dbt SL caches the result of a metric request; Looker PDTs persist a derived table on a schedule. All three reduce warehouse load — but they hit different layers.

Question. Given a fact table with 10 billion rows and a dashboard that asks "DAU by region by day for the last 90 days," design a caching strategy for each platform. Show the storage / freshness trade-off.

Input.

Asset	Detail
Fact table	`events` — 10B rows, ~3M new/day
Dashboard	DAU by region by day, 90-day window
SLA	<2s p95
Freshness	<1 hour stale acceptable

Code.

# Cube — pre-aggregation
cubes:
  - name: Events
    pre_aggregations:
      dau_by_region_by_day:
        measures:
          - daily_active_users
        dimensions:
          - region
        time_dimension: event_date
        granularity: day
        partition_granularity: month
        refresh_key:
          every: 1 hour

# dbt SL — cache settings (dbt Cloud)
saved_queries:
  - name: dau_by_region_by_day
    query_params:
      metrics: [daily_active_users]
      group_by:
        - Dimension('events__region')
        - TimeDimension('events__event_date', 'day')
      where:
        - "{{ TimeDimension('events__event_date') }} >= dateadd('day', -90, current_date)"
    cache:
      ttl: 3600  # seconds

# Looker — PDT
view: events_dau_pdt {
  derived_table: {
    sql:
      SELECT DATE(event_ts) AS event_date,
             region,
             COUNT(DISTINCT user_id) AS dau
      FROM events
      WHERE event_ts >= CURRENT_DATE - INTERVAL '90 DAY'
      GROUP BY 1, 2 ;;
    sql_trigger_value: SELECT DATE_TRUNC('hour', CURRENT_TIMESTAMP) ;;
  }
}

Step-by-step explanation.

The Cube pre-agg materialises a roll-up table (events__dau_by_region_by_day) partitioned by month and refreshed hourly. The dashboard's SQL hits this table — ~90 rows × number-of-regions, not 10B rows. Sub-second.
The dbt SL cache stores the result of the saved query in dbt Cloud's cache. Repeated requests with the same parameters get the cached row set. Cache misses re-run against the warehouse mart.
Looker's PDT persists a derived table in the warehouse. The sql_trigger_value rebuilds the PDT every hour. Every explore that references this view reads from the PDT.
All three trade some staleness for massive speed-up. The freshness SLA (<1 hour) fits each pattern. The storage cost is similar — one roll-up table per critical metric grain.

Output.

Platform	Query latency	Storage overhead	Freshness	Where the cache lives
Cube pre-agg	<500ms	1 small table per pre-agg	hourly refresh	Warehouse + Cube metadata
dbt SL cache	<2s (warm), seconds (cold)	result set in dbt Cloud	TTL-based	dbt Cloud
Looker PDT	<1s	1 derived table per PDT	hourly trigger	Warehouse

Rule of thumb. Pre-compute when the query pattern is predictable and high-volume; let the cache TTL absorb the long tail. Picking the wrong layer (e.g. caching the result for an exploratory cube where every consumer asks a different grain) wastes the cache hit ratio.

Worked example — multi-tenant row-level security at the semantic layer

Detailed explanation. A B2B SaaS analytics product needs to scope every chart to the calling tenant. Without a semantic layer, every BI tool re-implements the WHERE tenant_id = ? predicate — and the day someone forgets it, tenant A sees tenant B's revenue. With a semantic layer, the predicate lives in the cube definition and is enforced for every consumer.

Question. Show the RLS rule for the same metric in Cube, dbt SL, and LookML. Trace how a single user's JWT routes to the right tenant scope.

Input. A multi-tenant orders table with a tenant_id column. The user alice@tenantA.com has a JWT with tenant_id = "tenant_A".

Code.

# Cube — query rewrite
cubes:
  - name: Orders
    sql_table: orders
    public: true
    # security context injected from JWT claim
    sql: SELECT * FROM orders WHERE tenant_id = '{COMPILE_CONTEXT.securityContext.tenant_id}'
    measures:
      total_revenue:
        sql: amount
        type: sum

# dbt SL — runtime filter via dbt Cloud security context
semantic_models:
  - name: orders
    model: ref('fct_orders')
    entities:
      - name: order_id
        type: primary
    defaults:
      agg_time_dimension: order_date
    # MetricFlow respects access controls declared in dbt_project.yml
metrics:
  - name: total_revenue
    type: simple
    type_params:
      measure: revenue_sum
    filter: "{{ Dimension('orders__tenant_id') }} = '{{ session.tenant_id }}'"

# Looker — access_filter
explore: orders {
  access_filter: {
    field: orders.tenant_id
    user_attribute: tenant_id
  }
}

Step-by-step explanation.

Alice opens her tenant-A dashboard. Her JWT carries tenant_id = "tenant_A".
The semantic layer extracts the claim into its security context.
The cube / semantic_model / explore wraps every emitted SQL with WHERE tenant_id = 'tenant_A'. There is no path to send a query that skips the predicate.
The warehouse executes the scoped query; only tenant-A rows are returned. The metric total_revenue is computed over tenant-A rows only.
When Bob from tenant B opens the same dashboard URL, his JWT carries tenant_id = "tenant_B". The same metric definition resolves to a different SQL — and a different number — without any code change.

Output.

User	tenant_id claim	SQL emitted	total_revenue
Alice (tenant A)	tenant_A	`WHERE tenant_id = 'tenant_A'`	412,300
Bob (tenant B)	tenant_B	`WHERE tenant_id = 'tenant_B'`	158,900
Internal data team (no tenant)	—	denied / explicit override	n/a

Rule of thumb. If your product is multi-tenant, the semantic layer is the only place tenant isolation belongs. Every other location (BI workbook filter, embedded chart query, app-side SDK call) is a cross-tenant data-leak waiting to ship.

Semantic layer interview question on caching and security

A senior interviewer might frame it as: "Your CEO dashboard polls 'DAU by region by day' every minute and runs into Snowflake credit overspend. The same dashboard is also embedded into a customer-facing portal where 200 tenants need their own scoped numbers. Design the semantic-layer caching and RLS strategy."

Solution Using pre-aggregations + tenant-scoped cache keys

cubes:
  - name: Events
    sql: |
      SELECT * FROM events
      WHERE tenant_id = '{COMPILE_CONTEXT.securityContext.tenant_id}'
    measures:
      dau:
        type: count_distinct
        sql: user_id
    dimensions:
      event_date:
        type: time
        sql: event_ts
      region:
        type: string
        sql: region
    pre_aggregations:
      tenant_dau_region_day:
        measures: [dau]
        dimensions: [region]
        time_dimension: event_date
        granularity: day
        partition_granularity: month
        refresh_key:
          every: 5 minute
        # tenant_id is added to the cache key automatically because
        # it appears in the cube SQL via COMPILE_CONTEXT

Step-by-step trace.

Step	Caller	Cache key contains tenant_id?	Outcome
1	Internal CEO dashboard polls	yes (internal tenant)	hits warm pre-agg, 200ms
2	Tenant A's portal polls	yes (tenant_A)	hits warm pre-agg for tenant A, 250ms
3	Tenant B's portal polls	yes (tenant_B)	hits warm pre-agg for tenant B, 250ms
4	Refresh tick (every 5 min)	per-tenant rebuild	one Snowflake query per active tenant
5	Cold tenant (no traffic for 1 day)	pre-agg expires	next request rebuilds — 2s once
6	Audit query for credit usage	total Snowflake spend = N tenants × 12 queries/hour	~95% reduction vs naive

Output:

Metric	Before	After
Snowflake credits / day	240	14
p95 latency (CEO dashboard)	6.2s	0.21s
p95 latency (tenant portal)	4.8s	0.25s
Cross-tenant leak risk	high (per-app plumbing)	zero (layer-enforced)

Why this works — concept by concept:

Pre-aggregations as the cost killer — Snowflake credits scale with scanned bytes, not with queries served. Routing every poll to a roll-up table of ~10K rows instead of 10B drops the bytes scanned by 6 orders of magnitude.
Per-tenant cache key — the tenant_id appears in the cube SQL via COMPILE_CONTEXT, so Cube partitions the pre-agg storage by tenant. Tenant A's data is physically in a different row set from tenant B's.
Refresh granularity matches SLA — a 5-minute refresh is "fresh enough" for product analytics. Tightening to 1 minute would re-spend Snowflake credits; loosening to 1 hour would break the freshness contract.
Cold-tenant elasticity — pre-aggs for inactive tenants expire and only get rebuilt on demand. Pay for what you query.
Single security predicate — the WHERE tenant_id = ... line lives once. Every consumer (CEO dashboard, embedded portal, LLM agent) inherits it without writing tenant logic.
Cost — O(active_tenants × refresh_ticks) warehouse queries per day, each scanning O(1 day of events). Independent of consumer poll rate.

SQL
Topic — group-by
GROUP BY and granularity problems (SQL)

Practice →

3. The three platforms compared

Cube vs dbt Semantic Layer vs LookML — the scoring rubric every analytics-engineering lead should keep at hand

The mental model in one line: Cube is the standalone OSS engine with the widest BI fan-out, the dbt Semantic Layer is the dbt-native option that piggybacks on the model layer you already own, and LookML is the original — tightly coupled to Looker as the consumer. The right pick is mostly a function of which consumers your team must serve, not the model file's syntax.

Cube.dev (formerly Cube.js) — the standalone OSS engine.

What it is. An open-source semantic engine, written in Node.js + Rust, that publishes metric definitions over REST, GraphQL, and SQL APIs. Started as Cube.js in 2019, rebranded to Cube.dev.
Strengths. Widest BI fan-out (any tool that speaks SQL or HTTP can consume it). Pre-aggregations are best-in-class. Self-hostable via Docker; managed via Cube Cloud. Excellent for embedded analytics and LLM agents because of the REST / GraphQL surfaces.
Tradeoffs. Maintains its own model files (cubes) — duplication if you already model in dbt. The OSS edition lacks some governance features that ship in Cube Cloud (lineage, RBAC UI, query history).
Data model. cube → measures, dimensions, joins, segments, pre_aggregations. A cube maps roughly to a fact / dim table.

dbt Semantic Layer (powered by MetricFlow) — the dbt-native option.

What it is. Metric definitions that live next to dbt models as semantic_model and metric YAML, compiled by MetricFlow into SQL. Available in dbt Cloud (managed) and dbt Core (CLI / open-source).
Strengths. Lives in your existing dbt repo — same PR review, same CI, same lineage. Best-in-class time-spine and cumulative metrics. Direct integration with Tableau, Hex, Mode, Power BI, Sigma, and Lightdash via the dbt SL JDBC connector.
Tradeoffs. The premium hosted Semantic Layer is gated to dbt Cloud Team / Enterprise plans (dbt Core has MetricFlow but not the cached server). Smaller embedded / API surface than Cube. AI / agent fan-out is improving but lags Cube's GraphQL story.
Data model. semantic_model → entities, dimensions, measures; metric → simple / ratio / derived / cumulative. Entities are the primary / foreign key declarations that drive joins.

Looker LookML — the original.

What it is. The semantic modelling language that ships inside Looker. Mature since 2014; the reference implementation of "metric definitions live next to the BI tool."
Strengths. Mature governance (Git-integrated workspace, content validation, IDE). Persistent derived tables (PDTs) are battle-tested. Deep integration with Looker's explore experience.
Tradeoffs. Tightly coupled to Looker as the consumer. Other BI tools cannot natively consume LookML — you would expose Looker's SQL Runner or pipe via API. The license cost scales with Looker user seats, which can dominate the BI budget.
Data model. view → dimensions, measures, filters; explore → joins; model → packages explores together.

Scoring rubric — five axes that decide the pick.

Axis	Cube.dev	dbt Semantic Layer	LookML
Openness (consumer fan-out)	★★★★★	★★★★	★★
Time-to-value (if no existing model)	★★★	★★★★ (if dbt already in place)	★★
Cost (TCO)	$ — OSS, Cube Cloud paid	$$ — dbt Cloud Team/Enterprise	$$$ — per Looker seat
Embedded / LLM	★★★★★	★★★	★
Governance & lineage	★★★★ (Cube Cloud)	★★★★★ (lives in dbt repo)	★★★★★

Common interview probes on platform choice.

"When would you pick Cube over dbt SL?" — when the consumer mix is heavily embedded analytics, LLM agents, or non-dbt BI tools, and when pre-aggregations are the dominant cost saver.
"When would you pick dbt SL over Cube?" — when the team already lives in a dbt repo and the consumers are mostly Tableau / Power BI / Hex / Mode through the JDBC connector.
"When would you stay on LookML?" — when Looker is already the standard BI tool, the team values mature governance, and there is no near-term need to serve embedded or AI consumers.
"What is the migration cost LookML → dbt SL?" — typically rewriting view / explore files as semantic_models, plus carefully porting calculated fields and access filters. Usually staged metric by metric, not as a big-bang.

Worked example — score the three platforms for an embedded analytics SaaS

Detailed explanation. A B2B SaaS company sells an analytics product embedded in customer apps. The consumer mix is: 80% embedded React charts (REST/GraphQL), 15% internal Tableau dashboards, 5% an early LLM agent. They already use dbt for upstream models.

Question. Score the three platforms against this consumer mix and pick the primary semantic layer.

Input — weighted axes.

Axis	Weight	Cube	dbt SL	LookML
Embedded REST/GraphQL	40%	5	3	1
Internal BI (Tableau)	15%	4	5	3
LLM agent	5%	5	3	2
dbt integration	20%	3	5	2
Cost / OSS option	10%	5	3	1
Governance	10%	4	5	5

Code.

weights = {
    "embedded":   0.40,
    "internal_bi": 0.15,
    "llm":        0.05,
    "dbt":        0.20,
    "cost":       0.10,
    "governance": 0.10,
}
scores = {
    "Cube":   {"embedded":5, "internal_bi":4, "llm":5, "dbt":3, "cost":5, "governance":4},
    "dbt_SL": {"embedded":3, "internal_bi":5, "llm":3, "dbt":5, "cost":3, "governance":5},
    "LookML": {"embedded":1, "internal_bi":3, "llm":2, "dbt":2, "cost":1, "governance":5},
}
weighted = {p: round(sum(s[a]*weights[a] for a in weights), 2) for p, s in scores.items()}
print(weighted)
# {'Cube': 4.4, 'dbt_SL': 3.85, 'LookML': 2.0}

Step-by-step explanation.

Embedded REST/GraphQL is the dominant axis (40%). Cube's REST + GraphQL APIs score a 5; dbt SL's JDBC + API score a 3; LookML's "no native embedded API" scores a 1.
Internal Tableau is well served by all three but easiest to wire through dbt SL's JDBC.
The LLM-agent axis favours Cube because of the GraphQL schema and the published meta endpoint that lists all cubes / dimensions / measures as a tool surface.
dbt integration favours dbt SL — it lives in the same repo.
The weighted scores collapse to ~4.4 for Cube, 3.85 for dbt SL, 2.0 for LookML — Cube wins this scenario.
The team picks Cube as the primary semantic layer. dbt continues to own the model layer (staging / marts), and Cube reads from the dbt marts.

Output.

Platform	Weighted score	Decision
Cube.dev	4.40	primary
dbt SL	3.85	alternative if embedded shrinks
LookML	2.00	not a fit (no Looker consumer)

Rule of thumb. Score by consumer mix, not by syntax preference. The semantic layer's job is to serve consumers; the file format matters only to the small team of analytics engineers maintaining it.

Worked example — Cube data model in detail

Detailed explanation. A Cube schema is built from cube blocks. Each cube wraps a SQL table (or view), exposes measures and dimensions, declares joins to other cubes, and optionally defines segments (named filters) and pre_aggregations (materialised roll-ups).

Question. Translate a small star schema (orders, customers, regions) into a Cube schema with one revenue measure and one region dimension. Show the joins declared once and re-used.

Input — star schema.

Table	Role	Key columns
orders	fact	order_id, customer_id, amount, order_date
customers	dim	customer_id, region_id, name
regions	dim	region_id, region_name

Code.

cubes:
  - name: Orders
    sql_table: analytics.fct_orders
    joins:
      - name: Customers
        relationship: many_to_one
        sql: "{Orders}.customer_id = {Customers}.customer_id"
    measures:
      total_revenue:
        sql: amount
        type: sum
        format: currency
      order_count:
        type: count
    dimensions:
      order_id:
        sql: order_id
        type: number
        primary_key: true
      order_date:
        sql: order_date
        type: time
    segments:
      paid_orders:
        sql: "{CUBE}.status = 'paid'"

  - name: Customers
    sql_table: analytics.dim_customers
    joins:
      - name: Regions
        relationship: many_to_one
        sql: "{Customers}.region_id = {Regions}.region_id"
    dimensions:
      customer_id:
        sql: customer_id
        type: number
        primary_key: true
      name:
        sql: name
        type: string

  - name: Regions
    sql_table: analytics.dim_regions
    dimensions:
      region_id:
        sql: region_id
        type: number
        primary_key: true
      region_name:
        sql: region_name
        type: string

Step-by-step explanation.

Each cube maps to a warehouse table. Orders is the fact; Customers and Regions are dims.
The joins block on Orders declares the join to Customers once. The joins block on Customers declares the join to Regions once. Cube composes the chain: a query for Orders.total_revenue grouped by Regions.region_name traverses both joins automatically.
measures define aggregations on the cube's table. dimensions define group-by axes; primary_key: true marks the row identity for the cube.
segments are named filters. A dashboard can ask "give me Orders.total_revenue for segment paid_orders" without re-typing the WHERE status = 'paid' predicate.
Adding a new metric is one new measure block. Adding a new dim is one new dimension. Cubes scale linearly with metric count.

Output. A consumer can now ask: "Orders.total_revenue grouped by Regions.region_name filtered to Orders.paid_orders segment, last 30 days." The semantic layer composes the join chain and the segment filter — the consumer writes no SQL.

Rule of thumb. Declare every join once, every segment once, every measure once. The rule "don't repeat the SQL" is what makes the layer pay for itself within the first six metrics.

Worked example — dbt Semantic Layer data model in detail

Detailed explanation. A dbt Semantic Layer schema is built from semantic_models (which define entities, dimensions, and measures on a dbt model) and metrics (which express the business-level KPIs computed from those measures). MetricFlow turns metric requests into SQL.

Question. Re-express the same orders / customers / regions star schema as dbt Semantic Layer YAML. Show the entity-based joins and the simple metric for total revenue.

Input. dbt models fct_orders, dim_customers, dim_regions exist and have unique_key columns.

Code.

# models/semantic/orders.yml
semantic_models:
  - name: orders
    model: ref('fct_orders')
    defaults:
      agg_time_dimension: order_date
    entities:
      - name: order_id
        type: primary
      - name: customer_id
        type: foreign
    dimensions:
      - name: order_date
        type: time
        type_params:
          time_granularity: day
    measures:
      - name: revenue_sum
        agg: sum
        expr: amount

  - name: customers
    model: ref('dim_customers')
    entities:
      - name: customer_id
        type: primary
      - name: region_id
        type: foreign
    dimensions:
      - name: name
        type: categorical

  - name: regions
    model: ref('dim_regions')
    entities:
      - name: region_id
        type: primary
    dimensions:
      - name: region_name
        type: categorical

metrics:
  - name: total_revenue
    label: Total Revenue
    type: simple
    type_params:
      measure: revenue_sum

Step-by-step explanation.

Each semantic_model wraps an existing dbt model. entities declare the primary / foreign keys MetricFlow uses to auto-resolve joins.
customer_id is primary in customers and foreign in orders — MetricFlow knows that orders joins to customers on customer_id without an explicit JOIN ... ON block.
Similarly, region_id is primary in regions and foreign in customers — the chain orders → customers → regions is implicit.
dimensions declare group-by axes; agg_time_dimension defaults the time grain for time-series queries.
measures are the aggregation building blocks. A metric of type simple wraps a single measure into a named, dashboard-facing KPI.
Other metric types compose more complex KPIs: ratio (numerator / denominator), derived (an arithmetic expression over other metrics), cumulative (rolling totals with date-spine support).

Output. A consumer can ask: "total_revenue grouped by region__region_name for the last 30 days." MetricFlow resolves the join chain via entities and emits SQL — no JOIN ... ON typed by the consumer.

Rule of thumb. dbt SL's entity model is most powerful when your dbt marts already follow Kimball-style conventions (one primary key per table, foreign keys named consistently). Greenfield dbt + dbt SL projects converge faster than retrofits.

Worked example — LookML data model in detail

Detailed explanation. A LookML schema is built from view files (mapping to tables) and explore files (declaring joins between views). Each view exposes dimensions and measures; each explore lists the join blocks. The Looker UI shells out queries against the explore.

Question. Re-express the same star schema in LookML — a view per table, an explore that joins them, a measure for total revenue.

Input. Looker connection to the same warehouse with fct_orders, dim_customers, dim_regions.

Code.

# views/orders.view.lkml
view: orders {
  sql_table_name: analytics.fct_orders ;;

  dimension: order_id {
    primary_key: yes
    type: number
    sql: ${TABLE}.order_id ;;
  }
  dimension_group: order {
    type: time
    timeframes: [date, week, month, quarter, year]
    sql: ${TABLE}.order_date ;;
  }
  dimension: customer_id {
    type: number
    sql: ${TABLE}.customer_id ;;
  }
  measure: total_revenue {
    type: sum
    sql: ${TABLE}.amount ;;
    value_format_name: usd
  }
  measure: order_count {
    type: count
  }
}

# views/customers.view.lkml
view: customers {
  sql_table_name: analytics.dim_customers ;;
  dimension: customer_id {
    primary_key: yes
    type: number
    sql: ${TABLE}.customer_id ;;
  }
  dimension: region_id {
    type: number
    sql: ${TABLE}.region_id ;;
  }
  dimension: name { type: string sql: ${TABLE}.name ;; }
}

# views/regions.view.lkml
view: regions {
  sql_table_name: analytics.dim_regions ;;
  dimension: region_id {
    primary_key: yes
    type: number
    sql: ${TABLE}.region_id ;;
  }
  dimension: region_name { type: string sql: ${TABLE}.region_name ;; }
}

# models/sales.model.lkml
explore: orders {
  join: customers {
    type: left_outer
    relationship: many_to_one
    sql_on: ${orders.customer_id} = ${customers.customer_id} ;;
  }
  join: regions {
    type: left_outer
    relationship: many_to_one
    sql_on: ${customers.region_id} = ${regions.region_id} ;;
  }
}

Step-by-step explanation.

Each view wraps a warehouse table. dimension and measure blocks declare the columns and aggregations.
The explore: orders block declares the join chain. join: customers joins orders → customers; join: regions joins customers → regions. The relationship hint (many_to_one) lets Looker pick the right SQL form.
The dimension_group shortcut auto-generates order_date, order_week, order_month, etc. — every common time-grain dimension for free.
value_format_name: usd formats the measure as currency in the BI surface.
A Looker user opens the orders explore, selects regions.region_name and orders.total_revenue, and Looker emits the join chain transparently.

Output. Same number, same governance, same star schema — expressed in LookML files inside a Git-integrated Looker workspace.

Rule of thumb. LookML's per-view structure is more verbose than YAML alternatives but reads beautifully in code review. If Looker is the only BI surface, the verbosity is offset by IDE features (autocomplete, content validation, LookML test runner). When the BI surface fans out beyond Looker, the verbosity becomes a tax.

Semantic layer interview question on platform selection

A senior analytics-engineering interviewer might ask: "You inherit a company on Looker with 200 dashboards. The CTO wants to add an embedded analytics product and an LLM agent within 12 months. Walk me through the semantic layer migration plan."

Solution Using a staged "10 top metrics" migration to Cube alongside Looker

# Stage 1: stand up Cube alongside Looker (Cube reads from the same warehouse marts).
# Stage 2: identify the 10 top metrics by Looker query volume.
# Stage 3: rewrite those 10 metrics in Cube (or dbt SL), reference dbt marts.
# Stage 4: point the new embedded product and LLM agent at Cube.
# Stage 5: dual-publish — Looker continues to serve the 200 dashboards;
#          Cube serves the new surfaces.
# Stage 6: as Looker dashboards retire or get rebuilt, port them to Cube one by one.

migration_plan:
  parallel_run_months: 12
  metric_priority:
    - weekly_active_users
    - daily_active_users
    - total_revenue
    - new_signups
    - churn_rate
    - retention_d7
    - retention_d30
    - average_order_value
    - conversion_rate
    - net_promoter_score
  consumers:
    - embedded_react_charts -> Cube REST
    - llm_slack_agent       -> Cube GraphQL
    - existing_looker       -> LookML (untouched)
    - new_internal_bi       -> Cube SQL API

Step-by-step trace.

Stage	Action	Risk	Mitigation
1	Cube deployed alongside Looker	low	both read same warehouse, no migration
2	Top-10 metrics catalogued from Looker query history	low	weighted by query volume + dashboard count
3	Top-10 rewritten in Cube YAML	medium	unit-test each metric against Looker output for 30 days
4	Embedded product + LLM agent ship on Cube	medium	new code path, no impact on Looker
5	Dual-publish, no Looker dashboards touched	low	full backwards compatibility
6	Looker dashboards ported as part of normal roadmap	spread over months	each port is one PR, reviewable

Output:

Surface	Pre-migration	Post-migration
Looker dashboards	200	200 (unchanged)
Embedded React charts	0	new on Cube
LLM Slack agent	0	new on Cube
New internal BI	n/a	on Cube SQL API
Source of truth metrics	LookML (200 metric defs)	Cube (10 top metrics) + LookML (long tail)

Why this works — concept by concept:

Parallel run, not big-bang — Looker and Cube co-exist for 12 months. No "stop-the-world" cutover. Risk distributed across the migration window.
Top-10 metric priority — analytics-engineering effort focuses on the metrics that power the new consumer surfaces. The long tail of 190 dashboards stays on Looker until a natural rebuild.
Embedded + LLM on Cube — the new surfaces are wired only to the new layer. Their existence does not depend on Looker uptime, license seats, or LookML rewrites.
Same warehouse marts — both layers read from the dbt-managed marts. The underlying data is one copy; the metric definitions are layered above.
Unit-test against Looker — for each ported metric, dual-run for 30 days and compare numbers daily. Drift > 0.5% blocks the migration of that metric.
Cost — incremental dual-platform cost during the 12-month run, offset by the new revenue streams (embedded product, LLM agent) that depend on Cube.

SQL
Topic — joins
JOIN problems for entity resolution (SQL)

Practice →

4. Defining a metric in each platform

One Weekly Active Users metric, three vocabularies — the side-by-side comparison every interviewer probes

The mental model in one line: the same Weekly Active Users metric is count_distinct user_id over the trailing 7 days, but each platform asks you to spell it differently — Cube's measure type: countDistinct, dbt SL's simple metric over a count_distinct measure, and LookML's measure type: count_distinct. Translating between the three is a syntax exercise once the semantics are clear.

The same metric, three vocabularies.

Cube. cube exposes a count_distinct measure on user_id. The "weekly" granularity is supplied by the consumer via the timeDimensions block, or pre-baked as a segment.
dbt SL. A semantic_model exposes a measure count_distinct(user_id). A metric of type simple wraps the measure and is queried with a granularity: week time dimension.
LookML. A view exposes a measure type: count_distinct. The dimension_group auto-generates a *_week dimension that the explore groups by.

Where joins are declared.

Cube. Per-cube joins block: relationship: many_to_one + sql: "{Cube}.k = {Other}.k".
dbt SL. Per-semantic_model entities: declare primary and foreign entities; MetricFlow infers the join.
LookML. Per-explore join: type: left_outer + sql_on: ${a.k} = ${b.k}.

Granularity, time dimensions, and date-spine handling.

Cube. Time dimensions support granularity: day | week | month | quarter | year. Pre-aggregations can be partitioned by month or week for cost control. Date-spine is handled implicitly by the consumer query.
dbt SL. Best-in-class time spine — declare a time_spine model once in semantic_models.yml, and MetricFlow uses it to fill in zero rows for missing dates in cumulative metrics. Granularities: second | minute | hour | day | week | month | quarter | year.
LookML. dimension_group auto-generates every common timeframe. PDTs can be partitioned by date. No native date-spine — analysts hand-roll a "calendar" view if needed.

Derived and ratio metrics.

Cube. Compose by referencing other measures in a derived measure's sql: e.g. conversion_rate: sql: "{purchases} * 1.0 / NULLIF({sessions}, 0)".
dbt SL. First-class metric types: ratio (numerator + denominator) and derived (expression over named metrics). The cleanest of the three for compound KPIs.
LookML. Compose with measure type: number and an expression referencing other measures: ${purchases} * 1.0 / NULLIF(${sessions}, 0).

Filters, segments, and parameter inputs.

Cube. segments are named filters reusable across queries. Templated parameters via { FILTER_PARAMS } for runtime injection.
dbt SL. filter blocks on metrics for static filters; where clauses on saved queries for runtime filters. Less templating, more declarative.
LookML. filter blocks on views and parameter blocks for runtime input. liquid template language for advanced rewrites.

Caching, materialisation, and roll-ups.

Cube. pre_aggregations are the headline feature — declare the roll-up grain and refresh schedule; Cube auto-routes matching queries.
dbt SL. dbt Cloud Semantic Layer cache; saved queries can be persisted as tables via dbt's materialized: table.
LookML. PDTs persisted on a schedule via sql_trigger_value or datagroup. Most mature of the three but warehouse-coupled.

Versioning a metric definition.

All three live in Git. Cube has meta.version per cube; dbt SL inherits dbt's version and defined_in semantics; LookML has Looker Workspaces (Git-backed branches) with content validation.

Worked example — Weekly Active Users in Cube

Detailed explanation. A Cube schema exposes a weekly_active_users measure as a count_distinct of user_id over a 7-day rolling window. The consumer queries it with a dateRange filter on the time dimension.

Question. Write the Cube definition for weekly_active_users and show the consumer query that returns the WAU per week for the trailing 12 weeks.

Input. An events table with user_id, event_ts, event_name.

Code.

cubes:
  - name: Events
    sql_table: analytics.fct_events
    measures:
      active_users:
        type: countDistinct
        sql: user_id
        title: Active users
    dimensions:
      event_ts:
        sql: event_ts
        type: time
      event_name:
        sql: event_name
        type: string
      user_id:
        sql: user_id
        type: number
    segments:
      logged_in:
        sql: "{CUBE}.event_name = 'login'"
    pre_aggregations:
      weekly_active_rollup:
        measures: [active_users]
        time_dimension: event_ts
        granularity: week
        partition_granularity: month
        refresh_key:
          every: 1 hour

// Consumer query — REST/GraphQL/SQL all hit the same definition
{
  "measures": ["Events.active_users"],
  "timeDimensions": [{
    "dimension": "Events.event_ts",
    "granularity": "week",
    "dateRange": ["2026-03-22", "2026-06-14"]
  }]
}

Step-by-step explanation.

The active_users measure is countDistinct(user_id). Cube treats it as a measure usable at any granularity.
The consumer asks for Events.active_users grouped by Events.event_ts at weekly granularity, over a 12-week date range.
Cube checks pre-aggregations; the weekly_active_rollup is partitioned by month at weekly granularity, so it matches. The query reads the pre-agg, not the raw fact.
The result is one row per week; "WAU" emerges naturally from count_distinct at the week grain. No bespoke "WAU" formula is needed because the measure + granularity combination is WAU.

Output (sample).

week	active_users
2026-03-22	18,210
2026-03-29	18,405
2026-04-05	18,890
...	...
2026-06-14	19,432

Rule of thumb. In Cube, the same count_distinct measure can serve as DAU, WAU, MAU depending on the granularity the consumer asks for. Don't define three measures — define one and let the time dimension do the work.

Worked example — Weekly Active Users in dbt Semantic Layer

Detailed explanation. The dbt Semantic Layer defines the same metric as a simple metric over a count_distinct measure on the events semantic model. MetricFlow injects the granularity at query time.

Question. Write the dbt SL YAML for weekly_active_users and the MetricFlow CLI / Python call to return WAU per week for the trailing 12 weeks.

Input. A dbt model fct_events with user_id, event_ts, event_name. A time_spine model is configured.

Code.

# models/semantic/events.yml
semantic_models:
  - name: events
    model: ref('fct_events')
    defaults:
      agg_time_dimension: event_ts
    entities:
      - name: user_id
        type: foreign
      - name: event_id
        type: primary
    dimensions:
      - name: event_ts
        type: time
        type_params:
          time_granularity: day
      - name: event_name
        type: categorical
    measures:
      - name: distinct_users
        agg: count_distinct
        expr: user_id

metrics:
  - name: weekly_active_users
    label: Weekly Active Users
    type: simple
    type_params:
      measure: distinct_users

# MetricFlow CLI consumer call
mf query \
  --metrics weekly_active_users \
  --group-by metric_time__week \
  --start-time 2026-03-22 \
  --end-time   2026-06-14

Step-by-step explanation.

distinct_users is the measure (the count_distinct LEGO brick). weekly_active_users is the simple metric that wraps it as a dashboard-facing KPI.
The metric_time__week group-by tells MetricFlow to aggregate the measure at the week grain using agg_time_dimension: event_ts.
MetricFlow joins to the time spine (declared once in semantic_models.yml) so weeks with zero activity still appear as zero rows — not as missing rows.
The query returns one row per week. Consumers can be Tableau (via JDBC), Hex (native dbt SL integration), or any tool that speaks the SL API.

Output (sample).

metric_time__week	weekly_active_users
2026-03-22	18,210
2026-03-29	18,405
2026-04-05	18,890
...	...
2026-06-14	19,432

Rule of thumb. When the metric is "count something distinct at a time grain," reach for a simple metric over a count_distinct measure. Reserve ratio, derived, and cumulative for the metrics that genuinely need them.

Worked example — Weekly Active Users in LookML

Detailed explanation. A LookML view exposes a count_distinct measure on user_id. A dimension_group auto-generates event_week. The explore groups by event_week and pivots on active_users.

Question. Write the LookML view + explore for the same WAU metric and the Looker query (or SQL Runner equivalent) for the trailing 12 weeks.

Input. A Looker connection to the same fct_events warehouse table.

Code.

# views/events.view.lkml
view: events {
  sql_table_name: analytics.fct_events ;;

  dimension: event_id {
    primary_key: yes
    type: number
    sql: ${TABLE}.event_id ;;
  }
  dimension: user_id {
    type: number
    sql: ${TABLE}.user_id ;;
    hidden: yes
  }
  dimension_group: event {
    type: time
    timeframes: [date, week, month, quarter, year]
    sql: ${TABLE}.event_ts ;;
  }
  dimension: event_name {
    type: string
    sql: ${TABLE}.event_name ;;
  }
  measure: active_users {
    type: count_distinct
    sql: ${user_id} ;;
    label: "Active users"
  }
}

# models/events.model.lkml
explore: events {
  description: "Activity events fact for WAU/DAU/MAU"
}

-- SQL emitted by Looker for the trailing 12-week WAU query
SELECT DATE_TRUNC('week', ${TABLE}.event_ts) AS event_week,
       COUNT(DISTINCT ${TABLE}.user_id)      AS active_users
FROM   analytics.fct_events AS events
WHERE  ${TABLE}.event_ts >= CURRENT_DATE - INTERVAL '84 day'
GROUP BY 1
ORDER BY 1;

Step-by-step explanation.

The dimension_group: event auto-generates event_date, event_week, event_month, event_quarter, event_year. Selecting event_week in the Looker UI drives the DATE_TRUNC('week', ...) in the emitted SQL.
The measure: active_users is count_distinct ${user_id}. Looker pairs it with the chosen time grain to produce DAU / WAU / MAU.
The explore is intentionally minimal — events is a single-table fact, so no joins are needed for this metric. Joins to users, regions, etc. would be added in the same explore.
The Looker UI generates the emitted SQL automatically. Power users can drop to SQL Runner; the explore is the typical surface.

Output (sample, identical to Cube / dbt SL).

event_week	active_users
2026-03-22	18,210
2026-03-29	18,405
2026-04-05	18,890
...	...
2026-06-14	19,432

Rule of thumb. In LookML, the dimension_group is the productivity unlock. Define one time field, get every common grain for free. The verbosity tax is paid up front; the daily authoring tax is small.

Worked example — derived and ratio metrics across all three

Detailed explanation. A "conversion rate" metric is purchases / sessions. Each platform expresses it differently — Cube via a number-typed measure referencing two sum measures, dbt SL via a ratio metric, LookML via a number measure that references two count measures.

Question. Define conversion_rate = purchases / sessions in all three platforms. Show the safe-division pattern (NULLIF on the denominator).

Input. Cubes / semantic_models / views for sessions (with a purchases flag column).

Code.

# Cube — derived measure
cubes:
  - name: Sessions
    sql_table: analytics.fct_sessions
    measures:
      sessions_count:
        type: count
      purchases_count:
        type: count
        filters:
          - sql: "{CUBE}.purchase_flag = true"
      conversion_rate:
        type: number
        sql: "{purchases_count} * 1.0 / NULLIF({sessions_count}, 0)"
        format: percent

# dbt SL — first-class ratio metric
metrics:
  - name: conversion_rate
    type: ratio
    type_params:
      numerator: purchases_count
      denominator: sessions_count

# LookML — number measure with safe division
view: sessions {
  measure: sessions_count {
    type: count
  }
  measure: purchases_count {
    type: count
    filters: [purchase_flag: "yes"]
  }
  measure: conversion_rate {
    type: number
    sql: ${purchases_count} * 1.0 / NULLIF(${sessions_count}, 0) ;;
    value_format_name: percent_2
  }
}

Step-by-step explanation.

Cube and LookML express the ratio as a derived measure — purchases / sessions with NULLIF to protect against zero division.
dbt SL provides a first-class ratio type. MetricFlow generates the NULLIF-style protection automatically and ensures both metrics are aggregated at the same granularity before the ratio is computed.
All three return the same number for any given granularity. The dbt SL form is the most concise; the Cube and LookML forms make the safe-division explicit.
Formatting (format: percent) lives in the layer so every consumer renders the metric the same way (e.g. 12.4%, not 0.124).

Output.

platform	conversion_rate (this week)
Cube	12.4%
dbt SL	12.4%
LookML	12.4%

Rule of thumb. For first-order ratios, dbt SL's typed metric is the cleanest. For ratios with conditional filters on numerator and denominator (e.g. "conversion rate of paid users"), all three platforms let you wrap the measure in a filter — choose the one whose syntax your team is already fluent in.

Semantic layer interview question on cross-platform metric translation

A senior interviewer might frame: "We are migrating from LookML to dbt SL. Translate this LookML measure into the dbt SL equivalent: a count_distinct user_id filtered to purchase events, grouped at weekly granularity, with a 30-day rolling window option."

Solution Using a dbt SL `simple` metric plus a `cumulative` overlay

# models/semantic/events.yml
semantic_models:
  - name: events
    model: ref('fct_events')
    defaults:
      agg_time_dimension: event_ts
    entities:
      - name: event_id
        type: primary
      - name: user_id
        type: foreign
    dimensions:
      - name: event_ts
        type: time
        type_params: { time_granularity: day }
      - name: event_name
        type: categorical
    measures:
      - name: distinct_purchasers
        agg: count_distinct
        expr: user_id
        agg_time_dimension: event_ts
        # measure filter — only purchase events feed this measure
        filter: "{{ Dimension('events__event_name') }} = 'purchase'"

metrics:
  - name: weekly_purchasers
    label: Weekly purchasers
    type: simple
    type_params:
      measure: distinct_purchasers

  - name: purchasers_30d_rolling
    label: 30-day rolling purchasers
    type: cumulative
    type_params:
      measure: distinct_purchasers
      window: 30 days

Step-by-step trace.

Step	Action	Notes
1	LookML measure `count_distinct(user_id) filter purchase` → dbt SL measure `distinct_purchasers`	filter expressed in YAML, not Looker liquid
2	LookML `event_week` dimension → dbt SL `metric_time__week` group-by	granularity injected at query time
3	New requirement: 30-day rolling window	dbt SL `cumulative` metric with `window: 30 days`
4	Time spine joined automatically	gaps filled with zero rows
5	Consumer queries: `weekly_purchasers` OR `purchasers_30d_rolling`	one schema, two surfaces

Output:

metric_time__week	weekly_purchasers	purchasers_30d_rolling
2026-05-31	4,120	14,800
2026-06-07	4,395	15,210
2026-06-14	4,602	15,690

Why this works — concept by concept:

Measure-level filter — declaring the event_name = 'purchase' filter on the measure means every metric built from it inherits the filter. No risk that a downstream metric forgets the filter.
Metric vs measure separation — measures are LEGO bricks; metrics are the dashboard-facing KPIs. The same distinct_purchasers measure powers both the weekly and the rolling-30-day metric.
Cumulative metric type — MetricFlow handles the rolling-window math (joining each row to a 30-day lookback range) without the analyst hand-rolling a window function.
Time spine join — gaps in activity become zero rows. Dashboards don't render "missing" weeks; they render "zero" weeks, which is what executives expect.
Granularity-agnostic measure — distinct_purchasers works at any grain. Query at metric_time__day for DAU-style; query at metric_time__week for WAU-style; query at metric_time__month for MAU-style. One measure, three dashboards.
Cost — one warehouse pass per granularity per refresh; MetricFlow plans the SQL once and reuses across consumers.

SQL
Topic — case-expression
Conditional metric and CASE expression problems (SQL)

Practice →

5. Consumer fan-out — who queries the semantic layer

One query API, many surfaces — Tableau, Power BI, Hex, Mode, embedded apps, and LLM agents from the same metric file

The mental model in one line: a well-placed semantic layer is consumed by every analytics surface in your stack — BI tools through SQL or JDBC, notebooks through the SQL endpoint, embedded apps through REST or GraphQL, and LLM agents through the published schema as a tool surface. Each consumer gets the same metric definition, the same RLS predicates, and the same cache benefits.

The consumer map in one paragraph.

Heavy BI tools — Tableau, Power BI, Looker, Sigma, MicroStrategy — speak JDBC / ODBC or a vendor-specific connector.
Notebooks — Jupyter, Hex, Deepnote, Mode, Databricks notebooks — speak SQL via JDBC or a Python client.
Embedded analytics — React / Vue / Angular charts inside SaaS products — speak REST or GraphQL.
LLM agents — Slack bots, custom GPTs, Anthropic / OpenAI tools — speak the semantic layer's schema as a tool surface, then the layer emits SQL.

BI tools.

Looker. Native LookML consumer. The semantic layer is Looker for Looker.
Tableau. Reads dbt SL via the dbt Cloud connector. Reads Cube via the Cube SQL API. Reads LookML indirectly via Looker SQL Runner or by exporting an extract.
Power BI. Reads dbt SL via the dbt Cloud / Tableau-style connector. Reads Cube via SQL API.
Hex / Mode / Sigma. Best-in-class native dbt SL integrations; also speak Cube SQL API.

Notebooks.

Jupyter. A cube-jupyter-client Python package or a generic JDBC driver loads metric results into a DataFrame.
Hex. First-class dbt SL integration — drag a metric into a cell, get a DataFrame.
Deepnote. Speaks SQL over JDBC against the layer.
Databricks notebooks. Useful for ML feature engineering — read the metric, train on it.

Embedded analytics — Cube's GraphQL/REST surface.

REST. A POST to /cubejs-api/v1/load with a JSON query. Returns a JSON result with rows and metadata.
GraphQL. A cube(where: ...) { events { weeklyActiveUsers } } query against the published schema. Strongly typed.
JWT auth. Every embedded request carries a JWT signed by the host app. The layer extracts claims (tenant_id, user_id, role) and binds them to RLS predicates.
Front-end SDKs. @cubejs-client/react, @cubejs-client/vue give you <QueryRenderer> components that take a query and render charts.

AI/LLM agents — text-to-SQL using the semantic layer as the grounding context.

The agent receives the question: "What was the WAU last week broken down by region?"
The agent calls a tool that returns the published cubes / measures / dimensions / joins as a structured schema.
The agent constructs a semantic-layer query (not raw SQL): Events.active_users grouped by Regions.region_name, last week.
The layer compiles to SQL, runs against the warehouse, returns the result.
The agent renders the answer in natural language with the metric name and definition cited — auditable.

Cost-control: caching layers, materialised aggregates, query budgets.

Caching. Per-consumer or global; TTL or invalidation on metric deploy.
Materialised aggregates. Cube pre-aggs; dbt-materialised saved queries; Looker PDTs.
Query budgets. Per-tenant or per-consumer rate limits — the embedded React chart cannot accidentally DDoS the warehouse because the layer enforces a poll interval.

Authentication, RLS, tenant isolation across consumers.

Auth. Each consumer attaches a JWT or service-account token. The layer resolves the identity once.
RLS. Per-tenant or per-user predicates injected before the warehouse SQL is emitted.
Tenant isolation. Cache keys partition by tenant claim. No cross-tenant cache pollution.
Audit. Every query is logged with the calling identity, the metric requested, the SQL emitted, and the result row count.

Migration patterns.

LookML → dbt SL. Port the 10 top metrics first; keep Looker live for the long tail. Dual-run for 30 days per metric; reconcile numbers daily.
LookML → Cube. Same staged pattern; Cube becomes the primary for embedded + LLM, Looker keeps the existing dashboards.
Greenfield — pick by consumer mix. Mostly embedded + LLM → Cube. Already on dbt + mostly Tableau/Hex/Mode → dbt SL. Already on Looker with no near-term embedded plans → stay on LookML.

Worked example — wiring the same metric into Tableau, Hex, and a React app

Detailed explanation. Three consumers want the same weekly_active_users metric. Each speaks a different protocol. The semantic layer's published definitions are the same file; the wiring is per-consumer.

Question. Show the connection / query snippet for Tableau (JDBC), Hex (native SL integration), and a React embedded chart (REST).

Input. A semantic layer with weekly_active_users published. Endpoints: SQL at :13306, REST at /cubejs-api/v1/load, GraphQL at /cubejs-api/v1/graphql.

Code.

-- Tableau / JDBC consumer
SELECT MEASURE(weekly_active_users) AS wau,
       event_week
FROM events
GROUP BY event_week
ORDER BY event_week;

# Hex notebook — dbt SL integration (Hex syntax)
import pandas as pd
from hex_sl import query

df = query(
    metrics=["weekly_active_users"],
    group_by=["metric_time__week"],
    start_time="2026-03-22",
    end_time="2026-06-14",
)
df.head()

// React embedded chart — Cube REST via @cubejs-client/react
import { CubeProvider, useCubeQuery } from "@cubejs-client/react";
import { Line } from "react-chartjs-2";

function WauChart() {
  const { resultSet } = useCubeQuery({
    measures: ["Events.active_users"],
    timeDimensions: [{
      dimension: "Events.event_ts",
      granularity: "week",
      dateRange: ["2026-03-22", "2026-06-14"],
    }],
  });
  if (!resultSet) return <p>Loading…</p>;
  return <Line data={resultSet.chartPivot()} />;
}

Step-by-step explanation.

Tableau opens a JDBC connection to the layer. The query reads like normal SQL, but the MEASURE(weekly_active_users) syntax tells the layer to resolve the named metric (rather than treat weekly_active_users as a column).
Hex's native integration takes Python-style arguments and translates them into the dbt SL query API. The result is a DataFrame ready for further analysis.
The React component uses the @cubejs-client/react hook. The query object is the same JSON shape as a REST request; the component re-renders on result.
All three consumers see the same number because they all read from the same metric definition on the layer.

Output (one number, three surfaces).

Consumer	Protocol	Code surface	Returned (this week)
Tableau	JDBC SQL	`SELECT MEASURE(weekly_active_users)`	19,432
Hex	SL Python	`query(metrics=[...])`	19,432
React chart	REST/JS	`useCubeQuery(...)`	19,432

Rule of thumb. Pick the consumer protocol that matches the host environment — JDBC for heavy BI, native SDK for first-class notebook integration, REST/GraphQL for embedded. The metric definition is invariant; only the wiring changes.

Worked example — RLS across a B2B SaaS embedded chart

Detailed explanation. A B2B SaaS product embeds a chart of "your active users this week" in every customer's tenant dashboard. The same chart code ships to 200 tenants; each tenant must see only their own number. RLS at the semantic layer makes the front-end code identical for every tenant.

Question. Show the JWT flow, the layer's RLS rewrite, and the resulting SQL for two different tenants.

Input. Two tenants: tenant_A (Alice) and tenant_B (Bob). Both load the same React embedded component.

Code.

// Host app generates a JWT with the tenant_id claim
const token = jwt.sign(
  { tenant_id: currentUser.tenant_id, sub: currentUser.id },
  process.env.CUBE_API_SECRET,
  { expiresIn: "10m" }
);

// Front-end attaches the token to every request
<CubeProvider
  cubeApi={cubejs(token, { apiUrl: "https://semantic.example.com/cubejs-api/v1" })}
>
  <WauChart />
</CubeProvider>

# semantic layer — cube definition with RLS
cubes:
  - name: Events
    sql: |
      SELECT *
      FROM analytics.fct_events
      WHERE tenant_id = '{COMPILE_CONTEXT.securityContext.tenant_id}'
    measures:
      active_users:
        type: countDistinct
        sql: user_id

-- SQL emitted for Alice (tenant_A)
SELECT COUNT(DISTINCT user_id) AS active_users
FROM   (SELECT * FROM analytics.fct_events WHERE tenant_id = 'tenant_A') events
WHERE  event_ts >= DATE_TRUNC('week', CURRENT_DATE);

-- SQL emitted for Bob (tenant_B)
SELECT COUNT(DISTINCT user_id) AS active_users
FROM   (SELECT * FROM analytics.fct_events WHERE tenant_id = 'tenant_B') events
WHERE  event_ts >= DATE_TRUNC('week', CURRENT_DATE);

Step-by-step explanation.

The host app signs a JWT containing tenant_id. The layer's API secret verifies the signature.
The React component is the same for both tenants — no tenant logic in front-end code.
The semantic layer's cube SQL templates the tenant_id from the verified JWT into the FROM clause. The warehouse only ever sees scoped data.
The cache key partitions by tenant_id. Tenant A's cache and tenant B's cache are physically distinct entries.
Adding a third tenant requires zero code changes — just a new JWT with tenant_id = "tenant_C".

Output.

Caller	tenant_id	Returned active_users
Alice	tenant_A	412
Bob	tenant_B	158
Internal (no JWT)	—	request denied

Rule of thumb. Multi-tenant SaaS without semantic-layer RLS is a leak waiting to ship. Front-end tenant logic will be forgotten — at some endpoint, in some refactor — and a customer will see another customer's data. The layer-side predicate is the only one that holds.

Worked example — LLM agent grounding via the semantic layer

Detailed explanation. An LLM agent receives the question "What was WAU last week by region, and how does it compare to the week before?" The agent's tools include cube_meta (returns published cubes / measures / dimensions / joins) and cube_query (executes a structured query). The agent never writes raw SQL.

Question. Walk through the tool calls, the structured query the agent constructs, the layer's compiled SQL, and the natural-language answer.

Input. Cube with Events.active_users measure, Regions.region_name dimension, joined via Events → Customers → Regions.

Code.

// Tool: cube_meta (called once at session start)
{
  "cubes": [
    {
      "name": "Events",
      "measures": ["Events.active_users"],
      "dimensions": ["Events.event_ts", "Events.event_name"],
      "joins": [{"to": "Customers", "on": "customer_id"}]
    },
    {
      "name": "Customers",
      "joins": [{"to": "Regions", "on": "region_id"}]
    },
    {
      "name": "Regions",
      "dimensions": ["Regions.region_name"]
    }
  ]
}

// Tool: cube_query (called by the agent to answer the question)
{
  "measures": ["Events.active_users"],
  "dimensions": ["Regions.region_name"],
  "timeDimensions": [{
    "dimension": "Events.event_ts",
    "granularity": "week",
    "dateRange": ["2026-06-01", "2026-06-14"]
  }]
}

-- The semantic layer compiles to (sketch)
SELECT r.region_name,
       DATE_TRUNC('week', e.event_ts) AS event_week,
       COUNT(DISTINCT e.user_id) AS active_users
FROM   analytics.fct_events e
JOIN   analytics.dim_customers c ON e.customer_id = c.customer_id
JOIN   analytics.dim_regions   r ON c.region_id  = r.region_id
WHERE  e.event_ts >= '2026-06-01'
  AND  e.event_ts <  '2026-06-15'
GROUP BY r.region_name, DATE_TRUNC('week', e.event_ts)
ORDER BY event_week, r.region_name;

Step-by-step explanation.

The agent loads cube_meta to see the published schema. It learns: there is an Events.active_users measure, a Regions.region_name dimension, and an implicit join chain between them.
The agent constructs a structured cube_query — not SQL — using the measures and dimensions it just learned. The query asks for Events.active_users grouped by Regions.region_name at weekly granularity over the last 14 days.
The semantic layer compiles the structured query to SQL, including the join chain Events → Customers → Regions. The agent never wrote a JOIN clause.
The result rows come back per region per week. The agent computes the week-over-week delta in natural language and presents the answer.
Critically: the agent could not have hallucinated a wrong join. The schema published by cube_meta is the only surface it has access to.

Output (sample).

region_name	event_week	active_users	wow_delta
EU	2026-06-01	9,210	n/a
EU	2026-06-08	9,440	+2.5%
US	2026-06-01	7,890	n/a
US	2026-06-08	8,150	+3.3%
APAC	2026-06-01	1,210	n/a
APAC	2026-06-08	1,332	+10.1%

The agent renders: "WAU last week was 18,922, up 4.0% from 18,310 the prior week. APAC grew fastest at +10.1%." — and cites Events.active_users as the metric.

Rule of thumb. Ground LLM agents on the semantic layer's schema, not on the raw warehouse schema. The constrained surface eliminates the entire category of "hallucinated JOIN" failures and makes every agent response auditable.

Worked example — migration from LookML to dbt SL, staged by metric

Detailed explanation. A company on Looker decides to migrate to dbt SL. Rather than a big-bang rewrite, they identify the 10 top metrics by query volume and rewrite those first. The other 190 LookML measures stay live until they are naturally retired or rebuilt.

Question. Outline the staged migration plan with a 30-day dual-run reconciliation per metric.

Input. 200 LookML measures; top 10 cover 80% of dashboard query volume.

Code.

migration:
  approach: staged
  parallel_run_days: 30
  top_metrics:
    - weekly_active_users
    - daily_active_users
    - total_revenue
    - new_signups
    - churn_rate
    - retention_d7
    - retention_d30
    - average_order_value
    - conversion_rate
    - lifetime_value
  reconciliation:
    cadence: daily
    tolerance_pct: 0.5
    block_threshold_pct: 1.0
    owner: analytics-engineering

Step-by-step explanation.

Inventory all LookML measures and rank by query volume from Looker's i__looker.history system table. Pick the top 10 — they cover most of the business.
Rewrite each measure as a dbt SL metric (often a simple over a count_distinct or sum measure). For derived measures, use ratio or derived types.
Dual-run for 30 days. Every day, a reconciliation job queries both LookML and dbt SL for the same metric, same filters, same granularity. If the numbers differ by > 0.5%, raise a warning; > 1.0%, block the migration of that metric.
After 30 days of clean reconciliation, mark the dbt SL metric as primary. New dashboards point at dbt SL; the LookML measure stays read-only for one quarter, then is deleted.
Repeat for the next batch of 10 metrics. The long tail (190 measures) is migrated opportunistically as dashboards are rebuilt.

Output.

Stage	Metrics migrated	LookML measures live	Dual-run pass rate
Stage 1 (months 1–2)	10 (top 10)	200	100%
Stage 2 (months 3–4)	30 (next 20)	170	100%
Stage 3 (months 5–6)	80 (next 50)	120	98%
Stage 4 (months 7–12)	200 (all)	0	99% overall

Rule of thumb. Stage the migration by query volume, not alphabetically and not by team ownership. The top 10 metrics carry the migration's business value; the long tail can sit on LookML for as long as Looker is licensed.

Semantic layer interview question on cross-consumer reliability

A senior interviewer might frame it as: "Your CTO walks in with a complaint: 'The number on the executive dashboard is different from the number in the embedded customer portal and different again from what the Slack bot says.' Walk me through the diagnosis and the fix."

Solution Using semantic-layer consolidation as the audit-and-fix pattern

# Step 1: diagnose — list every place "weekly_active_users" is computed
diagnosis:
  exec_dashboard:
    source: Looker calculated field
    formula: count_distinct user_id where event_ts >= ...
  embedded_portal:
    source: Tableau workbook
    formula: count_distinct user_id where event_name = 'login' and event_ts >= ...
  slack_bot:
    source: ad-hoc Snowflake query in Python
    formula: count_distinct user_id where event_ts >= dateadd(day, -7, current_date)

# Step 2: fix — publish one metric, point every surface at it
metrics:
  - name: weekly_active_users
    description: Distinct users with any event in the trailing 7 days.
    type: simple
    type_params:
      measure: distinct_users
    filter: "{{ TimeDimension('events__event_ts') }} >= dateadd('day', -7, current_date)"

# Step 3: rewire consumers
consumers:
  exec_dashboard:  semantic_layer_sql_endpoint
  embedded_portal: semantic_layer_rest_endpoint
  slack_bot:       semantic_layer_graphql_endpoint

Step-by-step trace.

Step	Consumer	Pre-fix number	Why it differed	Post-fix number
1	Exec dashboard	19,221	included all events	18,922
2	Embedded portal	14,650	filtered to `login` only	18,922
3	Slack bot	18,990	off-by-one in date window	18,922
4	All three after fix	18,922	all reading from same metric	18,922

Output:

Consumer	Source of `weekly_active_users`	Discrepancy after consolidation
Exec dashboard	semantic layer SQL endpoint	0
Embedded portal	semantic layer REST endpoint	0
Slack LLM bot	semantic layer GraphQL endpoint	0

Why this works — concept by concept:

Diagnosis first — every "numbers don't match" incident starts as a survey of every place the metric is computed. The semantic layer's value proposition is precisely this consolidation.
One file, one definition — the migrated metric lives once in weekly_active_users.yml. There is no other place to edit it.
Consumer rewiring — each surface stops computing the metric in-tool and starts reading from the layer's endpoint. The viz / chart / bot code shrinks.
Cache convergence — within one cache TTL of the fix landing, every consumer reports the same number. The "numbers different by consumer" symptom physically cannot reproduce.
Audit trail — every future change to the metric flows through a PR. The CTO's "why did this number change?" question is answered by git log weekly_active_users.yml.
Cost — usually a reduction — three separate consumer-side computations collapse into one cached semantic-layer query.

SQL
Topic — sql
End-to-end SQL practice for analytics engineering

Practice →

Cheat sheet — semantic layer recipes

All-in on Looker, no near-term embedded. Stay on LookML, integrate dbt for the upstream model layer, and let Looker continue to be the consumer. The semantic layer is Looker.
dbt-native, multiple BI tools (Tableau / Power BI / Hex / Mode). Adopt the dbt Semantic Layer + MetricFlow. Definitions live next to dbt models; CI catches metric breaks at PR time.
Embedded analytics or AI agents are the dominant consumer. Choose Cube.dev for the REST / GraphQL surface. The semantic layer becomes the agent's tool surface.
Need pre-aggregated cubes for sub-second BI on billion-row facts. Use Cube's pre_aggregations — declare the roll-up grain, partition by month, refresh on a TTL.
Need date spine + cumulative / rolling-window metrics. dbt SL has best-in-class time-spine support; cumulative metrics with window: 30 days are first-class.
Need to ground an LLM agent on governed metrics. Publish the semantic layer's schema as the agent's tool surface. The agent constructs structured queries; the layer compiles to SQL.
Multi-tenant SaaS embedded analytics. Route every consumer through the semantic layer with JWT-driven RLS. The cube sql template inlines tenant_id from securityContext.
Migrating off LookML. Stage by query volume — port the top 10 metrics first, dual-run for 30 days, reconcile daily, and let the long tail retire naturally over the next 12 months.
Picking between Cube and dbt SL. If consumers are 60%+ embedded / API / LLM, lean Cube. If consumers are 60%+ existing BI tools and you already own a dbt repo, lean dbt SL.
Defining a metric once. Pick the LEGO-brick measure (count_distinct, sum, count) and let the platform compose the metric. Avoid hand-rolling separate measures for DAU, WAU, MAU — use one and vary granularity.
Safe division in derived metrics. Wrap the denominator in NULLIF(..., 0) (Cube and LookML) or use the typed ratio metric (dbt SL).
Cache eviction on metric edit. Make sure the semantic layer flushes the cache on deploy — otherwise consumers see the old number for the duration of the TTL after a definition change.
Audit trail. Every metric definition lives in git. git log <metric.yml> is the answer to "why did this number change?" — for compliance, finance, and the CEO.

Frequently asked questions

Do I need a semantic layer if I use dbt?

You can run dbt without a semantic layer — your marts will be clean and well-tested, and analysts will write SQL on top of them. The semantic layer matters the moment more than one consumer asks for the same metric. Without it, the metric is re-implemented in each BI tool, each notebook, and each embedded chart — and the numbers drift. The dbt Semantic Layer (powered by MetricFlow) is the natural choice if you already use dbt, because metric definitions live in the same repo as your models and flow through the same PR review and CI. If your consumers include embedded analytics or LLM agents, Cube is often the better fit because of its REST/GraphQL surface; in that case you keep dbt for the model layer and put Cube on top.

Can I use the dbt Semantic Layer without dbt Cloud?

Partially. MetricFlow (the engine behind the dbt Semantic Layer) ships as open source and runs from dbt Core via the mf CLI — you can define semantic_models and metrics, validate them, and query metrics locally. What you do not get without dbt Cloud Team or Enterprise is the hosted Semantic Layer server, the caching tier, the JDBC connector for Tableau / Power BI / Hex / Mode, and the official integrations. For most teams that pay for any BI tool today, the dbt Cloud tier pays for itself in eliminated metric-duplication and reduced warehouse spend; for pure OSS shops, MetricFlow + a custom query gateway is a viable path but more work.

Is Cube.dev free?

The Cube Core engine is open source and free to self-host — Docker, your warehouse, your servers. Cube Cloud is the paid managed offering, which adds the IDE, hosted pre-aggregation runners, deployment automation, query history, role-based access control, and lineage. Teams typically start on Cube Core for evaluation and adopt Cube Cloud once metric count grows beyond 50 or embedded analytics SLAs require professional infrastructure. The OSS edition is genuinely usable in production — Cube was the first semantic layer with this OSS-with-paid-managed model and remains the most popular standalone semantic engine.

How does LookML compare to the dbt Semantic Layer?

LookML is the original semantic modelling language and is tightly coupled to Looker as the consumer surface — view and explore files map to the Looker UI, the IDE, and the SQL Runner. The dbt Semantic Layer is consumer-agnostic and lives in your existing dbt repo. LookML wins on IDE polish, content validation, and the fact that Looker users get a fully managed experience. dbt SL wins on portability (Tableau, Power BI, Hex, Mode all consume it natively), on cost (no per-seat Looker license), and on lineage (metrics live next to the models they read from). The most common 2026 migration pattern is staged: dbt SL for new dashboards and the top 10 metrics, LookML for the long tail until naturally retired.

Can LLM agents query a semantic layer?

Yes — and this is the use case that pushed every major BI vendor to ship a semantic-layer story in 2024–2026. The agent reads the published schema (cubes / semantic_models / views, plus measures, dimensions, and joins) as a tool surface, then constructs structured queries against the layer. The layer compiles to SQL. The agent never writes raw SQL itself, which eliminates the "hallucinated JOIN" failure mode that plagues text-to-SQL on raw warehouse schemas. Cube's GraphQL surface and dbt SL's API are both well-suited for this; the LookML route works via Looker's API but with more friction. For LLM-heavy roadmaps, Cube is the most common pick.

What's the difference between a metrics layer and a semantic layer?

The terms overlap heavily in 2026 marketing. A "metrics layer" emphasises the named KPIs (WAU, MAU, revenue, churn, retention) — the dashboard-facing numbers. A "semantic layer" emphasises the broader modelling surface — measures, dimensions, joins, entity relationships, segments — that lets you derive metrics. dbt SL, Cube, and LookML are all semantic layers in this fuller sense; LookML calls itself "semantic modelling," dbt SL talks about "semantic_models," and Cube talks about "cubes." The 2020-era "metrics layer" companies (Transform, Supergrain, Trace, GoodData) either pivoted into full semantic layers or were acquired by larger semantic-layer / BI vendors. Practically, when you read "metrics layer" today, assume "semantic layer" — they refer to the same engineering object.

Practice on PipeCode

Drill the aggregation practice library → for the sum / count / count_distinct measures that every semantic layer composes metrics from.
Rehearse on joins problems → for the entity-resolution patterns Cube, dbt SL, and LookML automate.
Sharpen group-by drills → for the granularity reasoning every measure definition leans on.
Layer the case-expression library → for conditional metric patterns ("count only paid orders").
Stack the window-functions practice library → for rolling, cumulative, and ranking metrics.
Layer the filtering library → for the segment / where-clause patterns metric definitions inherit.
For the broader surface, read top data engineering interview questions →.
Stack the prerequisites with the only 5 skills you need to become a data engineer →.
Sharpen the SQL axis with the SQL for data engineering interviews course →.
For schema craft and the modelling fundamentals every semantic layer leans on, work through the data modelling for data engineering interviews course →.

Pipecode.ai is Leetcode for Data Engineering — every semantic-layer recipe above ships with hands-on practice rooms where you write the `count_distinct` measure, the entity-based join, and the safe-division ratio against real graded inputs. PipeCode pairs every reading with 450+ DE-focused problems and a real-time scoring engine, so you can rehearse the analytics-engineering moves behind Cube, dbt SL, and LookML against the same SQL fundamentals every interviewer probes.

Practice aggregation now →
JOIN drills →

dbt Model Contracts, Constraints & Versioning: Production Patterns

Gowtham Potureddi — Mon, 15 Jun 2026 13:29:34 +0000

dbt model contracts are the single biggest reason teams stopped breaking dashboards on Mondays. Before dbt 1.5 the only thing standing between a renamed column and a Tuesday-morning incident was a tribal Slack ping; after 1.5 a contract.enforced block fails the PR in CI before the rename ever lands. The shape of your warehouse — the column names, the data types, the not-null promises — is now a first-class artefact your repo owns.

This guide walks the dbt contracts + dbt constraints + dbt model versions triple end to end: where each one fits, how the dbt-Core 1.5+ feature timeline lined them up, and the dbt production patterns that make contract enforcement, schema evolution, and dbt versioning survive contact with a multi-team analytics org. Each section ships a worked example with code, a step-by-step trace, an output, and a concept-by-concept Why-this-works.

When you want hands-on reps alongside the reading, drill the data modelling practice library →, rehearse on dimensional modelling problems →, and tighten the schema-evolution muscles with slowly-changing-data drills →.

On this page

Why dbt models need contracts in production
Anatomy of a dbt model contract
Constraints — primary key, foreign key, not null, check
Versioning strategy for public models
Rollout and deprecation playbook
Cheat sheet — dbt contract recipes
Frequently asked questions
Practice on PipeCode

1. Why dbt models need contracts in production

Contracts catch the kind of bug dbt tests cannot — the interface bug, not the value bug

The one-sentence invariant: dbt tests guarantee that the rows in a model are correct; dbt model contracts guarantee that the shape of the model itself is correct — the columns it exposes, the types of those columns, and the nullability promises downstream consumers depend on. Once you internalise that "tests are about values, contracts are about interfaces," the whole production-hardening surface starts to make sense.

The three places interface bugs hide.

Silent column renames. Someone renames customer_email to email_address in stg_customers.sql. Every test still passes (the new column has the same values), every dashboard breaks at midnight when it tries to read the old name. No PR reviewer caught it because the column was added and the old one was removed in the same commit — the diff just looked like "edited a SELECT clause."
Data type drift. A staging model exposed order_total as numeric(18,2). Someone refactors and the new SQL emits numeric(38,18). The dashboard still works in dev (Postgres is loose about precision), then a Tableau live connection on Redshift fails on the first row because the consumer expected the old precision.
Nullability flips. dim_customer.signup_at was always non-null because the upstream model filtered out incomplete rows. A refactor removes the filter for performance. Now signup_at is sometimes NULL — downstream reverse-ETL crashes on the first NULL it sees.

The dbt-Core 1.5+ feature timeline.

dbt 1.5 (April 2023) shipped model contracts (contract.enforced: true) and constraints (the four kinds: not_null, unique, primary_key, foreign_key, plus check). This is the moment dbt projects gained a way to declare the public shape of a model and have the build fail if the shape drifts.
dbt 1.5 also shipped model versions — the versions: block, latest_version, deprecation_date, and ref('model', v=1) cross-version references. Together with contracts these three features form the "stable interface" toolkit.
dbt 1.6+ (July 2023 onwards) added access: modifiers (private, protected, public) and groups — so a model can be marked private to a single group of authors and ref() from outside that group fails to compile.
dbt 1.7+ (Q4 2023 onwards) added the unit testing framework — orthogonal to contracts but synergistic, because unit tests assert the rows that a contracted model produces.

Where contracts fit between tests, constraints, observability.

dbt tests. Run after the model materialises; they re-query the table and assert row-level facts (unique, not_null, accepted_values, custom singular tests). They are row-shaped assertions.
dbt contracts. Run before the model materialises; they assert that the SELECT's projected columns match the declared columns: block in YAML — names, types, and constraints. They are interface-shaped assertions that fail fast in PR CI.
dbt constraints. Translate the YAML declaration into DDL where the warehouse supports it; otherwise they remain informational metadata. They are contract reinforcement — when paired with a warehouse that enforces them, they fail the load instead of poisoning a downstream join.
Data observability platforms (Monte Carlo, Bigeye, Lightup). Detect drift in production after the fact — useful, but reactive. Contracts make the same drift a PR-time failure, which is two orders of magnitude cheaper to fix.

The 2026 reality.

Contracts are now table-stakes for public models. Any model ref()-ed from outside its owning group, exported to reverse-ETL, or surfaced in BI should have contract.enforced: true.
Constraints are warehouse-dependent. Postgres and Redshift (mostly) enforce them; Snowflake and BigQuery treat most as informational. dbt translates declarations to DDL in both cases, but the runtime behaviour differs.
Versions are how dbt does SemVer. Breaking changes get a version bump (v2, v3); non-breaking additions stay on the same version. deprecation_date and latest_version give you a 30–90 day overlap window to migrate consumers.

Worked example — the silent column rename that broke Monday

Detailed explanation. A weekend refactor of dim_customer renames signup_at to signed_up_at. Every dbt test passes (the values are unchanged). On Monday, three Looker tiles, a HubSpot reverse-ETL sync, and a Snowflake share to a partner all fail. Total time-to-detect: 14 hours. Total cost: 11 stakeholder threads and one apology email.

Question. Show the dbt YAML diff for adding contract.enforced: true to dim_customer and demonstrate how the same rename would fail in CI instead.

Input — current models/marts/customer/dim_customer.yml.

field	value
name	dim_customer
materialized	table
columns	customer_id, signup_at, email
tests	unique on customer_id, not_null on signup_at

Code.

# models/marts/customer/dim_customer.yml
version: 2

models:
  - name: dim_customer
    config:
      materialized: table
      contract:
        enforced: true        # <- the upgrade
    columns:
      - name: customer_id
        data_type: bigint
        constraints:
          - type: not_null
          - type: primary_key
      - name: signup_at        # <- the contract anchor
        data_type: timestamp
        constraints:
          - type: not_null
      - name: email
        data_type: varchar

-- models/marts/customer/dim_customer.sql AFTER the rename
SELECT
    customer_id,
    signed_up_at,             -- renamed from signup_at
    email
FROM {{ ref('stg_customer') }};

Step-by-step explanation.

dbt 1.5+ compiles dim_customer.sql against the YAML contract. It runs the SELECT once in a transaction (or as a dry-run on warehouses that support it) and inspects the returned column metadata.
The contract declares signup_at as a column. The SELECT returns signed_up_at instead. dbt diffs the two sets and emits a contract violation.
The CI job — dbt build --select state:modified+ — fails. The PR cannot be merged. The "Monday morning incident" became a "Friday afternoon code-review comment."
The author either rolls back the rename (cheap) or coordinates a versioning bump (dim_customer_v2) so consumers can migrate on their own schedule.

Output.

Compilation Error in model dim_customer
  This model has an enforced contract that failed.
  Please ensure the name, data_type, and number of columns in your contract
  match the columns in your model's definition.

  | column_name      | definition_type | contract_type | mismatch_reason     |
  | ---------------- | --------------- | ------------- | ------------------- |
  | signed_up_at     | TIMESTAMP       |               | missing in contract |
  | signup_at        |                 | TIMESTAMP     | missing in definition|

Rule of thumb. Every model that is ref()-ed from outside its group, or that has any non-dbt consumer (BI, reverse-ETL, share), should carry contract.enforced: true. The cost is a one-time YAML block; the saving is every "why did the dashboard explode?" incident you never have to write a postmortem for.

Worked example — tests catch a value bug, contracts catch an interface bug

Detailed explanation. A common confusion: "I already have a not_null test on this column — why do I also need a contract?" Tests run after the model loads and re-query the warehouse. They catch the column being NULL today. Contracts encode the promise that the column exists, has a name, has a type, and may have a not-null constraint — and they fail the build before the model materialises.

Question. A staging model stg_orders accidentally drops the order_id column in a refactor. Compare what happens with only dbt tests vs with a contract.

Input — the broken refactor.

-- BEFORE refactor (correct)
SELECT
    order_id,
    customer_id,
    amount
FROM {{ source('raw', 'orders') }};

-- AFTER refactor (accidentally drops order_id)
SELECT
    customer_id,
    amount
FROM {{ source('raw', 'orders') }};

Code — tests-only YAML.

version: 2
models:
  - name: stg_orders
    columns:
      - name: order_id
        tests: [unique, not_null]
      - name: customer_id
        tests: [not_null]

Code — tests + contract YAML.

version: 2
models:
  - name: stg_orders
    config:
      contract:
        enforced: true
    columns:
      - name: order_id
        data_type: bigint
        constraints: [{ type: not_null }, { type: unique }]
      - name: customer_id
        data_type: bigint
        constraints: [{ type: not_null }]
      - name: amount
        data_type: numeric

Step-by-step explanation.

Tests only. dbt build runs the broken SELECT. The model materialises successfully (it just has two columns now). Then dbt tries to test order_id — and gets a "column does not exist" error from the warehouse. The test "fails" but with a runtime database error, not a contract-style error. Worse: the table is already broken in the dev schema by the time the test runs.
Tests + contract. dbt build compiles the model against the contract before running it. The contract declares three columns; the SELECT only projects two. The compile fails with a clear contract-violation message naming the missing column. Nothing materialises; nothing breaks.
The contract catches the bug two phases earlier in the dbt graph (compile, not test) and emits a domain-specific error ("contract violation: missing column order_id") instead of a warehouse error.

Output.

Strategy	Detected at	Error type	Side effects
Tests only	After build, during test	warehouse "column not found"	broken table left in dev schema
Tests + contract	At compile, before build	dbt "contract violation"	nothing materialised

Rule of thumb. Tests are still essential — they catch value drift (NULLs creeping in, a unique key suddenly duplicating). Contracts catch interface drift (columns disappearing, types changing). You want both. Think "belt + braces," not "either/or."

Worked example — contracts on incremental models

Detailed explanation. A common worry: "does contract.enforced work with incremental materialisation?" Yes, with one caveat: dbt enforces the contract on every full-refresh build and on the schema check at the start of every incremental run. The incremental delta INSERT must produce the contracted column set, or the run fails.

Question. Show the YAML for a contracted incremental fact model fct_orders and explain when contract enforcement runs.

Code.

version: 2
models:
  - name: fct_orders
    config:
      materialized: incremental
      unique_key: order_id
      on_schema_change: fail   # belt-and-braces: explicit schema-change policy
      contract:
        enforced: true
    columns:
      - name: order_id
        data_type: bigint
        constraints: [{ type: not_null }, { type: primary_key }]
      - name: customer_id
        data_type: bigint
        constraints:
          - type: not_null
          - type: foreign_key
            expression: "{{ ref('dim_customer') }} (customer_id)"
      - name: order_ts
        data_type: timestamp
        constraints: [{ type: not_null }]
      - name: amount
        data_type: numeric
        constraints: [{ type: not_null }]

Step-by-step explanation.

Full refresh. dbt build --full-refresh --select fct_orders runs the SELECT, validates the projected columns against the contract, then drops-and-recreates the table with the declared DDL (including constraints, where supported). The contract is checked once, decisively.
Incremental run. dbt build --select fct_orders (no --full-refresh) inspects the existing target table and compares its column set to the contract. If they match, dbt runs the incremental delta SELECT, validates its projection against the contract, then INSERTs / MERGEs into the target.
on_schema_change: fail is critical when contracts are on. Without it, dbt's default incremental behaviour might append a new column silently — which would still pass the contract check (the new column is in both the SELECT and the table) but would drift the contract's declared shape over time. Fail-on-change keeps the table strictly in sync with the YAML.

Output. A contracted incremental model behaves like a frozen interface from the consumer's perspective. The table at version N exposes exactly the columns in the contract, with exactly the declared types, on every load — and any drift in the SQL that would change that shape is caught before INSERT.

Rule of thumb. Set on_schema_change: fail whenever contract.enforced: true is on for an incremental model. The two flags compose to give you "the table never changes shape without a YAML edit" — which is exactly what your downstream consumers want.

dbt interview question on the contracts vs tests axis

A senior interviewer often probes: "Walk me through the difference between a dbt test and a dbt contract. Give me one scenario where a contract catches a bug that tests cannot, and one where a test catches a bug that contracts cannot."

Solution Using the contracts-tests matrix

# models/marts/customer/dim_customer.yml
version: 2
models:
  - name: dim_customer
    config:
      materialized: table
      contract:
        enforced: true
    columns:
      - name: customer_id
        data_type: bigint
        constraints: [{ type: not_null }, { type: primary_key }]
        tests: [unique, not_null]
      - name: email
        data_type: varchar
        tests:
          - not_null
          - dbt_utils.expression_is_true:
              expression: "email like '%@%'"

Step-by-step trace.

Bug scenario	Tests-only outcome	Contract-only outcome	Both outcome
`customer_id` renamed to `cust_id` in SQL	runtime warehouse error during test	PR fails at compile	PR fails at compile
`customer_id` type changed `bigint` → `string`	tests still pass (values unique, non-null)	PR fails at compile	PR fails at compile
`email` column suddenly contains NULLs	not_null test fails post-build	contract still passes (column exists)	not_null test fails post-build
`email` column missing the `@`	expression test fails post-build	contract still passes	expression test fails post-build

The matrix surfaces the orthogonality crisply: contracts catch shape changes (rename, type drift, missing column); tests catch value changes (NULL appearing where it shouldn't, a unique key duplicating, a format violation). Neither subsumes the other.

Output:

Bug class	Catch with	Catches before	Detection cost
Renamed column	contract	model materialises	low (compile-time)
Type drift	contract	model materialises	low (compile-time)
NULL creeping in	tests	downstream consumer	medium (post-build)
Format violation	tests	downstream consumer	medium (post-build)

Why this works — concept by concept:

contract.enforced as a compile-time gate — runs before any DDL is issued. dbt compiles the SELECT, inspects the projected columns via the warehouse's metadata (or a dry-run plan), and diffs them against the YAML. Mismatches abort the build.
dbt tests as a post-build sentinel — run after the model materialises. They re-query the table and assert row-level facts. Cheap to write, but they catch issues after the broken table exists in dev / CI.
The two are orthogonal axes — contracts cover the columns-types-nullability axis, tests cover the values-and-relationships axis. Mature projects use both, with the contract as the first line of defence and the tests as the audit layer.
on_schema_change as the third leg — for incremental models, the contract pins the current shape; on_schema_change: fail ensures the shape cannot drift silently between contract edits. Without it, the table can grow extra columns invisibly.
Cost — contracts add O(columns) compile-time work per build (negligible); tests add one SELECT per test per build. Both are dominated by the actual model build time on any realistic dataset.

Data modeling
Topic — design
Design problems (data modeling)

Practice →

2. Anatomy of a dbt model contract

`contract.enforced: true` plus a filled-out columns block is the entire vocabulary — but every field has a precise job

The mental model in one line: a dbt contract is a YAML declaration that names every column, its data type, its constraints, and its description — and contract.enforced: true makes dbt verify the SELECT matches that declaration before the model is allowed to materialise. The block is small; the semantics are precise; the failure mode is "build aborts," not "warning printed and continues."

The five mandatory pieces.

config.contract.enforced: true — the master switch. Without it, the rest of the YAML is documentation. With it, dbt diffs the SELECT against the columns block at compile time and aborts on mismatch.
columns: - name: ... — every column the model projects must appear in the columns block, by name, in any order. Extra YAML columns not in the SELECT, or extra SELECT columns not in YAML, both fail the contract.
data_type: — the warehouse-canonical type (bigint, varchar, numeric(18,2), timestamp, boolean). dbt normalises common synonyms (int8 → bigint on Postgres) but it pays to use the exact word the warehouse echoes.
constraints: — a list of constraint declarations. Each has a type: (not_null, unique, primary_key, foreign_key, check) and optional fields (name:, expression:, columns: for composite, warn_unenforced: / warn_unsupported:).
description: — free-form prose; surfaced in dbt docs and the catalog. Not strictly enforced but is the single best place to document the semantic intent of the column for downstream consumers.

Compile-time vs run-time enforcement.

Compile-time (the default). dbt asks the warehouse to plan the SELECT without running it, inspects the projected columns from the plan metadata, and diffs them against the contract. Cheap and fast — milliseconds per model. Fails the PR in CI before any data moves.
Run-time. On warehouses that enforce constraints (Postgres, Redshift for some, Databricks Unity Catalog), the CREATE TABLE statement carries the constraints as actual DDL. Inserting a NULL into a not_null column raises a database error at write time. This is in addition to the compile-time contract check, not instead of it.
The handshake. Contracts give you the compile-time interface guarantee; constraints (on enforcing warehouses) give you the run-time value guarantee. They overlap on names like not_null but cover different failure modes.

A "shape" assertion, not a "value" assertion.

The contract checks that order_id is declared as bigint and the SELECT produces a bigint column called order_id. It does not check that any particular row's order_id is non-null.
Adding constraints: [{ type: not_null }] to the contract is the bridge — it asks dbt to also attempt warehouse-level enforcement of "no NULL values in this column." On Postgres that becomes a NOT NULL DDL clause. On Snowflake it becomes informational metadata (the warehouse does not enforce).
For the value-level audit you still want tests: [not_null] — that runs a SELECT COUNT(*) FROM t WHERE col IS NULL after the build and asserts zero.

Interactions with materialisation.

materialized: table — full DDL re-created on each build. Constraints are emitted as part of the CREATE TABLE. Contracts checked at compile.
materialized: view — view definition checked at compile. Constraints in the YAML are documentation only because most warehouses do not attach constraints to views.
materialized: incremental — full DDL on --full-refresh; incremental INSERT / MERGE on normal runs. Contracts checked on every run (compile-time). Combine with on_schema_change: fail.
materialized: ephemeral — no DDL; the model is inlined as a CTE in consumers. Contracts cannot apply (no projected table). dbt warns if you try.

Common interview probes on contract anatomy.

"What happens if the SELECT projects an extra column not in the contract?" — contract violation, build aborts.
"What happens if YAML declares an extra column not in the SELECT?" — same — contract violation.
"Is column order part of the contract?" — no. dbt diffs the set of columns, not the ordering.
"Does the contract validate types end-to-end?" — yes, but the matching is dialect-aware (int and bigint are not interchangeable; numeric and numeric(18,2) can differ depending on warehouse).

Worked example — turning an unmodelled `dim_customer` into a public contracted model

Detailed explanation. The team has been treating dim_customer as "internal" for a year. As of this quarter, the marketing-ops team wants to ref() it from a new mart, and reverse-ETL is going to sync it to HubSpot. That makes it public by every definition that matters. Time to ship a contract.

Question. Promote dim_customer.sql from an unmodelled table to a contracted, constrained, public-ready model. Show the YAML diff and explain each line.

Input — current YAML.

version: 2
models:
  - name: dim_customer

Code — promoted YAML.

version: 2
models:
  - name: dim_customer
    description: >
      One row per customer. Source of truth for downstream marts, BI tiles,
      and reverse-ETL syncs to HubSpot. Schema is public — bump the version
      for any breaking change.
    config:
      materialized: table
      contract:
        enforced: true
      access: public
      group: customer
    columns:
      - name: customer_id
        data_type: bigint
        description: "Surrogate key; stable across all loads."
        constraints:
          - type: not_null
          - type: primary_key
        tests: [unique, not_null]
      - name: email
        data_type: varchar
        description: "Primary contact email; lowercased, trimmed."
        constraints:
          - type: not_null
          - type: unique
          - type: check
            expression: "email like '%@%.%'"
        tests:
          - not_null
          - unique
      - name: signup_at
        data_type: timestamp
        description: "First successful account creation (UTC)."
        constraints: [{ type: not_null }]
        tests: [not_null]
      - name: tier
        data_type: varchar
        description: "Loyalty tier — one of {bronze, silver, gold}."
        constraints:
          - type: check
            expression: "tier in ('bronze','silver','gold')"
        tests:
          - accepted_values:
              values: [bronze, silver, gold]

Step-by-step explanation.

description: is now mandatory in spirit — it is the first thing a consumer reads in dbt docs. Keep it short, concrete, and oriented toward consumers, not authors.
materialized: table makes the constraint DDL meaningful. On Postgres the table will be created with customer_id bigint PRIMARY KEY NOT NULL, email varchar UNIQUE NOT NULL, ....
contract.enforced: true is the master switch. The first time you dbt build this model, the SELECT must already project exactly {customer_id, email, signup_at, tier} with matching types — otherwise the build fails.
access: public and group: customer declare the visibility of the model. Combined with the contract, this is dbt's full "public API" pattern: a ref('dim_customer') from any other group will be allowed; from within the same group it is free. A private model can ignore most of this YAML.
Each column has both contract constraints: and dbt tests:. The constraints are compile-time + DDL-time guards (the warehouse may enforce them); the tests are post-build value audits. The redundancy is the point — belt and braces.

Output. First build on Postgres emits:

CREATE TABLE analytics.dim_customer (
    customer_id bigint NOT NULL,
    email       varchar NOT NULL,
    signup_at   timestamp NOT NULL,
    tier        varchar,
    PRIMARY KEY (customer_id),
    UNIQUE (email),
    CHECK (email like '%@%.%'),
    CHECK (tier in ('bronze','silver','gold'))
);

On Snowflake the same DDL is emitted but most constraints land as informational metadata (visible in INFORMATION_SCHEMA but not enforced on INSERT). dbt then runs the tests post-build and asserts the value-level facts.

Rule of thumb. When you promote a model to public, ship the whole anatomy in one PR: description, contract.enforced: true, access: public, group:, full columns: block with types + constraints + descriptions, and matching tests. Splitting it across multiple PRs is how teams end up with partially-contracted models that look safe in the catalog but skip half the enforcement.

Worked example — what dbt does to the warehouse on first build

Detailed explanation. Knowing the exact CREATE TABLE that dbt emits per warehouse is the difference between "I trust the contract" and "I checked what landed." Each warehouse translates the YAML differently, and the gaps are the source of most "I thought my FK was enforced" surprises.

Question. Given the contracted dim_customer from above, write out the literal CREATE TABLE statements dbt emits on Postgres, Snowflake, BigQuery, and Redshift.

Code — Postgres (full enforcement).

CREATE TABLE analytics.dim_customer (
    customer_id bigint NOT NULL,
    email       varchar NOT NULL,
    signup_at   timestamp NOT NULL,
    tier        varchar,
    CONSTRAINT dim_customer_pk PRIMARY KEY (customer_id),
    CONSTRAINT dim_customer_email_uk UNIQUE (email),
    CONSTRAINT dim_customer_email_chk CHECK (email like '%@%.%'),
    CONSTRAINT dim_customer_tier_chk  CHECK (tier in ('bronze','silver','gold'))
);

Code — Snowflake (mostly informational).

CREATE OR REPLACE TABLE analytics.dim_customer (
    customer_id bigint NOT NULL,
    email       varchar NOT NULL,
    signup_at   timestamp_ntz NOT NULL,
    tier        varchar,
    CONSTRAINT dim_customer_pk PRIMARY KEY (customer_id) NOT ENFORCED,
    CONSTRAINT dim_customer_email_uk UNIQUE (email) NOT ENFORCED
);
-- CHECK and FK in Snowflake are not supported / informational; dbt logs a warning

Code — BigQuery (only NOT NULL + primary-key metadata).

CREATE OR REPLACE TABLE `proj.analytics.dim_customer` (
    customer_id INT64 NOT NULL,
    email       STRING NOT NULL,
    signup_at   TIMESTAMP NOT NULL,
    tier        STRING,
    PRIMARY KEY (customer_id) NOT ENFORCED
);
-- UNIQUE / CHECK not supported as DDL; dbt logs a warning

Code — Redshift (NOT NULL enforced, others informational).

CREATE TABLE analytics.dim_customer (
    customer_id bigint NOT NULL,
    email       varchar NOT NULL,
    signup_at   timestamp NOT NULL,
    tier        varchar,
    PRIMARY KEY (customer_id),   -- informational only
    UNIQUE (email)               -- informational only
);

Step-by-step explanation.

Postgres is the only of the four to enforce every declared constraint at the database level. Inserting a NULL into signup_at, a duplicate email, or an invalid tier value all raise an error at write time.
Snowflake enforces NOT NULL and that is it. PRIMARY KEY and UNIQUE are declared as NOT ENFORCED for documentation / catalog / query-planner-hint purposes. CHECK and FOREIGN KEY are not supported as DDL at all — dbt logs a warning and drops them.
BigQuery enforces NOT NULL. As of recent versions it supports PRIMARY KEY ... NOT ENFORCED and FOREIGN KEY ... NOT ENFORCED for query-planner hints only. UNIQUE and CHECK are not supported.
Redshift enforces NOT NULL. PRIMARY KEY, UNIQUE, and FOREIGN KEY are accepted syntactically but are informational only (the optimizer may use them as hints; insertions are not blocked).
The contract itself is a compile-time guarantee on all four — dbt diffs the SELECT against the YAML regardless of warehouse. The runtime enforcement is what differs.

Output.

Warehouse	NOT NULL	UNIQUE	PRIMARY KEY	FOREIGN KEY	CHECK
Postgres	enforced	enforced	enforced	enforced	enforced
Snowflake	enforced	informational	informational	not supported	not supported
BigQuery	enforced	not supported	informational	informational	not supported
Redshift	enforced	informational	informational	informational	not supported

Rule of thumb. Always pair contracted constraints with matching dbt tests. The constraint is the warehouse-side aspiration; the test is the actual audit. On enforcing warehouses (Postgres) you may consider the test redundant — but the moment your project becomes multi-warehouse, the tests are the only thing that keeps the behaviour identical.

Worked example — a contracted view (and why most teams use tables)

Detailed explanation. Contracts on materialized: view are compile-time only — the column projection of the view's SELECT is diffed against the YAML, but no DDL constraints are attached (views in most warehouses cannot carry constraints). This is sometimes a deal-breaker; more often it is the correct choice for cheap, derived models.

Question. Show a contracted view for vw_active_customers (filters dim_customer to non-deleted rows) and explain what the contract does and does not guarantee.

Code.

version: 2
models:
  - name: vw_active_customers
    config:
      materialized: view
      contract:
        enforced: true
    columns:
      - name: customer_id
        data_type: bigint
        constraints: [{ type: not_null }]
      - name: email
        data_type: varchar
        constraints: [{ type: not_null }]
      - name: tier
        data_type: varchar

-- models/marts/customer/vw_active_customers.sql
SELECT
    customer_id,
    email,
    tier
FROM {{ ref('dim_customer') }}
WHERE deleted_at IS NULL;

Step-by-step explanation.

dbt compiles the view, inspects the SELECT's projection, and diffs against the YAML — same as for a table.
dbt then issues CREATE OR REPLACE VIEW analytics.vw_active_customers AS SELECT customer_id, email, tier FROM analytics.dim_customer WHERE deleted_at IS NULL;. No constraints attach.
The contract guarantees: at compile time, the SELECT projects exactly {customer_id, email, tier} with matching types. After deploy, queries against the view always see those three columns with those types.
The contract does not guarantee NULL-safety at the warehouse level. If dim_customer.email happens to contain a NULL row that passes the deleted_at IS NULL filter, the view will return it. The contract only documents the intent; you still need a test (tests: [not_null]) to audit value-level facts.

Output. The view materialises as a stable interface. Downstream consumers can rely on the column set; they cannot rely on the constraints being enforced at write time (because the view does not write — it reads from a base table). All value-level promises must come from tests on the base table or on the view itself.

Rule of thumb. Contracted views are great for "cheap, stable façades" — filter-only or projection-only models that wrap a public table. The moment you need actual constraint enforcement, switch to materialized: table. The cost is one storage copy; the benefit is real DDL guarantees.

dbt interview question on contract anatomy

A senior interviewer often probes: "I gave you a model that is ref()-ed by five downstream marts and a reverse-ETL sync. Walk me through the minimum YAML I should ship to make it contract-safe, and explain what each field defends against."

Solution Using the full public-model pattern

version: 2
models:
  - name: dim_product
    description: >
      One row per product. Public — every breaking schema change ships
      as a new version (v2, v3) with a 60-day deprecation window.
    config:
      materialized: table
      contract:
        enforced: true
      access: public
      group: product
    columns:
      - name: product_id
        data_type: bigint
        description: "Stable surrogate key."
        constraints: [{ type: not_null }, { type: primary_key }]
        tests: [unique, not_null]
      - name: sku
        data_type: varchar
        description: "Vendor SKU — uppercase, no whitespace."
        constraints: [{ type: not_null }, { type: unique }]
        tests: [unique, not_null]
      - name: category_id
        data_type: bigint
        description: "FK to dim_category.category_id."
        constraints:
          - type: not_null
          - type: foreign_key
            expression: "{{ ref('dim_category') }} (category_id)"
        tests:
          - not_null
          - relationships:
              to: ref('dim_category')
              field: category_id
      - name: price
        data_type: numeric(18,2)
        description: "List price in USD."
        constraints:
          - type: not_null
          - type: check
            expression: "price >= 0"
        tests:
          - not_null
          - dbt_utils.expression_is_true:
              expression: "price >= 0"

Step-by-step trace.

YAML field	What it defends against	Catches at
`contract.enforced: true`	renamed / removed / retyped columns in SQL	compile
`access: public` + `group:`	accidental `ref()` from outside the owning group on private models	compile
`data_type:` on every column	type drift (`bigint` → `int`, `numeric(18,2)` → `numeric(38,18)`)	compile
`not_null` constraint	NULL insertion (Postgres / Redshift / Snowflake / BigQuery)	run-time
`primary_key` constraint	duplicate keys (Postgres only); query-plan hint elsewhere	run-time / planner
`foreign_key` constraint	orphan rows (Postgres only); query-plan hint elsewhere	run-time / planner
`check` constraint	invalid values (Postgres only); informational elsewhere	run-time
`tests:` block	actual value drift in production after build	post-build

Output:

Layer	Guarantees	Cost
Contract	Column set, names, types	Compile-time (ms per model)
Constraints (Postgres)	NULL-safety, uniqueness, referential integrity, check	DDL + insertion overhead
Constraints (Snowflake / BigQuery / Redshift)	NULL-safety only; rest are catalog metadata	Negligible
Tests	Value-level audits	One SELECT per test per build

Why this works — concept by concept:

contract.enforced as the interface lock — the YAML becomes the source of truth for "what columns does this model expose," and dbt fails any build that drifts from it. Consumers can refactor with confidence.
access: public + group — visibility metadata. Private models can be refactored freely within their group; public models are the ones that need versions when the shape changes. This is dbt's analog to "public API vs internal helper."
Constraints as the warehouse-side aspiration — the YAML declares the constraint; the warehouse may or may not enforce it. Either way, the declaration shows up in dbt docs and the catalog, making the intent discoverable.
Tests as the audit — every constraint should have a matching test, because the test runs identically on all warehouses. Tests are the dialect-independent way to guarantee value semantics.
Description as the consumer doc — surfaced in dbt docs and in IDE tooltips. Costs five seconds; saves the consumer from a Slack ping every time they want to know "is signup_at UTC or local?"
Cost — compile-time overhead is negligible (milliseconds per model). The biggest "cost" is the discipline to keep the YAML in sync with the SQL — which is exactly the discipline you want.

Data modeling
Topic — dimensional-modeling
Dimensional modeling problems (data modeling)

Practice →

3. Constraints — primary key, foreign key, not null, check

Five constraint kinds, five very different stories about whether the warehouse actually enforces them

The mental model in one line: dbt declares five constraint kinds in YAML; each warehouse picks a different subset to actually enforce at write time, and the rest live as informational metadata for the catalog and the query planner. Once you can name which constraints land as real DDL on your warehouse, the rest of the constraint conversation is about choosing where tests fill the gap.

The five constraint kinds.

not_null — "no row of this column may be NULL." Every major warehouse enforces this at INSERT time. The cheapest, most universal, and most useful constraint.
unique — "no two rows share this value." Postgres enforces; Snowflake / BigQuery / Redshift declare informationally.
primary_key — "this column (or set) is the row identity." Implies not_null + unique. Postgres enforces both halves; the others treat as informational metadata that the query planner may consult.
foreign_key — "this column references a column in another table." Postgres enforces (subject to indexes); Snowflake / BigQuery declare informationally; Redshift declares informationally.
check — "this column satisfies a boolean expression." Postgres enforces. Snowflake / BigQuery / Redshift do not support CHECK as DDL — dbt logs a warning and skips.

Single-column vs composite constraints.

Single-column. Declare inside the column's constraints: list. Most natural for not_null / unique / primary_key.
Composite. Declare at the model level under model-level constraints:. Example: a composite primary key on (order_id, line_no). Each constraint declaration includes a columns: list naming the participating columns.

Informational vs enforced — the practical impact.

Enforced. The warehouse refuses INSERTs / MERGEs that would violate the constraint. Bugs surface at write time, often immediately, with a clear error from the database.
Informational. The constraint is recorded in the warehouse catalog but not checked at write time. The query planner may use it to rewrite joins (e.g. eliminate a DISTINCT when joining on a primary key). Bugs surface downstream, often hours or days later.
The practical rule. On informational warehouses (Snowflake / BigQuery / Redshift), the constraint is documentation + query-planner hint. You still need a matching dbt test to actually audit the values.

Constraint + test — belt and braces, not duplicated work.

The constraint tells the warehouse what the model promises. On enforcing warehouses, it is real. On informational ones, it is a hint.
The test tells dbt (and the CI / scheduler) to run a value-level audit after every build. It works identically on every warehouse and surfaces silent drift.
For mature projects, ship both. The constraint is the declaration of intent; the test is the verification.

Foreign-key gotchas in warehouses with no FK enforcement.

Snowflake allows FOREIGN KEY ... NOT ENFORCED syntactically (some versions). dbt emits the DDL where possible; otherwise warns and drops.
BigQuery supports FOREIGN KEY ... NOT ENFORCED for query-planner hints (since late 2023). The constraint is metadata only.
Redshift accepts FOREIGN KEY syntactically; the optimiser uses it as a join hint but does not enforce.
Postgres is the outlier — FKs are real, but they require an index on the referenced column (otherwise INSERT performance suffers).
The pragmatic FK pattern on non-Postgres warehouses. Declare the FK in YAML for documentation and catalog clarity, then add a dbt relationships test for the actual audit:

- name: customer_id
  constraints:
    - type: foreign_key
      expression: "{{ ref('dim_customer') }} (customer_id)"
  tests:
    - relationships:
        to: ref('dim_customer')
        field: customer_id

Common interview probes on constraints.

"On Snowflake, does declaring a PRIMARY KEY actually prevent duplicates?" — no; it is informational. Add a unique test.
"What is the difference between primary_key and unique + not_null?" — semantically identical (PK = unique + not null); syntactically PK is one declaration, the catalog distinguishes them, and the query planner treats PK as "the canonical row identity."
"When would you skip declaring an FK?" — when the referenced table is enormous and the FK overhead would matter (rare in analytics warehouses; common in OLTP). In analytics, declare the FK informationally on every column that joins to a dimension.
"Why do constraints not duplicate tests?" — they cover different failure modes. Constraints prevent bad writes (where supported); tests audit existing data post-build. You need both.

Worked example — a contracted star-schema fact with FKs to dims

Detailed explanation. A fct_orders fact table joins to three dimensions: dim_customer, dim_product, dim_date. The contract declares an FK to each, plus a composite PK on (order_id, line_no) for the order-line grain, plus a check on quantity.

Question. Write the YAML for a contracted, constrained fct_orders model with three FKs and a composite PK.

Input — model SQL.

-- models/marts/sales/fct_orders.sql
SELECT
    order_id,
    line_no,
    customer_id,
    product_id,
    date_id,
    quantity,
    unit_price,
    quantity * unit_price AS line_amount
FROM {{ ref('int_orders') }};

Code — YAML with composite PK and three FKs.

version: 2
models:
  - name: fct_orders
    description: "Fact table at the order-line grain."
    config:
      materialized: table
      contract:
        enforced: true
      access: public
      group: sales

    constraints:
      - type: primary_key
        columns: [order_id, line_no]
      - type: foreign_key
        columns: [customer_id]
        expression: "{{ ref('dim_customer') }} (customer_id)"
      - type: foreign_key
        columns: [product_id]
        expression: "{{ ref('dim_product') }} (product_id)"
      - type: foreign_key
        columns: [date_id]
        expression: "{{ ref('dim_date') }} (date_id)"

    columns:
      - name: order_id
        data_type: bigint
        constraints: [{ type: not_null }]
      - name: line_no
        data_type: integer
        constraints:
          - type: not_null
          - type: check
            expression: "line_no >= 1"
      - name: customer_id
        data_type: bigint
        constraints: [{ type: not_null }]
        tests:
          - relationships:
              to: ref('dim_customer')
              field: customer_id
      - name: product_id
        data_type: bigint
        constraints: [{ type: not_null }]
        tests:
          - relationships:
              to: ref('dim_product')
              field: product_id
      - name: date_id
        data_type: integer
        constraints: [{ type: not_null }]
      - name: quantity
        data_type: integer
        constraints:
          - type: not_null
          - type: check
            expression: "quantity > 0"
      - name: unit_price
        data_type: numeric(18,2)
        constraints:
          - type: not_null
          - type: check
            expression: "unit_price >= 0"
      - name: line_amount
        data_type: numeric(18,2)
        constraints: [{ type: not_null }]

Step-by-step explanation.

The composite primary_key is declared at the model level — constraints: directly under the model, with a columns: list naming the two participating columns. Composite PKs cannot be declared inside a single column's block because no single column owns the constraint.
The three foreign_key constraints are also declared at the model level (one per FK). Each names the local columns: and the expression: referencing the target table and column.
Column-level constraints: carry not_null and check for each column. Note the check (quantity > 0) and check (unit_price >= 0) — these are quality guarantees that surface as DDL on Postgres and as informational hints elsewhere.
The relationships tests on customer_id and product_id are the dbt-side audit that catches orphans on any warehouse, regardless of FK enforcement.
On Postgres the DDL is fully enforced: any INSERT with a missing FK target, duplicate (order_id, line_no), or quantity <= 0 raises an error. On Snowflake / BigQuery / Redshift the constraints are informational; the not_null portion still enforces, but PK / FK / CHECK do not.

Output. A fct_orders table whose interface is locked: composite PK, three FKs, quantity-must-be-positive, unit-price-must-be-non-negative. Any drift in the SELECT fails the build at compile time; any orphan in the data fails the relationships test post-build.

Rule of thumb. Composite keys always live at the model level. Single-column constraints live inside the column block. Every FK should be paired with a relationships test (or it is a hint, not a guarantee, on most warehouses).

Worked example — a check constraint that catches a tier typo

Detailed explanation. The product team wants to lock the allowed values of tier to {bronze, silver, gold}. On Postgres a CHECK (tier in (...)) constraint will refuse the offending INSERT. On Snowflake the check is unsupported as DDL but the dbt accepted_values test still audits the same property.

Question. Show the YAML for the tier column with both a check constraint and an accepted_values test, and explain when each fires.

Code.

- name: tier
  data_type: varchar
  description: "Loyalty tier — one of {bronze, silver, gold}."
  constraints:
    - type: not_null
    - type: check
      expression: "tier in ('bronze','silver','gold')"
  tests:
    - not_null
    - accepted_values:
        values: [bronze, silver, gold]

Step-by-step explanation.

On Postgres. The CREATE TABLE includes tier varchar NOT NULL, CONSTRAINT dim_customer_tier_chk CHECK (tier in ('bronze','silver','gold')). Any INSERT with tier = 'platinum' raises a database error and aborts the transaction.
On Snowflake. The CHECK is not supported as DDL; dbt emits a warning ("CHECK constraint is not supported on Snowflake — skipping") and the constraint becomes documentation only.
The accepted_values test. After build, dbt runs SELECT COUNT(*) FROM dim_customer WHERE tier NOT IN ('bronze','silver','gold') and asserts the count is zero. This works identically on every warehouse.
The combined effect: Postgres catches the bad row at write time; Snowflake catches it post-build. Either way, the bad row never makes it to production — but the latency-to-detection differs by minutes (Postgres) vs the test phase (Snowflake).

Output. A tier column whose semantics are documented in YAML, enforced at write time on Postgres, audited post-build on every warehouse. The constraint and the test together cover every workflow.

Rule of thumb. Always pair check constraints with matching accepted_values or expression_is_true tests. The constraint is the warehouse-side aspiration; the test is the cross-warehouse guarantee.

Worked example — composite unique on a deduplicated staging model

Detailed explanation. A staging model stg_orders should have one row per (source_system, source_order_id). The single-column order_id is not unique on its own — different source systems can collide. A composite unique constraint expresses the actual identity.

Question. Write the YAML composite unique for (source_system, source_order_id) on stg_orders.

Code.

version: 2
models:
  - name: stg_orders
    config:
      materialized: table
      contract:
        enforced: true

    constraints:
      - type: unique
        columns: [source_system, source_order_id]

    columns:
      - name: source_system
        data_type: varchar
        constraints: [{ type: not_null }]
      - name: source_order_id
        data_type: varchar
        constraints: [{ type: not_null }]
      - name: order_id
        data_type: bigint
        description: "Surrogate key, not unique alone — see composite unique."
      - name: order_ts
        data_type: timestamp
        constraints: [{ type: not_null }]

    tests:
      - dbt_utils.unique_combination_of_columns:
          combination_of_columns: [source_system, source_order_id]

Step-by-step explanation.

The composite unique constraint is declared at model level with columns: [source_system, source_order_id]. On Postgres it becomes UNIQUE (source_system, source_order_id) — enforced.
On Snowflake / BigQuery / Redshift the constraint is informational. The dbt_utils.unique_combination_of_columns test fills the audit gap — it runs a post-build SELECT that GROUPs by the combination and asserts every group has size 1.
The order_id column carries a description that explains it is not unique alone — important for downstream consumers who might be tempted to JOIN on order_id alone.

Output. A staging model whose composite identity is declared, enforced where supported, and audited on every warehouse via the dbt-utils test. New consumers reading the YAML immediately see "the natural key is (source_system, source_order_id)."

Rule of thumb. When the natural key is composite, declare it as a composite unique (model-level) and always add a matching dbt_utils.unique_combination_of_columns test. The combination handles both "informational warehouse" and "value drift" failure modes.

dbt interview question on constraint enforcement

A senior interviewer often probes: "You are on Snowflake. Why does it matter that your contract declares primary_key and foreign_key constraints if Snowflake doesn't enforce them? Walk me through what value you get and what you still need."

Solution Using the constraint + test split

version: 2
models:
  - name: fct_orders
    config:
      materialized: incremental
      unique_key: [order_id, line_no]
      on_schema_change: fail
      contract:
        enforced: true
      access: public
      group: sales

    constraints:
      - type: primary_key
        columns: [order_id, line_no]
      - type: foreign_key
        columns: [customer_id]
        expression: "{{ ref('dim_customer') }} (customer_id)"

    columns:
      - name: order_id
        data_type: bigint
        constraints: [{ type: not_null }]
      - name: line_no
        data_type: integer
        constraints: [{ type: not_null }]
      - name: customer_id
        data_type: bigint
        constraints: [{ type: not_null }]
        tests:
          - relationships:
              to: ref('dim_customer')
              field: customer_id
      - name: amount
        data_type: numeric(18,2)
        constraints: [{ type: not_null }]

    tests:
      - dbt_utils.unique_combination_of_columns:
          combination_of_columns: [order_id, line_no]

Step-by-step trace.

Bug class	Snowflake DDL guards?	dbt test guards?	What you would lose without each
NULL `order_id`	yes (NOT NULL is enforced)	yes (column-level not_null is implied by PK declaration; explicit test optional)	nothing extra needed
Duplicate `(order_id, line_no)`	no — PK is informational	yes (`unique_combination_of_columns`)	the only line of defence on Snowflake
Orphan `customer_id`	no — FK is informational	yes (`relationships`)	the only line of defence on Snowflake
Renamed `customer_id` column in SQL	n/a — contract catches at compile	n/a	compile-time guarantee from contract.enforced
Type drift `numeric` → `varchar`	n/a — contract catches at compile	n/a	compile-time guarantee from contract.enforced

The declaration of primary_key and foreign_key in the YAML still buys you four things on Snowflake: (1) catalog metadata (visible in dbt docs, useful for downstream consumers); (2) query-planner hints (Snowflake's optimiser uses informational PKs / FKs to rewrite joins and skip DISTINCT operations); (3) contract-level type enforcement (the column types are pinned even if the constraint is informational); and (4) documentation of intent (the next engineer reading the YAML knows the model's identity story).

Output:

Warehouse	NOT NULL	UNIQUE / PK	FK	CHECK	Tests audit gap
Postgres	enforced	enforced	enforced	enforced	optional belt-and-braces
Snowflake	enforced	informational	informational	not supported	mandatory
BigQuery	enforced	not supported	informational	not supported	mandatory
Redshift	enforced	informational	informational	not supported	mandatory

Why this works — concept by concept:

Constraints as catalog metadata — even when not enforced, the declarations appear in INFORMATION_SCHEMA, dbt docs, and the catalog. This is how lineage tools (Atlan, Castor, Stemma) discover the relationships and render the right diagrams.
Query-planner hints — Snowflake's optimiser will, for example, skip a DISTINCT pass when joining on a column declared unique/PK. Same on BigQuery for FK-driven join elimination. The constraint is "advisory" but has real performance impact.
Contract.enforced type pinning — independent of constraint enforcement. The contract diff at compile catches renames and type drift on every warehouse — that part is rock-solid regardless of constraint reality.
dbt tests as the cross-warehouse audit — unique, not_null, relationships, accepted_values, and the dbt_utils.* family run identically on every warehouse. They are the portable enforcement layer.
The pairing as the actual production pattern — declare the constraint (for catalog + planner + contract), add the matching test (for audit). The intentional redundancy is what makes the project survive a warehouse migration.
Cost — constraints add zero DDL cost on informational warehouses; minimal DDL cost on Postgres (one index per UNIQUE/PK). Tests cost one SELECT per test per build — already part of any mature dbt CI.

Data modeling
Topic — cardinality
Cardinality problems (data modeling)

Practice →

4. Versioning strategy for public models

Versions are how dbt does SemVer — breaking changes get a new number, non-breaking stay on the same one

The mental model in one line: a dbt model version is a sibling model with the same logical name but a different shape, identified by a v= suffix; you publish v2 alongside v1, give v1 a deprecation_date, and let consumers migrate on their own schedule. Versions are the cleanest way to ship breaking changes without a war-room rollout.

The versions: block.

Top-level declaration. Inside the model YAML, add a versions: list. Each entry declares a v: number and optional overrides (description, columns, contract, defined_in).
latest_version: — names the version that ref('model') (without a v= argument) resolves to. Consumers without a v= get the latest by default.
defined_in: — the SQL filename for that version. If absent, defaults to model_vN.sql. Useful when versions live in separate files for clarity.
deprecation_date: — a date after which the version should not be used. dbt emits warnings during compile if any consumer still references a deprecated version after the date.

SemVer for data — the three rules of thumb.

MAJOR (v2, v3). Breaking change — column removed, renamed, retyped to an incompatible type, semantics changed (e.g. "amount in USD" → "amount in customer's local currency"). Consumers must migrate.
MINOR. Non-breaking addition — new column added at the end, new constraint added (where consumers are not relying on its absence), new test. Stays on the same version number.
PATCH. Doc-only or comment change. Stays on the same version.

The breaking-vs-non-breaking heuristic.

Breaking. Anything a downstream SELECT * would notice as a removal or rename. Anything a downstream WHERE or JOIN predicate would silently drop rows over (e.g. nullability flip on a join key). Anything a downstream type cast would fail on (e.g. bigint → string).
Non-breaking. Adding a new column at the end (downstream SELECT * gets one extra column; downstream named-column queries are unaffected). Adding a new test. Adding a new constraint that the data already satisfies (it just becomes formally checked).
The grey zone. Tightening a constraint (e.g. relaxing not_null to nullable, or vice versa). Treat tightening as non-breaking if consumers are not relying on the relaxed state; treat relaxing as breaking because a previously non-null column becoming nullable can crash downstream type-narrowed code.

Cross-version refs.

ref('model') — resolves to latest_version.
ref('model', v=1) — resolves to the v1 incarnation. Lets consumers stay on the old version explicitly.
ref('model', v=2) — resolves to the v2 incarnation.
Naming. Physical tables get the suffix: dim_customer_v1, dim_customer_v2. dbt manages the suffixing automatically; consumers only see logical names.

Common interview probes on versioning.

"When would you bump a version vs just edit the model?" — bump when the change is breaking for a public consumer. Edit when it is private, or when the change is non-breaking (additive column, doc, test).
"What is deprecation_date for?" — to advertise the sunset of an older version. dbt warns on compile if consumers still reference it after that date.
"Can two versions of the same model run in the same dbt project?" — yes; they materialise to separate physical tables (suffixed _v1, _v2). Each can have its own contract, columns, and constraints.
"How do I roll back a version?" — keep v1 alive (do not remove it) until you've confirmed v2 has zero issues. Roll back by re-pointing latest_version to v1.

Worked example — shipping v2 of `fct_orders` with a renamed column

Detailed explanation. The team needs to rename order_amount (USD) to order_amount_usd for clarity, in preparation for adding order_amount_eur later. This is a breaking change for every consumer that already references order_amount. Time to ship v2.

Question. Show the YAML and SQL diff for promoting fct_orders from v1 to v2 with the renamed column. Set a 60-day deprecation_date on v1.

Input — current single-version YAML.

version: 2
models:
  - name: fct_orders
    config:
      contract: { enforced: true }
      access: public
    columns:
      - name: order_id
        data_type: bigint
      - name: customer_id
        data_type: bigint
      - name: order_amount
        data_type: numeric(18,2)

Code — versioned YAML.

version: 2
models:
  - name: fct_orders
    config:
      access: public
    latest_version: 2
    versions:
      - v: 1
        config:
          contract: { enforced: true }
        deprecation_date: 2026-08-15  # 60 days from today
        columns:
          - name: order_id
            data_type: bigint
          - name: customer_id
            data_type: bigint
          - name: order_amount
            data_type: numeric(18,2)
            description: "Order total, USD. Renamed to order_amount_usd in v2."
      - v: 2
        config:
          contract: { enforced: true }
        columns:
          - name: order_id
            data_type: bigint
          - name: customer_id
            data_type: bigint
          - name: order_amount_usd
            data_type: numeric(18,2)
            description: "Order total in USD."

Code — SQL files.

-- models/marts/sales/fct_orders_v1.sql (unchanged, kept alive)
SELECT
    order_id,
    customer_id,
    order_amount
FROM {{ ref('int_orders') }};

-- models/marts/sales/fct_orders_v2.sql (new, the renamed column)
SELECT
    order_id,
    customer_id,
    order_amount AS order_amount_usd
FROM {{ ref('int_orders') }};

-- A downstream consumer that wants to stay on v1 explicitly
SELECT customer_id, sum(order_amount) AS total
FROM {{ ref('fct_orders', v=1) }}
GROUP BY 1;

-- A downstream consumer on the latest version (v2)
SELECT customer_id, sum(order_amount_usd) AS total_usd
FROM {{ ref('fct_orders') }}    -- latest_version = 2
GROUP BY 1;

Step-by-step explanation.

latest_version: 2 makes ref('fct_orders') resolve to v2 — every new consumer gets the new shape by default. Existing consumers using ref('fct_orders', v=1) stay on v1.
The versions: list declares both versions side-by-side. Each version has its own columns: block — v1 keeps order_amount, v2 has order_amount_usd.
deprecation_date: 2026-08-15 on v1 tells dbt to start warning consumers 60 days from now. After the deprecation date, any compile that still references v1 emits a "this version is deprecated" warning (and can be configured to error).
Two SQL files (fct_orders_v1.sql, fct_orders_v2.sql) materialise to two physical tables (fct_orders_v1, fct_orders_v2). Both load on every dbt run; storage cost is the only overhead.
Consumers migrate at their own pace by changing ref('fct_orders', v=1) → ref('fct_orders') (or v=2) and updating their column references.

Output. Two physical tables alongside each other. Consumers see the rename as a publish event (v2 is now available) rather than a break event (the column disappeared from under them). The 60-day window gives every team enough runway to plan the migration without a war room.

Rule of thumb. Every breaking change to a public model gets a version bump. Every rename is breaking. Every type narrowing is breaking. Every dropped column is breaking. If you are not sure, default to "ship a v2" — the storage cost of an overlap window is trivial compared to the social cost of a Monday incident.

Worked example — adding a column without a version bump

Detailed explanation. Adding a new column at the end of a model is non-breaking for every consumer that uses named columns. SELECT customer_id, amount FROM fct_orders continues to return the same two columns. SELECT * consumers get one extra column, but the existing ones are unchanged. No version bump needed.

Question. Show the YAML for adding currency to fct_orders without bumping the version.

Code — model-level edit.

- v: 2
  config:
    contract: { enforced: true }
  columns:
    - name: order_id
      data_type: bigint
    - name: customer_id
      data_type: bigint
    - name: order_amount_usd
      data_type: numeric(18,2)
    - name: currency                 # <- new column appended at end
      data_type: varchar
      constraints: [{ type: not_null }]
      description: "ISO 4217 currency code."

-- SQL update
SELECT
    order_id,
    customer_id,
    order_amount AS order_amount_usd,
    coalesce(currency, 'USD') AS currency
FROM {{ ref('int_orders') }};

Step-by-step explanation.

The new currency column is appended at the end of the columns: block. The contract diff at compile sees one extra column in the SELECT — but it is also in the YAML, so the diff matches. The build succeeds.
Existing consumers that wrote SELECT order_id, customer_id, order_amount_usd FROM fct_orders continue to work unchanged — they never named currency, so the new column does not affect them.
New consumers can opt-in to currency simply by adding it to their SELECT.
No version bump needed because nothing breaks for existing consumers. The semantic versioning rule is "minor change → same version" — this is the canonical minor change.
The coalesce(currency, 'USD') AS currency backfills a default for any historical rows where currency was NULL — important because we declared not_null on the new column and the contract would fail otherwise.

Output. A table with one extra column. Existing dashboards, marts, and reverse-ETL syncs are unaffected. New consumers can immediately use the new column. The cost is one YAML edit + one SQL edit + one PR.

Rule of thumb. Append new columns; never insert them. Add columns; never rename them. Loosen constraints with care; tighten them freely (after verifying the data already satisfies the tighter form). These three rules turn 80% of schema evolutions into non-breaking changes that ship in a single PR with zero coordination.

Worked example — a doc-only patch with no contract change

Detailed explanation. A column's description is wrong. Updating it is a pure documentation change — no schema impact, no contract impact, no version impact.

Question. Show a patch that fixes a column description and explain why no version bump is needed.

Code.

- name: signup_at
  data_type: timestamp
  description: >
    Timestamp of first successful account creation, in UTC.
    Was previously documented as "local time" — that was wrong
    on every load. Corrected 2026-06-15.
  constraints: [{ type: not_null }]

Step-by-step explanation.

The patch only edits the description: field. No column rename, no type change, no constraint change.
The contract diff at compile is unchanged — same column name, same type, same constraints.
No consumer was reading description from the YAML at runtime, so no consumer breaks.
The catalog (dbt docs) refreshes with the new description on next build. The lineage tools (Atlan, Castor) refresh on their next pull.
No version bump because nothing about the interface changed. The semantic versioning rule is "patch → same version" — this is the canonical patch.

Output. Updated documentation, zero downstream impact. The cost is one PR with one YAML hunk.

Rule of thumb. Use description: for everything you wish you could write on the column. Future-you (and every consumer) will thank you. Treat description edits as a free PR — they need no version bump, no rollout coordination, no migration window.

Worked example — cross-version `ref()` from a downstream mart

Detailed explanation. A downstream mart agg_revenue_by_customer aggregates fct_orders. The mart owner wants to stay on v1 (with the old order_amount name) for one more quarter while their team plans the migration to v2.

Question. Show the SQL diff for the downstream mart to pin itself to fct_orders v1.

Code.

-- models/marts/sales/agg_revenue_by_customer.sql
SELECT
    customer_id,
    sum(order_amount) AS total_revenue
FROM {{ ref('fct_orders', v=1) }}
GROUP BY customer_id;

Step-by-step explanation.

The ref('fct_orders', v=1) macro resolves to the physical table fct_orders_v1 — the v1 incarnation, with the old column name order_amount.
The mart's SELECT uses order_amount (the v1 name). It compiles and runs against v1's contract, which still declares order_amount.
When the mart team is ready, they change ref('fct_orders', v=1) → ref('fct_orders') (or v=2) and rename order_amount → order_amount_usd in their SELECT. One PR per consumer.
The producer team can drop v1 once every consumer has migrated and the deprecation_date has passed.

Output. The mart stays on v1 indefinitely (or until v1 is removed). The producer ships v2 in parallel. Consumers migrate at their pace.

Rule of thumb. Cross-version ref() is the migration safety net. It lets every team plan its own migration without coordinating on the producer's calendar. The cost is one extra argument in the macro; the benefit is "every team owns its own schedule."

dbt interview question on versioning a public model

A senior interviewer often probes: "Walk me through publishing v2 of a public model that renames a column. What's in the YAML, what's in the SQL, how do consumers stay on v1, and when can you remove v1?"

Solution Using the publish-overlap-deprecate pattern

version: 2
models:
  - name: dim_customer
    config:
      access: public
      group: customer
    latest_version: 2
    versions:
      - v: 1
        config:
          contract: { enforced: true }
        deprecation_date: 2026-09-15  # 90 days from publish
        columns:
          - { name: customer_id, data_type: bigint }
          - { name: signup_at,   data_type: timestamp }   # renamed in v2
          - { name: email,       data_type: varchar }
      - v: 2
        config:
          contract: { enforced: true }
        columns:
          - { name: customer_id,   data_type: bigint }
          - { name: signed_up_at,  data_type: timestamp }  # the renamed column
          - { name: email,         data_type: varchar }

Step-by-step trace.

Date	Producer action	Consumer action	Effect
t=0	Publish v2 alongside v1; set `deprecation_date` 90 days out	Consumers continue on v1 by default until they migrate	both physical tables alive
t=0..30	Comms to consumers: "v2 published, 90-day window"	Forward-looking consumers migrate first	dbt warns on `ref('model', v=1)` after deprecation_date
t=30..75	Track v1 consumers via dbt selectors + query logs	Most consumers migrate; laggards get reminders	v1 traffic shrinks
t=75..90	Final reminder; sunset PR drafted	Last consumers migrate	v1 traffic approaches zero
t=90	Merge sunset PR — remove v1 from YAML and SQL	Any straggler `ref('model', v=1)` now fails to compile	clean state, only v2 alive

Output:

Stage	What exists	Who is affected	Cost
Publish v2	v1 + v2 both alive	nobody (consumers still on v1)	one PR for producer
Overlap window	v1 + v2 both alive	consumers migrate at own pace	storage cost of v1
Deprecation warnings	dbt compile warns on v=1	laggard consumers see warnings	none
Sunset v1	only v2 alive	nobody (everyone migrated)	one PR removing v1

Why this works — concept by concept:

Publish overlap as the migration safety net — v1 and v2 coexist for the deprecation window. Consumers migrate when they are ready, not when the producer demands. Zero coordination meetings required.
SemVer for data — the bump-or-not decision is a type decision (breaking → bump; non-breaking → same version). Once the team internalises the rule, every PR self-classifies and no one argues.
deprecation_date as the social contract — the date is the producer's promise to keep v1 alive that long. It is the consumer's deadline to migrate. dbt's warning at compile is the gentle nag that prevents the deadline from slipping unnoticed.
Cross-version ref() — the migration mechanism. Consumers explicitly pin to v1 with v=1; new consumers default to latest_version. The mechanism is the minimum coupling: one argument per ref.
Sunset PR as the final cut — removing v1 is one YAML edit + one SQL file delete. Any straggler consumer gets a clean compile error pointing at the removed version, not a silent break.
Cost — storage cost of the duplicate table during the overlap window. On most warehouses this is negligible for analytics-scale dims and facts. Compute cost is also low: v1 only loads what was already loading before; v2 loads in parallel.

Data modeling
Topic — slowly-changing-data
Slowly-changing-data problems (data modeling)

Practice →

5. Rollout and deprecation playbook

Coordinating dbt + BI + reverse-ETL on a single timeline — the four-phase rollout that retires v1 without drama

The mental model in one line: the rollout playbook has four phases — Publish, Overlap, Migrate, Sunset — and every stakeholder (producer, consumer, platform) has a defined role inside each phase. Tie the phases to dates in the YAML (deprecation_date) and in your comms calendar, and the social cost of a breaking change drops to near zero.

The four-phase playbook.

Phase 1 — Publish. Producer ships v2 in a single PR. v1 stays alive. deprecation_date is set on v1 (typically 30–90 days out). Comms go out: announcement, migration guide, FAQ, office hours.
Phase 2 — Overlap. Both versions run on every dbt build. Consumers migrate on their own schedule. Producer tracks adoption via dbt selectors and query logs. Comms cadence: weekly reminder, fortnightly tracker.
Phase 3 — Migrate. As deprecation_date approaches, producer surfaces remaining v1 consumers, opens tickets per team, runs office hours for stragglers. dbt compile warnings start firing.
Phase 4 — Sunset. After deprecation_date passes (with confirmation that v1 traffic is zero), producer ships a PR removing v1's YAML, SQL, and (eventually) the physical table.

The overlap window — how long?

30 days. Minimum for any non-trivial public model. Fine for internal teams with tight dbt slack channels.
60 days. A reasonable default for most production analytics orgs. Covers a typical sprint cadence and a vacation overlap.
90 days. For models used by many teams, by BI dashboards owned by non-engineers, or by external (partner-facing) consumers.
The pragmatic rule. Default to 60; bump to 90 if any consumer is non-technical or external; bump to 120 for regulated reporting where audit signoff is required.

Stakeholder comms template.

The announcement (day 0). Short Slack message + email: "We've published dim_customer_v2. v1 is deprecated as of today; sunset is YYYY-MM-DD (60 days). Migration guide: . Office hours: ."
The weekly reminder (day 7, 14, 21, ...). "v2 adoption: X/Y consumers migrated. Stragglers: . Office hours: ."
The pre-sunset warning (day -7). "Sunset in 7 days. Outstanding v1 consumers: . Please migrate or open a ticket for an extension."
The sunset PR (day 0 + window). "v1 removed. v2 is now the only version. Postmortem doc: ."

Tracking consumer migration.

dbt list --select +dim_customer_v1 — every model that downstream-references v1. The list shrinks as consumers migrate.
Query logs — warehouse query history filtered to dim_customer_v1 table name. Surfaces BI tools, reverse-ETL syncs, and ad-hoc consumers that dbt cannot see.
dbt exposures — declarative exposure: YAML blocks let you register BI dashboards, ML jobs, and external consumers as first-class graph nodes. dbt list --select +exposure:dim_customer_v1 then shows everything that depends on v1, including non-dbt artefacts.
The catalog / lineage tool — Atlan / Castor / Stemma surface upstream-downstream relationships including BI tiles. Often the most complete view.

Tying it all to CI.

PR CI. Run dbt build --defer --select state:modified+ on every PR — builds only the modified models (and downstream) against a baseline. Contracts and constraints catch interface changes at compile.
Slim CI. Use --defer against the prod state so the PR build doesn't need to rebuild every upstream model. Faster, cheaper, identical contract enforcement.
Block on contract violations. The contract failure is a build failure — make the PR check required for merge. No exceptions.
Deprecation warnings. Configure CI to fail (not just warn) when consumers reference a model past its deprecation_date. dbt 1.6+ has a --warn-error flag for this.

Coordinating with downstream BI and reverse-ETL.

Looker. Materialised LookML views referencing the dbt table by name need updating. Use a LookML view rename PR in the Looker repo when v2 is published.
Tableau. Live connections reference the table directly. Schedule a "Tableau update day" within the overlap window — extract → swap source → re-publish.
Hightouch / Census (reverse-ETL). Source models reference the dbt table by name. Update the source mapping when v2 is published.
Snowflake Share / BigQuery Authorised Views. External consumers see a view, not the underlying table. Re-create the share / authorised view against v2 during the overlap window so external consumers can migrate on their own schedule.

Postmortems for "contract broke prod" — what to add to the checklist.

Was the model marked contract.enforced: true? If not, why not.
Was the model marked access: public or group:? If not, why was it reachable from outside.
Was the change behind a version bump? If a breaking change shipped without a version, that is the primary root cause.
Did dbt CI catch it? If not, why — was state:modified+ not configured, was contract enforcement off in CI, was the test missing?
Did the comms go out? If not, why — and add to the playbook.
Was the rollback path documented? If not, add a "rollback PR" template.

Worked example — publishing v2 with a 60-day deprecation window

Detailed explanation. Walk through the producer's PR sequence for shipping v2 of dim_customer with a renamed column. Each PR is small and reviewable; the rollout is the sequence of PRs, not one giant change.

Question. Sketch the four PRs the producer ships during the rollout of dim_customer_v2.

Code — PR 1: Publish v2.

# models/marts/customer/dim_customer.yml — PR 1
version: 2
models:
  - name: dim_customer
    config: { access: public, group: customer }
    latest_version: 2
    versions:
      - v: 1
        deprecation_date: 2026-08-15
        config: { contract: { enforced: true } }
        columns:
          - { name: customer_id, data_type: bigint }
          - { name: signup_at,   data_type: timestamp }
          - { name: email,       data_type: varchar }
      - v: 2
        config: { contract: { enforced: true } }
        columns:
          - { name: customer_id,  data_type: bigint }
          - { name: signed_up_at, data_type: timestamp }
          - { name: email,        data_type: varchar }

Code — PR 2: Update one consumer (agg_revenue_by_customer).

-- agg_revenue_by_customer.sql — PR 2
SELECT
    c.customer_id,
    c.signed_up_at,                 -- was: c.signup_at
    sum(o.amount) AS total
FROM {{ ref('dim_customer') }} c    -- now resolves to v2
LEFT JOIN {{ ref('fct_orders') }} o
       ON o.customer_id = c.customer_id
GROUP BY 1, 2;

Code — PR 3: Track remaining v1 consumers.

# tracking script — PR 3 (CI cron job)
dbt list --select +dim_customer_v1 \
  --output name \
  > reports/v1_consumers.txt

# Plus warehouse query log scrape for BI tools
echo "Consumers still on dim_customer_v1:"
cat reports/v1_consumers.txt

Code — PR 4: Sunset v1 after the deprecation date.

# models/marts/customer/dim_customer.yml — PR 4 (after 2026-08-15)
version: 2
models:
  - name: dim_customer
    config: { access: public, group: customer }
    latest_version: 2
    versions:
      - v: 2
        config: { contract: { enforced: true } }
        columns:
          - { name: customer_id,  data_type: bigint }
          - { name: signed_up_at, data_type: timestamp }
          - { name: email,        data_type: varchar }
# v1 block removed; dim_customer_v1.sql file deleted

Step-by-step explanation.

PR 1 (day 0). Add v2, mark v1 deprecated. The PR is tiny: new YAML version block + new SQL file. CI verifies both versions contract-pass. Merged → both versions materialise on next build.
PR 2 (days 1–60). Each consumer team migrates in its own PR. The mart that owns agg_revenue_by_customer updates its SELECT and re-points ref('dim_customer') to the latest version (which is now v2). No coordination with other teams.
PR 3 (continuous). A CI job runs dbt list --select +dim_customer_v1 weekly and posts the shrinking list of remaining consumers to a Slack channel. Producer pings stragglers around day 45.
PR 4 (day 60+). Once dim_customer_v1 has zero remaining consumers, the producer removes the v1 block from YAML, deletes dim_customer_v1.sql, and (eventually, after one more clean build) drops the physical table.

Output.

Phase	Day	Producer	Consumer	Platform
Publish	0	PR 1 merged	—	both tables alive
Overlap	1–60	comms, tracking	migrate at own pace	shrinking v1 traffic
Sunset	60+	PR 4 merged	—	only v2 alive

Rule of thumb. Every rollout is a sequence of small PRs, not one big PR. The producer ships PR 1 and PR 4; consumer teams ship PR 2 themselves; PR 3 is the visibility layer. The sequence is reproducible across every breaking change you ever ship.

Worked example — exposures as the BI/reverse-ETL visibility layer

Detailed explanation. dbt exposures: are declarative YAML blocks that register downstream consumers (BI dashboards, reverse-ETL syncs, ML jobs) as first-class nodes in the dbt graph. They are the bridge between dbt's compile-time visibility and the real world of "who actually uses this model."

Question. Show the YAML for an exposure: registering a Looker dashboard that depends on dim_customer, and explain how it surfaces during the rollout.

Code.

version: 2
exposures:
  - name: customer_360_dashboard
    type: dashboard
    maturity: high
    url: https://looker.internal/dashboards/customer-360
    description: "Marketing-ops Customer 360 dashboard."
    depends_on:
      - ref('dim_customer')           # latest_version
      - ref('fct_orders')
    owner:
      name: Marketing Analytics
      email: marketing-analytics@example.com

Step-by-step explanation.

The exposure: registers the dashboard as a downstream node. dbt list --select +dim_customer now includes exposure:customer_360_dashboard in the output.
During the rollout, the producer runs dbt list --select +dim_customer_v1 and immediately sees if the dashboard is still on v1. The exposure makes the BI tile visible to dbt for the first time.
The owner: block tells the producer who to message — automated comms can ping marketing-analytics@example.com directly.
When the dashboard migrates to v2, the owner updates the exposure to ref('dim_customer', v=2) (or leaves it at ref('dim_customer') to follow latest_version). dbt re-runs the list and the dashboard drops off the v1 consumer roster.

Output. A dbt graph that includes BI dashboards as real nodes, with full ownership metadata. Rollouts can be coordinated end-to-end inside the dbt project — no separate spreadsheet of "what depends on what."

Rule of thumb. Register every important BI dashboard, reverse-ETL sync, and ML job as an exposure:. The five-minute cost per consumer pays back the first time you need to know "who am I about to break?" during a rollout.

Worked example — the contract-broke-prod postmortem template

Detailed explanation. When a contract breaks prod (rare but never zero), the postmortem is the artefact that drives the next playbook iteration. A reusable template keeps every postmortem comparable.

Question. Sketch the template structure for a "contract broke prod" postmortem.

Code — markdown template.

# Postmortem — dim_customer v1→v2 rollout incident

## Summary
[1-2 sentences: what broke, when, who noticed]

## Timeline
- t-7d  Publish v2 + 60-day deprecation_date on v1
- t-2d  Reminder ping in #data-platform
- t=0  v1 dropped (sunset PR merged)
- t+1h Looker tile X errors out; marketing-ops opens ticket
- t+2h Rollback PR re-introduces v1
- t+4h Resolution: tile migrated to v2 by hand; v1 dropped again

## Root cause
[Exact reason: e.g., exposure not registered for Looker tile X;
 weekly tracking script missed it; sunset PR proceeded with one
 unmigrated consumer]

## What worked
- contract.enforced caught two unrelated drift PRs during the overlap window
- Slack pings during weeks 4 and 6 surfaced 3 of 4 stragglers
- Rollback PR (re-add v1 block) restored service in ~30 minutes

## What didn't
- Looker tile X was not registered as an exposure
- Query-log scrape missed it because tile X uses an extract refreshed weekly

## Action items
- [ ] Register every Looker tile that reads marts/* as an exposure
- [ ] Extend rollout playbook with a "scrape extract schedules" step
- [ ] CI: fail (not warn) on compile when an unmigrated consumer references a deprecated version
- [ ] Update on-call runbook with "rollback PR" recipe

Step-by-step explanation.

Summary is the one-paragraph version a busy executive reads.
Timeline documents the events with t-relative times — easy to copy into other tooling.
Root cause names the specific gap (in this case: exposure not registered, weekly query-log scrape missed an extract).
What worked is the positive section — never skip it. Every postmortem needs to celebrate what the system did catch.
What didn't is the gap analysis. Be specific. "Comms were unclear" is not actionable; "Looker tile X was not registered as an exposure" is.
Action items are the playbook updates. Each one feeds back into the rollout checklist for the next release.

Output. A postmortem that teaches the next engineer. The playbook gets one new step. The CI gets one new check. The incident never happens the same way twice.

Rule of thumb. Every "contract broke prod" incident, no matter how small, gets a postmortem with at least one action item. The action item updates the playbook. The playbook updates everyone's defaults. This is how the rollout discipline compounds over years.

dbt interview question on the rollout playbook

A senior interviewer often probes: "Walk me through a 60-day rollout for replacing dim_customer with a breaking-change v2. What happens on day 0, day 30, day 60. Who pings whom. When does CI start failing instead of warning."

Solution Using the four-phase rollout

# 60-day rollout — dim_customer v2

## Day 0 — Publish
- PR 1 merges: v2 alongside v1, contract.enforced on both, deprecation_date = day 60
- Comms: Slack announcement + email to data-platform-consumers@
- Migration guide: pinned in #data-platform
- Office hours: open every Friday for the next 8 weeks
- CI: contract enforcement on, deprecation warnings on (compile warning, no fail yet)

## Days 1-30 — Overlap (warning phase)
- Producer publishes weekly "v1 consumer count" Slack post
- Consumer teams migrate; each ships their own PR re-pointing `ref('dim_customer')`
- CI: continues to warn on `ref('dim_customer', v=1)` references
- Tracking: dbt list + warehouse query log + exposure metadata

## Days 31-60 — Migrate (escalation phase)
- Day 30: producer opens a JIRA ticket for each remaining v1 consumer team
- Day 45: producer pings each ticket owner directly
- Day 55: pre-sunset reminder Slack post + email
- Day 58: CI flip — `--warn-error` enabled for deprecation warnings; PRs that still reference v1 fail
- Day 60: deprecation_date reached

## Day 60+ — Sunset
- Confirm zero v1 traffic via query log for past 48 hours
- PR 4 merges: v1 YAML block removed; SQL file deleted
- After one clean dbt run, drop the physical v1 table
- Post-rollout note in #data-platform: "v1 sunset complete; v2 is now the only version"
- Postmortem only if anything went wrong; otherwise a brief retro

## Rollback paths
- During overlap: revert PR 1 (re-add v1 block if it was removed prematurely)
- After sunset: re-create v1 from the v2 SQL with a one-PR add-back if a critical consumer surfaces late

Step-by-step trace.

Day	Producer action	Consumer state	CI behaviour	Risk if skipped
0	publish v2; deprecation_date set	all on v1	contract pass; v=1 ref compiles cleanly	rollout has no anchor date
7	weekly tracker post	early adopters migrating	unchanged	no visibility into adoption pace
30	open per-team tickets	~50% migrated	warn on v=1 ref	stragglers never feel urgency
45	direct pings to laggards	~80% migrated	warn	last 20% slip past deadline
58	flip CI to fail on v=1 ref	~95% migrated	fail on v=1 ref	sunset breaks last consumers
60	sunset PR; remove v1	100% migrated	only v2 references compile	hard break if any consumer remains
60+	drop physical v1 table	done	done	storage cost only

Output:

Day	What exists in warehouse	What CI does	Risk profile
0	v1 + v2 alive	warn on v=1	low — overlap covers everyone
30	v1 + v2 alive	warn on v=1	low — half migrated
58	v1 + v2 alive	fail on v=1	medium — forces last migrations
60	v1 + v2 alive	fail on v=1	resolved — final cut
60+	only v2 alive	normal	clean steady state

Why this works — concept by concept:

Publish-overlap-migrate-sunset as four discrete phases — each phase has a clear start, a clear end, and a clear set of stakeholder actions. The producer is never "trying to figure out what to do next."
deprecation_date as the social contract — the date is fixed at publish time and visible in YAML. Everyone — producer, consumer, BI owner — sees the same deadline.
CI escalation from warn to fail — the gradual ratchet (warn for 58 days, fail for 2 days, sunset) gives consumers maximum runway with a final forcing function.
Per-team JIRA tickets at day 30 — turns the comms from "broadcast" to "directed." Each laggard team has an owner and a deadline.
Exposures as the BI visibility layer — without them, the query-log scrape is your only signal for non-dbt consumers. With them, every dashboard and reverse-ETL sync is a first-class graph node.
Postmortem only on incident — most rollouts are uneventful. Reserve the postmortem ritual for the times when something genuinely went wrong; otherwise a brief retro is enough.
Cost — producer time: ~4 hours over 60 days. Consumer time: ~30 min per team per migration. Storage cost: one duplicate table for 60 days. Compared to the cost of one broken-Monday incident, this is rounding error.

Data modeling
Topic — event-modeling
Event modeling problems (data modeling)

Practice →

Cheat sheet — dbt contract recipes

Mark a model public. Add config.contract.enforced: true, fill out the columns: block with name + data_type + constraints + description for every column, add config.access: public and config.group:. Ship as v: 1 in a versions: block from day one — saves a YAML refactor when you ship v: 2 later.
Add a column without breaking anyone. Append the new column at the end of the columns: block, ship the YAML + SQL in one PR, and do not bump the version. The change is non-breaking because no existing consumer named the new column.
Rename a column. Ship v: 2 alongside v: 1. Give v1 a 30–90 day deprecation_date. Update one consumer per PR. Drop v1 after the deprecation date and zero remaining traffic. Never edit v1 to rename in place.
Tighten a constraint (loose → strict). Verify the data already satisfies the strict form (run the test once against prod data). Edit the YAML to add not_null / check / unique. Ship as a non-breaking change if the data already satisfies it; otherwise bump the version because the change can fail consumers who insert NULLs.
Loosen a constraint (strict → loose). Treat as breaking. Removing not_null means downstream consumers that rely on the non-null contract may now crash. Ship as v: 2.
FK to a dim on Snowflake / BigQuery / Redshift. Declare the FK in the YAML (informational metadata + catalog + query-planner hint) and add a tests: relationships: test for the value-level audit. Belt and braces.
FK to a dim on Postgres. Declare the FK in YAML; index the referenced column for INSERT performance; add a tests: relationships: test as an audit layer. The DDL enforcement is real; the test is the cross-warehouse guarantee.
Composite primary key. Declare at the model level under constraints: with columns: [a, b]. Add a matching dbt_utils.unique_combination_of_columns test.
Check constraint. Add a check constraint with an expression: (e.g. "price >= 0"). Pair with a dbt_utils.expression_is_true or accepted_values test. The constraint enforces on Postgres; the test enforces on every warehouse.
Enforce at PR time. Configure CI to run dbt build --defer --select state:modified+ against the prod state. Make the contract-failure check required for merge. Use --warn-error to escalate deprecation warnings into failures.
Track who still consumes v1. Run dbt list --select +dim_customer_v1 --output name in a weekly CI cron. Scrape warehouse query logs for non-dbt consumers. Register every BI tile and reverse-ETL sync as a exposures: block.
Sunset v1 cleanly. Confirm zero traffic in the 48 hours before the cut. Ship a single PR that removes the v1 YAML block + deletes the v1 SQL file. Drop the physical table only after one clean build verifies nothing references it.
Roll back a breaking change. During the overlap window: revert the PR that removed v1 (re-add the YAML block and SQL file). After sunset: open a fresh PR that re-introduces v1 with the same shape. Both paths are quick because the v1 SQL is in git history.
Document intent in every column. Add description: to every column. Future-you (and every consumer) will thank you. Description edits are doc-only patches with no version bump.

Frequently asked questions

Are dbt constraints enforced by the warehouse?

It depends on the warehouse and the constraint. NOT NULL is enforced everywhere. PRIMARY KEY and UNIQUE are enforced on Postgres; informational on Snowflake, BigQuery, and Redshift. FOREIGN KEY is enforced on Postgres; informational or unsupported elsewhere. CHECK is enforced on Postgres; unsupported on Snowflake, BigQuery, and Redshift. The contract itself (contract.enforced: true) is enforced at compile time on every warehouse — it's a dbt-side check that the SQL projection matches the YAML declaration, independent of warehouse capabilities. Always pair informational constraints with matching dbt tests (unique, not_null, relationships, accepted_values) — the test is the cross-warehouse audit layer that catches the bugs the warehouse cannot.

Do I need contracts if I already have dbt tests?

Yes — they catch different bug classes. Tests catch value drift after the model materialises: a NULL appearing where it shouldn't, a unique key duplicating, a format violation. They run after the build and require the broken table to already exist in dev / CI. Contracts catch interface drift at compile time: a column renamed, removed, or retyped in the SQL. They run before anything materialises and abort the build immediately, with a domain-specific error message. The two are orthogonal axes — contracts on the columns/types/nullability axis, tests on the values/relationships axis. Mature projects use both: the contract is the first line of defence at PR time, the tests are the post-build audit layer.

When should I bump a model version?

Use SemVer-for-data as the rule. MAJOR (v2, v3): bump for any breaking change — column removed, renamed, retyped to an incompatible type, semantics changed (e.g. "amount in USD" → "amount in local currency"), nullability flipped from non-null to nullable on a column consumers JOIN on. MINOR: do not bump for non-breaking additions — a new column appended at the end, a new constraint that the data already satisfies, a new test. PATCH: do not bump for doc-only edits (descriptions, comments). The pragmatic heuristic: if any consumer's existing SELECT, WHERE, or JOIN could behave differently, bump the version. If consumers are unaffected, edit in place.

Can I have contracts on incremental models?

Yes — contract.enforced: true works with materialized: incremental. dbt validates the contract on every run: at compile (the SELECT must project the contracted columns) and at the schema check that starts every incremental run (the existing target table must match). Combine with on_schema_change: fail so dbt aborts instead of silently appending new columns on schema drift. On a --full-refresh build, dbt drops and recreates the table with the full DDL (including constraints, where supported). On a normal incremental run, dbt validates the schema check, runs the delta SELECT, validates its projection against the contract, then INSERTs / MERGEs. The contract is enforced at exactly the points where drift could leak in.

How do I get foreign keys in Snowflake or BigQuery?

You can declare them in the contract YAML (type: foreign_key with an expression: referencing the target table and column), but the warehouse will not enforce them at write time — Snowflake records them as informational metadata (visible in INFORMATION_SCHEMA, useful to the query planner), and BigQuery supports FOREIGN KEY ... NOT ENFORCED as a query-planner hint only. For actual value-level FK enforcement on those warehouses, pair the declared constraint with a tests: relationships: test. The test runs SELECT count(*) FROM child WHERE child.fk NOT IN (SELECT pk FROM parent) and asserts zero — exactly what an enforcing FK would block, but at audit time instead of write time. This is the standard "belt and braces" pattern: the constraint declares intent, the test verifies the data.

What's the difference between contracts and dbt-expectations?

dbt contracts are part of dbt-Core (since 1.5). They validate the shape of a model — column names, data types, constraint declarations — at compile time, and translate constraints to warehouse DDL where supported. They are the interface-locking layer. dbt-expectations is a community package (modelled on Python's great_expectations library) that ships a large catalog of value-level tests — distribution tests, statistical tests, regex tests, percent-NULL tests, etc. They run post-build like any dbt test and audit values. The two are complementary: contracts lock the shape; dbt-expectations enriches the value-level audit beyond the built-in unique / not_null / accepted_values / relationships. Mature projects use contracts on every public model and dbt-expectations on top of dbt tests wherever statistical or distribution checks add signal.

Practice on PipeCode

Drill the data modelling practice library → for the schema-design, contract-readiness, and SCD interview surface.
Rehearse on dimensional modelling problems → for star-schema fact-and-dim contract design.
Tighten the schema-evolution muscles with slowly-changing-data drills → — versioning a public dim is the same problem class.
Stack the cardinality library → for "is this a 1:1, 1:N, or N:N relationship" probes that drive PK / FK / unique-constraint design.
Sharpen event-modelling problems → for the immutable-table contract patterns that show up in fact-table and event-source interview questions.
Layer the design problems library → for the broader "design this warehouse layer" interview surface.
For the broader DE surface, read top data engineering interview questions →.
Stack the prerequisites with the only 5 skills you need to become a data engineer →.
For long-form schema craft, work through data modelling for DE interviews →.
For the broader ETL design surface, take the ETL system design course →.

Pipecode.ai is Leetcode for Data Engineering — every contract recipe, constraint pattern, and rollout phase above ships with hands-on practice rooms where you design the YAML block, defend the version bump, and walk the four-phase deprecation playbook against real graded interview-style scenarios. PipeCode pairs every reading with 450+ DE-focused problems and a real-time scoring engine, so you never have to wonder whether your `dim_customer` v2 plan actually survives contact with a Looker dashboard, a HubSpot sync, and a Snowflake share at the same time.

Practice data modeling now →
Dimensional modelling drills →

dbt Model Contracts, Constraints & Versioning: Production Patterns

Gowtham Potureddi — Mon, 15 Jun 2026 13:29:33 +0000

On this page

Why dbt models need contracts in production
Anatomy of a dbt model contract
Constraints — primary key, foreign key, not null, check
Versioning strategy for public models
Rollout and deprecation playbook
Cheat sheet — dbt contract recipes
Frequently asked questions
Practice on PipeCode

1. Why dbt models need contracts in production

Contracts catch the kind of bug dbt tests cannot — the interface bug, not the value bug

The three places interface bugs hide.

Silent column renames. Someone renames customer_email to email_address in stg_customers.sql. Every test still passes (the new column has the same values), every dashboard breaks at midnight when it tries to read the old name. No PR reviewer caught it because the column was added and the old one was removed in the same commit — the diff just looked like "edited a SELECT clause."
Data type drift. A staging model exposed order_total as numeric(18,2). Someone refactors and the new SQL emits numeric(38,18). The dashboard still works in dev (Postgres is loose about precision), then a Tableau live connection on Redshift fails on the first row because the consumer expected the old precision.
Nullability flips. dim_customer.signup_at was always non-null because the upstream model filtered out incomplete rows. A refactor removes the filter for performance. Now signup_at is sometimes NULL — downstream reverse-ETL crashes on the first NULL it sees.

The dbt-Core 1.5+ feature timeline.

dbt 1.5 (April 2023) shipped model contracts (contract.enforced: true) and constraints (the four kinds: not_null, unique, primary_key, foreign_key, plus check). This is the moment dbt projects gained a way to declare the public shape of a model and have the build fail if the shape drifts.
dbt 1.5 also shipped model versions — the versions: block, latest_version, deprecation_date, and ref('model', v=1) cross-version references. Together with contracts these three features form the "stable interface" toolkit.
dbt 1.6+ (July 2023 onwards) added access: modifiers (private, protected, public) and groups — so a model can be marked private to a single group of authors and ref() from outside that group fails to compile.
dbt 1.7+ (Q4 2023 onwards) added the unit testing framework — orthogonal to contracts but synergistic, because unit tests assert the rows that a contracted model produces.

Where contracts fit between tests, constraints, observability.

dbt tests. Run after the model materialises; they re-query the table and assert row-level facts (unique, not_null, accepted_values, custom singular tests). They are row-shaped assertions.
dbt contracts. Run before the model materialises; they assert that the SELECT's projected columns match the declared columns: block in YAML — names, types, and constraints. They are interface-shaped assertions that fail fast in PR CI.
dbt constraints. Translate the YAML declaration into DDL where the warehouse supports it; otherwise they remain informational metadata. They are contract reinforcement — when paired with a warehouse that enforces them, they fail the load instead of poisoning a downstream join.
Data observability platforms (Monte Carlo, Bigeye, Lightup). Detect drift in production after the fact — useful, but reactive. Contracts make the same drift a PR-time failure, which is two orders of magnitude cheaper to fix.

The 2026 reality.

Contracts are now table-stakes for public models. Any model ref()-ed from outside its owning group, exported to reverse-ETL, or surfaced in BI should have contract.enforced: true.
Constraints are warehouse-dependent. Postgres and Redshift (mostly) enforce them; Snowflake and BigQuery treat most as informational. dbt translates declarations to DDL in both cases, but the runtime behaviour differs.
Versions are how dbt does SemVer. Breaking changes get a version bump (v2, v3); non-breaking additions stay on the same version. deprecation_date and latest_version give you a 30–90 day overlap window to migrate consumers.

Worked example — the silent column rename that broke Monday

Question. Show the dbt YAML diff for adding contract.enforced: true to dim_customer and demonstrate how the same rename would fail in CI instead.

Input — current models/marts/customer/dim_customer.yml.

field	value
name	dim_customer
materialized	table
columns	customer_id, signup_at, email
tests	unique on customer_id, not_null on signup_at

Code.

# models/marts/customer/dim_customer.yml
version: 2

models:
  - name: dim_customer
    config:
      materialized: table
      contract:
        enforced: true        # <- the upgrade
    columns:
      - name: customer_id
        data_type: bigint
        constraints:
          - type: not_null
          - type: primary_key
      - name: signup_at        # <- the contract anchor
        data_type: timestamp
        constraints:
          - type: not_null
      - name: email
        data_type: varchar

-- models/marts/customer/dim_customer.sql AFTER the rename
SELECT
    customer_id,
    signed_up_at,             -- renamed from signup_at
    email
FROM {{ ref('stg_customer') }};

Step-by-step explanation.

dbt 1.5+ compiles dim_customer.sql against the YAML contract. It runs the SELECT once in a transaction (or as a dry-run on warehouses that support it) and inspects the returned column metadata.
The contract declares signup_at as a column. The SELECT returns signed_up_at instead. dbt diffs the two sets and emits a contract violation.
The CI job — dbt build --select state:modified+ — fails. The PR cannot be merged. The "Monday morning incident" became a "Friday afternoon code-review comment."
The author either rolls back the rename (cheap) or coordinates a versioning bump (dim_customer_v2) so consumers can migrate on their own schedule.

Output.

Compilation Error in model dim_customer
  This model has an enforced contract that failed.
  Please ensure the name, data_type, and number of columns in your contract
  match the columns in your model's definition.

  | column_name      | definition_type | contract_type | mismatch_reason     |
  | ---------------- | --------------- | ------------- | ------------------- |
  | signed_up_at     | TIMESTAMP       |               | missing in contract |
  | signup_at        |                 | TIMESTAMP     | missing in definition|

Worked example — tests catch a value bug, contracts catch an interface bug

Question. A staging model stg_orders accidentally drops the order_id column in a refactor. Compare what happens with only dbt tests vs with a contract.

Input — the broken refactor.

-- BEFORE refactor (correct)
SELECT
    order_id,
    customer_id,
    amount
FROM {{ source('raw', 'orders') }};

-- AFTER refactor (accidentally drops order_id)
SELECT
    customer_id,
    amount
FROM {{ source('raw', 'orders') }};

Code — tests-only YAML.

version: 2
models:
  - name: stg_orders
    columns:
      - name: order_id
        tests: [unique, not_null]
      - name: customer_id
        tests: [not_null]

Code — tests + contract YAML.

version: 2
models:
  - name: stg_orders
    config:
      contract:
        enforced: true
    columns:
      - name: order_id
        data_type: bigint
        constraints: [{ type: not_null }, { type: unique }]
      - name: customer_id
        data_type: bigint
        constraints: [{ type: not_null }]
      - name: amount
        data_type: numeric

Step-by-step explanation.

Tests only. dbt build runs the broken SELECT. The model materialises successfully (it just has two columns now). Then dbt tries to test order_id — and gets a "column does not exist" error from the warehouse. The test "fails" but with a runtime database error, not a contract-style error. Worse: the table is already broken in the dev schema by the time the test runs.
Tests + contract. dbt build compiles the model against the contract before running it. The contract declares three columns; the SELECT only projects two. The compile fails with a clear contract-violation message naming the missing column. Nothing materialises; nothing breaks.
The contract catches the bug two phases earlier in the dbt graph (compile, not test) and emits a domain-specific error ("contract violation: missing column order_id") instead of a warehouse error.

Output.

Strategy	Detected at	Error type	Side effects
Tests only	After build, during test	warehouse "column not found"	broken table left in dev schema
Tests + contract	At compile, before build	dbt "contract violation"	nothing materialised

Worked example — contracts on incremental models

Question. Show the YAML for a contracted incremental fact model fct_orders and explain when contract enforcement runs.

Code.

version: 2
models:
  - name: fct_orders
    config:
      materialized: incremental
      unique_key: order_id
      on_schema_change: fail   # belt-and-braces: explicit schema-change policy
      contract:
        enforced: true
    columns:
      - name: order_id
        data_type: bigint
        constraints: [{ type: not_null }, { type: primary_key }]
      - name: customer_id
        data_type: bigint
        constraints:
          - type: not_null
          - type: foreign_key
            expression: "{{ ref('dim_customer') }} (customer_id)"
      - name: order_ts
        data_type: timestamp
        constraints: [{ type: not_null }]
      - name: amount
        data_type: numeric
        constraints: [{ type: not_null }]

Step-by-step explanation.

Full refresh. dbt build --full-refresh --select fct_orders runs the SELECT, validates the projected columns against the contract, then drops-and-recreates the table with the declared DDL (including constraints, where supported). The contract is checked once, decisively.
Incremental run. dbt build --select fct_orders (no --full-refresh) inspects the existing target table and compares its column set to the contract. If they match, dbt runs the incremental delta SELECT, validates its projection against the contract, then INSERTs / MERGEs into the target.
on_schema_change: fail is critical when contracts are on. Without it, dbt's default incremental behaviour might append a new column silently — which would still pass the contract check (the new column is in both the SELECT and the table) but would drift the contract's declared shape over time. Fail-on-change keeps the table strictly in sync with the YAML.

dbt interview question on the contracts vs tests axis

Solution Using the contracts-tests matrix

# models/marts/customer/dim_customer.yml
version: 2
models:
  - name: dim_customer
    config:
      materialized: table
      contract:
        enforced: true
    columns:
      - name: customer_id
        data_type: bigint
        constraints: [{ type: not_null }, { type: primary_key }]
        tests: [unique, not_null]
      - name: email
        data_type: varchar
        tests:
          - not_null
          - dbt_utils.expression_is_true:
              expression: "email like '%@%'"

Step-by-step trace.

Bug scenario	Tests-only outcome	Contract-only outcome	Both outcome
`customer_id` renamed to `cust_id` in SQL	runtime warehouse error during test	PR fails at compile	PR fails at compile
`customer_id` type changed `bigint` → `string`	tests still pass (values unique, non-null)	PR fails at compile	PR fails at compile
`email` column suddenly contains NULLs	not_null test fails post-build	contract still passes (column exists)	not_null test fails post-build
`email` column missing the `@`	expression test fails post-build	contract still passes	expression test fails post-build

Output:

Bug class	Catch with	Catches before	Detection cost
Renamed column	contract	model materialises	low (compile-time)
Type drift	contract	model materialises	low (compile-time)
NULL creeping in	tests	downstream consumer	medium (post-build)
Format violation	tests	downstream consumer	medium (post-build)

Why this works — concept by concept:

contract.enforced as a compile-time gate — runs before any DDL is issued. dbt compiles the SELECT, inspects the projected columns via the warehouse's metadata (or a dry-run plan), and diffs them against the YAML. Mismatches abort the build.
dbt tests as a post-build sentinel — run after the model materialises. They re-query the table and assert row-level facts. Cheap to write, but they catch issues after the broken table exists in dev / CI.
The two are orthogonal axes — contracts cover the columns-types-nullability axis, tests cover the values-and-relationships axis. Mature projects use both, with the contract as the first line of defence and the tests as the audit layer.
on_schema_change as the third leg — for incremental models, the contract pins the current shape; on_schema_change: fail ensures the shape cannot drift silently between contract edits. Without it, the table can grow extra columns invisibly.
Cost — contracts add O(columns) compile-time work per build (negligible); tests add one SELECT per test per build. Both are dominated by the actual model build time on any realistic dataset.

Data modeling
Topic — design
Design problems (data modeling)

Practice →

2. Anatomy of a dbt model contract

`contract.enforced: true` plus a filled-out columns block is the entire vocabulary — but every field has a precise job

The five mandatory pieces.

config.contract.enforced: true — the master switch. Without it, the rest of the YAML is documentation. With it, dbt diffs the SELECT against the columns block at compile time and aborts on mismatch.
columns: - name: ... — every column the model projects must appear in the columns block, by name, in any order. Extra YAML columns not in the SELECT, or extra SELECT columns not in YAML, both fail the contract.
data_type: — the warehouse-canonical type (bigint, varchar, numeric(18,2), timestamp, boolean). dbt normalises common synonyms (int8 → bigint on Postgres) but it pays to use the exact word the warehouse echoes.
constraints: — a list of constraint declarations. Each has a type: (not_null, unique, primary_key, foreign_key, check) and optional fields (name:, expression:, columns: for composite, warn_unenforced: / warn_unsupported:).
description: — free-form prose; surfaced in dbt docs and the catalog. Not strictly enforced but is the single best place to document the semantic intent of the column for downstream consumers.

Compile-time vs run-time enforcement.

Compile-time (the default). dbt asks the warehouse to plan the SELECT without running it, inspects the projected columns from the plan metadata, and diffs them against the contract. Cheap and fast — milliseconds per model. Fails the PR in CI before any data moves.
Run-time. On warehouses that enforce constraints (Postgres, Redshift for some, Databricks Unity Catalog), the CREATE TABLE statement carries the constraints as actual DDL. Inserting a NULL into a not_null column raises a database error at write time. This is in addition to the compile-time contract check, not instead of it.
The handshake. Contracts give you the compile-time interface guarantee; constraints (on enforcing warehouses) give you the run-time value guarantee. They overlap on names like not_null but cover different failure modes.

A "shape" assertion, not a "value" assertion.

The contract checks that order_id is declared as bigint and the SELECT produces a bigint column called order_id. It does not check that any particular row's order_id is non-null.
Adding constraints: [{ type: not_null }] to the contract is the bridge — it asks dbt to also attempt warehouse-level enforcement of "no NULL values in this column." On Postgres that becomes a NOT NULL DDL clause. On Snowflake it becomes informational metadata (the warehouse does not enforce).
For the value-level audit you still want tests: [not_null] — that runs a SELECT COUNT(*) FROM t WHERE col IS NULL after the build and asserts zero.

Interactions with materialisation.

materialized: table — full DDL re-created on each build. Constraints are emitted as part of the CREATE TABLE. Contracts checked at compile.
materialized: view — view definition checked at compile. Constraints in the YAML are documentation only because most warehouses do not attach constraints to views.
materialized: incremental — full DDL on --full-refresh; incremental INSERT / MERGE on normal runs. Contracts checked on every run (compile-time). Combine with on_schema_change: fail.
materialized: ephemeral — no DDL; the model is inlined as a CTE in consumers. Contracts cannot apply (no projected table). dbt warns if you try.

Common interview probes on contract anatomy.

"What happens if the SELECT projects an extra column not in the contract?" — contract violation, build aborts.
"What happens if YAML declares an extra column not in the SELECT?" — same — contract violation.
"Is column order part of the contract?" — no. dbt diffs the set of columns, not the ordering.
"Does the contract validate types end-to-end?" — yes, but the matching is dialect-aware (int and bigint are not interchangeable; numeric and numeric(18,2) can differ depending on warehouse).

Worked example — turning an unmodelled `dim_customer` into a public contracted model

Question. Promote dim_customer.sql from an unmodelled table to a contracted, constrained, public-ready model. Show the YAML diff and explain each line.

Input — current YAML.

version: 2
models:
  - name: dim_customer

Code — promoted YAML.

version: 2
models:
  - name: dim_customer
    description: >
      One row per customer. Source of truth for downstream marts, BI tiles,
      and reverse-ETL syncs to HubSpot. Schema is public — bump the version
      for any breaking change.
    config:
      materialized: table
      contract:
        enforced: true
      access: public
      group: customer
    columns:
      - name: customer_id
        data_type: bigint
        description: "Surrogate key; stable across all loads."
        constraints:
          - type: not_null
          - type: primary_key
        tests: [unique, not_null]
      - name: email
        data_type: varchar
        description: "Primary contact email; lowercased, trimmed."
        constraints:
          - type: not_null
          - type: unique
          - type: check
            expression: "email like '%@%.%'"
        tests:
          - not_null
          - unique
      - name: signup_at
        data_type: timestamp
        description: "First successful account creation (UTC)."
        constraints: [{ type: not_null }]
        tests: [not_null]
      - name: tier
        data_type: varchar
        description: "Loyalty tier — one of {bronze, silver, gold}."
        constraints:
          - type: check
            expression: "tier in ('bronze','silver','gold')"
        tests:
          - accepted_values:
              values: [bronze, silver, gold]

Step-by-step explanation.

description: is now mandatory in spirit — it is the first thing a consumer reads in dbt docs. Keep it short, concrete, and oriented toward consumers, not authors.
materialized: table makes the constraint DDL meaningful. On Postgres the table will be created with customer_id bigint PRIMARY KEY NOT NULL, email varchar UNIQUE NOT NULL, ....
contract.enforced: true is the master switch. The first time you dbt build this model, the SELECT must already project exactly {customer_id, email, signup_at, tier} with matching types — otherwise the build fails.
access: public and group: customer declare the visibility of the model. Combined with the contract, this is dbt's full "public API" pattern: a ref('dim_customer') from any other group will be allowed; from within the same group it is free. A private model can ignore most of this YAML.
Each column has both contract constraints: and dbt tests:. The constraints are compile-time + DDL-time guards (the warehouse may enforce them); the tests are post-build value audits. The redundancy is the point — belt and braces.

Output. First build on Postgres emits:

CREATE TABLE analytics.dim_customer (
    customer_id bigint NOT NULL,
    email       varchar NOT NULL,
    signup_at   timestamp NOT NULL,
    tier        varchar,
    PRIMARY KEY (customer_id),
    UNIQUE (email),
    CHECK (email like '%@%.%'),
    CHECK (tier in ('bronze','silver','gold'))
);

Worked example — what dbt does to the warehouse on first build

Question. Given the contracted dim_customer from above, write out the literal CREATE TABLE statements dbt emits on Postgres, Snowflake, BigQuery, and Redshift.

Code — Postgres (full enforcement).

CREATE TABLE analytics.dim_customer (
    customer_id bigint NOT NULL,
    email       varchar NOT NULL,
    signup_at   timestamp NOT NULL,
    tier        varchar,
    CONSTRAINT dim_customer_pk PRIMARY KEY (customer_id),
    CONSTRAINT dim_customer_email_uk UNIQUE (email),
    CONSTRAINT dim_customer_email_chk CHECK (email like '%@%.%'),
    CONSTRAINT dim_customer_tier_chk  CHECK (tier in ('bronze','silver','gold'))
);

Code — Snowflake (mostly informational).

CREATE OR REPLACE TABLE analytics.dim_customer (
    customer_id bigint NOT NULL,
    email       varchar NOT NULL,
    signup_at   timestamp_ntz NOT NULL,
    tier        varchar,
    CONSTRAINT dim_customer_pk PRIMARY KEY (customer_id) NOT ENFORCED,
    CONSTRAINT dim_customer_email_uk UNIQUE (email) NOT ENFORCED
);
-- CHECK and FK in Snowflake are not supported / informational; dbt logs a warning

Code — BigQuery (only NOT NULL + primary-key metadata).

CREATE OR REPLACE TABLE `proj.analytics.dim_customer` (
    customer_id INT64 NOT NULL,
    email       STRING NOT NULL,
    signup_at   TIMESTAMP NOT NULL,
    tier        STRING,
    PRIMARY KEY (customer_id) NOT ENFORCED
);
-- UNIQUE / CHECK not supported as DDL; dbt logs a warning

Code — Redshift (NOT NULL enforced, others informational).

CREATE TABLE analytics.dim_customer (
    customer_id bigint NOT NULL,
    email       varchar NOT NULL,
    signup_at   timestamp NOT NULL,
    tier        varchar,
    PRIMARY KEY (customer_id),   -- informational only
    UNIQUE (email)               -- informational only
);

Step-by-step explanation.

Postgres is the only of the four to enforce every declared constraint at the database level. Inserting a NULL into signup_at, a duplicate email, or an invalid tier value all raise an error at write time.
Snowflake enforces NOT NULL and that is it. PRIMARY KEY and UNIQUE are declared as NOT ENFORCED for documentation / catalog / query-planner-hint purposes. CHECK and FOREIGN KEY are not supported as DDL at all — dbt logs a warning and drops them.
BigQuery enforces NOT NULL. As of recent versions it supports PRIMARY KEY ... NOT ENFORCED and FOREIGN KEY ... NOT ENFORCED for query-planner hints only. UNIQUE and CHECK are not supported.
Redshift enforces NOT NULL. PRIMARY KEY, UNIQUE, and FOREIGN KEY are accepted syntactically but are informational only (the optimizer may use them as hints; insertions are not blocked).
The contract itself is a compile-time guarantee on all four — dbt diffs the SELECT against the YAML regardless of warehouse. The runtime enforcement is what differs.

Output.

Warehouse	NOT NULL	UNIQUE	PRIMARY KEY	FOREIGN KEY	CHECK
Postgres	enforced	enforced	enforced	enforced	enforced
Snowflake	enforced	informational	informational	not supported	not supported
BigQuery	enforced	not supported	informational	informational	not supported
Redshift	enforced	informational	informational	informational	not supported

Worked example — a contracted view (and why most teams use tables)

Question. Show a contracted view for vw_active_customers (filters dim_customer to non-deleted rows) and explain what the contract does and does not guarantee.

Code.

version: 2
models:
  - name: vw_active_customers
    config:
      materialized: view
      contract:
        enforced: true
    columns:
      - name: customer_id
        data_type: bigint
        constraints: [{ type: not_null }]
      - name: email
        data_type: varchar
        constraints: [{ type: not_null }]
      - name: tier
        data_type: varchar

-- models/marts/customer/vw_active_customers.sql
SELECT
    customer_id,
    email,
    tier
FROM {{ ref('dim_customer') }}
WHERE deleted_at IS NULL;

Step-by-step explanation.

dbt compiles the view, inspects the SELECT's projection, and diffs against the YAML — same as for a table.
dbt then issues CREATE OR REPLACE VIEW analytics.vw_active_customers AS SELECT customer_id, email, tier FROM analytics.dim_customer WHERE deleted_at IS NULL;. No constraints attach.
The contract guarantees: at compile time, the SELECT projects exactly {customer_id, email, tier} with matching types. After deploy, queries against the view always see those three columns with those types.
The contract does not guarantee NULL-safety at the warehouse level. If dim_customer.email happens to contain a NULL row that passes the deleted_at IS NULL filter, the view will return it. The contract only documents the intent; you still need a test (tests: [not_null]) to audit value-level facts.

dbt interview question on contract anatomy

Solution Using the full public-model pattern

version: 2
models:
  - name: dim_product
    description: >
      One row per product. Public — every breaking schema change ships
      as a new version (v2, v3) with a 60-day deprecation window.
    config:
      materialized: table
      contract:
        enforced: true
      access: public
      group: product
    columns:
      - name: product_id
        data_type: bigint
        description: "Stable surrogate key."
        constraints: [{ type: not_null }, { type: primary_key }]
        tests: [unique, not_null]
      - name: sku
        data_type: varchar
        description: "Vendor SKU — uppercase, no whitespace."
        constraints: [{ type: not_null }, { type: unique }]
        tests: [unique, not_null]
      - name: category_id
        data_type: bigint
        description: "FK to dim_category.category_id."
        constraints:
          - type: not_null
          - type: foreign_key
            expression: "{{ ref('dim_category') }} (category_id)"
        tests:
          - not_null
          - relationships:
              to: ref('dim_category')
              field: category_id
      - name: price
        data_type: numeric(18,2)
        description: "List price in USD."
        constraints:
          - type: not_null
          - type: check
            expression: "price >= 0"
        tests:
          - not_null
          - dbt_utils.expression_is_true:
              expression: "price >= 0"

Step-by-step trace.

YAML field	What it defends against	Catches at
`contract.enforced: true`	renamed / removed / retyped columns in SQL	compile
`access: public` + `group:`	accidental `ref()` from outside the owning group on private models	compile
`data_type:` on every column	type drift (`bigint` → `int`, `numeric(18,2)` → `numeric(38,18)`)	compile
`not_null` constraint	NULL insertion (Postgres / Redshift / Snowflake / BigQuery)	run-time
`primary_key` constraint	duplicate keys (Postgres only); query-plan hint elsewhere	run-time / planner
`foreign_key` constraint	orphan rows (Postgres only); query-plan hint elsewhere	run-time / planner
`check` constraint	invalid values (Postgres only); informational elsewhere	run-time
`tests:` block	actual value drift in production after build	post-build

Output:

Layer	Guarantees	Cost
Contract	Column set, names, types	Compile-time (ms per model)
Constraints (Postgres)	NULL-safety, uniqueness, referential integrity, check	DDL + insertion overhead
Constraints (Snowflake / BigQuery / Redshift)	NULL-safety only; rest are catalog metadata	Negligible
Tests	Value-level audits	One SELECT per test per build

Why this works — concept by concept:

contract.enforced as the interface lock — the YAML becomes the source of truth for "what columns does this model expose," and dbt fails any build that drifts from it. Consumers can refactor with confidence.
access: public + group — visibility metadata. Private models can be refactored freely within their group; public models are the ones that need versions when the shape changes. This is dbt's analog to "public API vs internal helper."
Constraints as the warehouse-side aspiration — the YAML declares the constraint; the warehouse may or may not enforce it. Either way, the declaration shows up in dbt docs and the catalog, making the intent discoverable.
Tests as the audit — every constraint should have a matching test, because the test runs identically on all warehouses. Tests are the dialect-independent way to guarantee value semantics.
Description as the consumer doc — surfaced in dbt docs and in IDE tooltips. Costs five seconds; saves the consumer from a Slack ping every time they want to know "is signup_at UTC or local?"
Cost — compile-time overhead is negligible (milliseconds per model). The biggest "cost" is the discipline to keep the YAML in sync with the SQL — which is exactly the discipline you want.

Data modeling
Topic — dimensional-modeling
Dimensional modeling problems (data modeling)

Practice →

3. Constraints — primary key, foreign key, not null, check

Five constraint kinds, five very different stories about whether the warehouse actually enforces them

The five constraint kinds.

not_null — "no row of this column may be NULL." Every major warehouse enforces this at INSERT time. The cheapest, most universal, and most useful constraint.
unique — "no two rows share this value." Postgres enforces; Snowflake / BigQuery / Redshift declare informationally.
primary_key — "this column (or set) is the row identity." Implies not_null + unique. Postgres enforces both halves; the others treat as informational metadata that the query planner may consult.
foreign_key — "this column references a column in another table." Postgres enforces (subject to indexes); Snowflake / BigQuery declare informationally; Redshift declares informationally.
check — "this column satisfies a boolean expression." Postgres enforces. Snowflake / BigQuery / Redshift do not support CHECK as DDL — dbt logs a warning and skips.

Single-column vs composite constraints.

Single-column. Declare inside the column's constraints: list. Most natural for not_null / unique / primary_key.
Composite. Declare at the model level under model-level constraints:. Example: a composite primary key on (order_id, line_no). Each constraint declaration includes a columns: list naming the participating columns.

Informational vs enforced — the practical impact.

Enforced. The warehouse refuses INSERTs / MERGEs that would violate the constraint. Bugs surface at write time, often immediately, with a clear error from the database.
Informational. The constraint is recorded in the warehouse catalog but not checked at write time. The query planner may use it to rewrite joins (e.g. eliminate a DISTINCT when joining on a primary key). Bugs surface downstream, often hours or days later.
The practical rule. On informational warehouses (Snowflake / BigQuery / Redshift), the constraint is documentation + query-planner hint. You still need a matching dbt test to actually audit the values.

Constraint + test — belt and braces, not duplicated work.

The constraint tells the warehouse what the model promises. On enforcing warehouses, it is real. On informational ones, it is a hint.
The test tells dbt (and the CI / scheduler) to run a value-level audit after every build. It works identically on every warehouse and surfaces silent drift.
For mature projects, ship both. The constraint is the declaration of intent; the test is the verification.

Foreign-key gotchas in warehouses with no FK enforcement.

Snowflake allows FOREIGN KEY ... NOT ENFORCED syntactically (some versions). dbt emits the DDL where possible; otherwise warns and drops.
BigQuery supports FOREIGN KEY ... NOT ENFORCED for query-planner hints (since late 2023). The constraint is metadata only.
Redshift accepts FOREIGN KEY syntactically; the optimiser uses it as a join hint but does not enforce.
Postgres is the outlier — FKs are real, but they require an index on the referenced column (otherwise INSERT performance suffers).
The pragmatic FK pattern on non-Postgres warehouses. Declare the FK in YAML for documentation and catalog clarity, then add a dbt relationships test for the actual audit:

- name: customer_id
  constraints:
    - type: foreign_key
      expression: "{{ ref('dim_customer') }} (customer_id)"
  tests:
    - relationships:
        to: ref('dim_customer')
        field: customer_id

Common interview probes on constraints.

"On Snowflake, does declaring a PRIMARY KEY actually prevent duplicates?" — no; it is informational. Add a unique test.
"What is the difference between primary_key and unique + not_null?" — semantically identical (PK = unique + not null); syntactically PK is one declaration, the catalog distinguishes them, and the query planner treats PK as "the canonical row identity."
"When would you skip declaring an FK?" — when the referenced table is enormous and the FK overhead would matter (rare in analytics warehouses; common in OLTP). In analytics, declare the FK informationally on every column that joins to a dimension.
"Why do constraints not duplicate tests?" — they cover different failure modes. Constraints prevent bad writes (where supported); tests audit existing data post-build. You need both.

Worked example — a contracted star-schema fact with FKs to dims

Question. Write the YAML for a contracted, constrained fct_orders model with three FKs and a composite PK.

Input — model SQL.

-- models/marts/sales/fct_orders.sql
SELECT
    order_id,
    line_no,
    customer_id,
    product_id,
    date_id,
    quantity,
    unit_price,
    quantity * unit_price AS line_amount
FROM {{ ref('int_orders') }};

Code — YAML with composite PK and three FKs.

version: 2
models:
  - name: fct_orders
    description: "Fact table at the order-line grain."
    config:
      materialized: table
      contract:
        enforced: true
      access: public
      group: sales

    constraints:
      - type: primary_key
        columns: [order_id, line_no]
      - type: foreign_key
        columns: [customer_id]
        expression: "{{ ref('dim_customer') }} (customer_id)"
      - type: foreign_key
        columns: [product_id]
        expression: "{{ ref('dim_product') }} (product_id)"
      - type: foreign_key
        columns: [date_id]
        expression: "{{ ref('dim_date') }} (date_id)"

    columns:
      - name: order_id
        data_type: bigint
        constraints: [{ type: not_null }]
      - name: line_no
        data_type: integer
        constraints:
          - type: not_null
          - type: check
            expression: "line_no >= 1"
      - name: customer_id
        data_type: bigint
        constraints: [{ type: not_null }]
        tests:
          - relationships:
              to: ref('dim_customer')
              field: customer_id
      - name: product_id
        data_type: bigint
        constraints: [{ type: not_null }]
        tests:
          - relationships:
              to: ref('dim_product')
              field: product_id
      - name: date_id
        data_type: integer
        constraints: [{ type: not_null }]
      - name: quantity
        data_type: integer
        constraints:
          - type: not_null
          - type: check
            expression: "quantity > 0"
      - name: unit_price
        data_type: numeric(18,2)
        constraints:
          - type: not_null
          - type: check
            expression: "unit_price >= 0"
      - name: line_amount
        data_type: numeric(18,2)
        constraints: [{ type: not_null }]

Step-by-step explanation.

The composite primary_key is declared at the model level — constraints: directly under the model, with a columns: list naming the two participating columns. Composite PKs cannot be declared inside a single column's block because no single column owns the constraint.
The three foreign_key constraints are also declared at the model level (one per FK). Each names the local columns: and the expression: referencing the target table and column.
Column-level constraints: carry not_null and check for each column. Note the check (quantity > 0) and check (unit_price >= 0) — these are quality guarantees that surface as DDL on Postgres and as informational hints elsewhere.
The relationships tests on customer_id and product_id are the dbt-side audit that catches orphans on any warehouse, regardless of FK enforcement.
On Postgres the DDL is fully enforced: any INSERT with a missing FK target, duplicate (order_id, line_no), or quantity <= 0 raises an error. On Snowflake / BigQuery / Redshift the constraints are informational; the not_null portion still enforces, but PK / FK / CHECK do not.

Worked example — a check constraint that catches a tier typo

Question. Show the YAML for the tier column with both a check constraint and an accepted_values test, and explain when each fires.

Code.

- name: tier
  data_type: varchar
  description: "Loyalty tier — one of {bronze, silver, gold}."
  constraints:
    - type: not_null
    - type: check
      expression: "tier in ('bronze','silver','gold')"
  tests:
    - not_null
    - accepted_values:
        values: [bronze, silver, gold]

Step-by-step explanation.

On Postgres. The CREATE TABLE includes tier varchar NOT NULL, CONSTRAINT dim_customer_tier_chk CHECK (tier in ('bronze','silver','gold')). Any INSERT with tier = 'platinum' raises a database error and aborts the transaction.
On Snowflake. The CHECK is not supported as DDL; dbt emits a warning ("CHECK constraint is not supported on Snowflake — skipping") and the constraint becomes documentation only.
The accepted_values test. After build, dbt runs SELECT COUNT(*) FROM dim_customer WHERE tier NOT IN ('bronze','silver','gold') and asserts the count is zero. This works identically on every warehouse.
The combined effect: Postgres catches the bad row at write time; Snowflake catches it post-build. Either way, the bad row never makes it to production — but the latency-to-detection differs by minutes (Postgres) vs the test phase (Snowflake).

Worked example — composite unique on a deduplicated staging model

Question. Write the YAML composite unique for (source_system, source_order_id) on stg_orders.

Code.

version: 2
models:
  - name: stg_orders
    config:
      materialized: table
      contract:
        enforced: true

    constraints:
      - type: unique
        columns: [source_system, source_order_id]

    columns:
      - name: source_system
        data_type: varchar
        constraints: [{ type: not_null }]
      - name: source_order_id
        data_type: varchar
        constraints: [{ type: not_null }]
      - name: order_id
        data_type: bigint
        description: "Surrogate key, not unique alone — see composite unique."
      - name: order_ts
        data_type: timestamp
        constraints: [{ type: not_null }]

    tests:
      - dbt_utils.unique_combination_of_columns:
          combination_of_columns: [source_system, source_order_id]

Step-by-step explanation.

The composite unique constraint is declared at model level with columns: [source_system, source_order_id]. On Postgres it becomes UNIQUE (source_system, source_order_id) — enforced.
On Snowflake / BigQuery / Redshift the constraint is informational. The dbt_utils.unique_combination_of_columns test fills the audit gap — it runs a post-build SELECT that GROUPs by the combination and asserts every group has size 1.
The order_id column carries a description that explains it is not unique alone — important for downstream consumers who might be tempted to JOIN on order_id alone.

dbt interview question on constraint enforcement

Solution Using the constraint + test split

version: 2
models:
  - name: fct_orders
    config:
      materialized: incremental
      unique_key: [order_id, line_no]
      on_schema_change: fail
      contract:
        enforced: true
      access: public
      group: sales

    constraints:
      - type: primary_key
        columns: [order_id, line_no]
      - type: foreign_key
        columns: [customer_id]
        expression: "{{ ref('dim_customer') }} (customer_id)"

    columns:
      - name: order_id
        data_type: bigint
        constraints: [{ type: not_null }]
      - name: line_no
        data_type: integer
        constraints: [{ type: not_null }]
      - name: customer_id
        data_type: bigint
        constraints: [{ type: not_null }]
        tests:
          - relationships:
              to: ref('dim_customer')
              field: customer_id
      - name: amount
        data_type: numeric(18,2)
        constraints: [{ type: not_null }]

    tests:
      - dbt_utils.unique_combination_of_columns:
          combination_of_columns: [order_id, line_no]

Step-by-step trace.

Bug class	Snowflake DDL guards?	dbt test guards?	What you would lose without each
NULL `order_id`	yes (NOT NULL is enforced)	yes (column-level not_null is implied by PK declaration; explicit test optional)	nothing extra needed
Duplicate `(order_id, line_no)`	no — PK is informational	yes (`unique_combination_of_columns`)	the only line of defence on Snowflake
Orphan `customer_id`	no — FK is informational	yes (`relationships`)	the only line of defence on Snowflake
Renamed `customer_id` column in SQL	n/a — contract catches at compile	n/a	compile-time guarantee from contract.enforced
Type drift `numeric` → `varchar`	n/a — contract catches at compile	n/a	compile-time guarantee from contract.enforced

Output:

Warehouse	NOT NULL	UNIQUE / PK	FK	CHECK	Tests audit gap
Postgres	enforced	enforced	enforced	enforced	optional belt-and-braces
Snowflake	enforced	informational	informational	not supported	mandatory
BigQuery	enforced	not supported	informational	not supported	mandatory
Redshift	enforced	informational	informational	not supported	mandatory

Why this works — concept by concept:

Constraints as catalog metadata — even when not enforced, the declarations appear in INFORMATION_SCHEMA, dbt docs, and the catalog. This is how lineage tools (Atlan, Castor, Stemma) discover the relationships and render the right diagrams.
Query-planner hints — Snowflake's optimiser will, for example, skip a DISTINCT pass when joining on a column declared unique/PK. Same on BigQuery for FK-driven join elimination. The constraint is "advisory" but has real performance impact.
Contract.enforced type pinning — independent of constraint enforcement. The contract diff at compile catches renames and type drift on every warehouse — that part is rock-solid regardless of constraint reality.
dbt tests as the cross-warehouse audit — unique, not_null, relationships, accepted_values, and the dbt_utils.* family run identically on every warehouse. They are the portable enforcement layer.
The pairing as the actual production pattern — declare the constraint (for catalog + planner + contract), add the matching test (for audit). The intentional redundancy is what makes the project survive a warehouse migration.
Cost — constraints add zero DDL cost on informational warehouses; minimal DDL cost on Postgres (one index per UNIQUE/PK). Tests cost one SELECT per test per build — already part of any mature dbt CI.

Data modeling
Topic — cardinality
Cardinality problems (data modeling)

Practice →

4. Versioning strategy for public models

Versions are how dbt does SemVer — breaking changes get a new number, non-breaking stay on the same one

The versions: block.

Top-level declaration. Inside the model YAML, add a versions: list. Each entry declares a v: number and optional overrides (description, columns, contract, defined_in).
latest_version: — names the version that ref('model') (without a v= argument) resolves to. Consumers without a v= get the latest by default.
defined_in: — the SQL filename for that version. If absent, defaults to model_vN.sql. Useful when versions live in separate files for clarity.
deprecation_date: — a date after which the version should not be used. dbt emits warnings during compile if any consumer still references a deprecated version after the date.

SemVer for data — the three rules of thumb.

MAJOR (v2, v3). Breaking change — column removed, renamed, retyped to an incompatible type, semantics changed (e.g. "amount in USD" → "amount in customer's local currency"). Consumers must migrate.
MINOR. Non-breaking addition — new column added at the end, new constraint added (where consumers are not relying on its absence), new test. Stays on the same version number.
PATCH. Doc-only or comment change. Stays on the same version.

The breaking-vs-non-breaking heuristic.

Breaking. Anything a downstream SELECT * would notice as a removal or rename. Anything a downstream WHERE or JOIN predicate would silently drop rows over (e.g. nullability flip on a join key). Anything a downstream type cast would fail on (e.g. bigint → string).
Non-breaking. Adding a new column at the end (downstream SELECT * gets one extra column; downstream named-column queries are unaffected). Adding a new test. Adding a new constraint that the data already satisfies (it just becomes formally checked).
The grey zone. Tightening a constraint (e.g. relaxing not_null to nullable, or vice versa). Treat tightening as non-breaking if consumers are not relying on the relaxed state; treat relaxing as breaking because a previously non-null column becoming nullable can crash downstream type-narrowed code.

Cross-version refs.

ref('model') — resolves to latest_version.
ref('model', v=1) — resolves to the v1 incarnation. Lets consumers stay on the old version explicitly.
ref('model', v=2) — resolves to the v2 incarnation.
Naming. Physical tables get the suffix: dim_customer_v1, dim_customer_v2. dbt manages the suffixing automatically; consumers only see logical names.

Common interview probes on versioning.

"When would you bump a version vs just edit the model?" — bump when the change is breaking for a public consumer. Edit when it is private, or when the change is non-breaking (additive column, doc, test).
"What is deprecation_date for?" — to advertise the sunset of an older version. dbt warns on compile if consumers still reference it after that date.
"Can two versions of the same model run in the same dbt project?" — yes; they materialise to separate physical tables (suffixed _v1, _v2). Each can have its own contract, columns, and constraints.
"How do I roll back a version?" — keep v1 alive (do not remove it) until you've confirmed v2 has zero issues. Roll back by re-pointing latest_version to v1.

Worked example — shipping v2 of `fct_orders` with a renamed column

Question. Show the YAML and SQL diff for promoting fct_orders from v1 to v2 with the renamed column. Set a 60-day deprecation_date on v1.

Input — current single-version YAML.

version: 2
models:
  - name: fct_orders
    config:
      contract: { enforced: true }
      access: public
    columns:
      - name: order_id
        data_type: bigint
      - name: customer_id
        data_type: bigint
      - name: order_amount
        data_type: numeric(18,2)

Code — versioned YAML.

version: 2
models:
  - name: fct_orders
    config:
      access: public
    latest_version: 2
    versions:
      - v: 1
        config:
          contract: { enforced: true }
        deprecation_date: 2026-08-15  # 60 days from today
        columns:
          - name: order_id
            data_type: bigint
          - name: customer_id
            data_type: bigint
          - name: order_amount
            data_type: numeric(18,2)
            description: "Order total, USD. Renamed to order_amount_usd in v2."
      - v: 2
        config:
          contract: { enforced: true }
        columns:
          - name: order_id
            data_type: bigint
          - name: customer_id
            data_type: bigint
          - name: order_amount_usd
            data_type: numeric(18,2)
            description: "Order total in USD."

Code — SQL files.

-- models/marts/sales/fct_orders_v1.sql (unchanged, kept alive)
SELECT
    order_id,
    customer_id,
    order_amount
FROM {{ ref('int_orders') }};

-- models/marts/sales/fct_orders_v2.sql (new, the renamed column)
SELECT
    order_id,
    customer_id,
    order_amount AS order_amount_usd
FROM {{ ref('int_orders') }};

-- A downstream consumer that wants to stay on v1 explicitly
SELECT customer_id, sum(order_amount) AS total
FROM {{ ref('fct_orders', v=1) }}
GROUP BY 1;

-- A downstream consumer on the latest version (v2)
SELECT customer_id, sum(order_amount_usd) AS total_usd
FROM {{ ref('fct_orders') }}    -- latest_version = 2
GROUP BY 1;

Step-by-step explanation.

latest_version: 2 makes ref('fct_orders') resolve to v2 — every new consumer gets the new shape by default. Existing consumers using ref('fct_orders', v=1) stay on v1.
The versions: list declares both versions side-by-side. Each version has its own columns: block — v1 keeps order_amount, v2 has order_amount_usd.
deprecation_date: 2026-08-15 on v1 tells dbt to start warning consumers 60 days from now. After the deprecation date, any compile that still references v1 emits a "this version is deprecated" warning (and can be configured to error).
Two SQL files (fct_orders_v1.sql, fct_orders_v2.sql) materialise to two physical tables (fct_orders_v1, fct_orders_v2). Both load on every dbt run; storage cost is the only overhead.
Consumers migrate at their own pace by changing ref('fct_orders', v=1) → ref('fct_orders') (or v=2) and updating their column references.

Worked example — adding a column without a version bump

Question. Show the YAML for adding currency to fct_orders without bumping the version.

Code — model-level edit.

- v: 2
  config:
    contract: { enforced: true }
  columns:
    - name: order_id
      data_type: bigint
    - name: customer_id
      data_type: bigint
    - name: order_amount_usd
      data_type: numeric(18,2)
    - name: currency                 # <- new column appended at end
      data_type: varchar
      constraints: [{ type: not_null }]
      description: "ISO 4217 currency code."

-- SQL update
SELECT
    order_id,
    customer_id,
    order_amount AS order_amount_usd,
    coalesce(currency, 'USD') AS currency
FROM {{ ref('int_orders') }};

Step-by-step explanation.

The new currency column is appended at the end of the columns: block. The contract diff at compile sees one extra column in the SELECT — but it is also in the YAML, so the diff matches. The build succeeds.
Existing consumers that wrote SELECT order_id, customer_id, order_amount_usd FROM fct_orders continue to work unchanged — they never named currency, so the new column does not affect them.
New consumers can opt-in to currency simply by adding it to their SELECT.
No version bump needed because nothing breaks for existing consumers. The semantic versioning rule is "minor change → same version" — this is the canonical minor change.
The coalesce(currency, 'USD') AS currency backfills a default for any historical rows where currency was NULL — important because we declared not_null on the new column and the contract would fail otherwise.

Worked example — a doc-only patch with no contract change

Detailed explanation. A column's description is wrong. Updating it is a pure documentation change — no schema impact, no contract impact, no version impact.

Question. Show a patch that fixes a column description and explain why no version bump is needed.

Code.

- name: signup_at
  data_type: timestamp
  description: >
    Timestamp of first successful account creation, in UTC.
    Was previously documented as "local time" — that was wrong
    on every load. Corrected 2026-06-15.
  constraints: [{ type: not_null }]

Step-by-step explanation.

The patch only edits the description: field. No column rename, no type change, no constraint change.
The contract diff at compile is unchanged — same column name, same type, same constraints.
No consumer was reading description from the YAML at runtime, so no consumer breaks.
The catalog (dbt docs) refreshes with the new description on next build. The lineage tools (Atlan, Castor) refresh on their next pull.
No version bump because nothing about the interface changed. The semantic versioning rule is "patch → same version" — this is the canonical patch.

Output. Updated documentation, zero downstream impact. The cost is one PR with one YAML hunk.

Worked example — cross-version `ref()` from a downstream mart

Question. Show the SQL diff for the downstream mart to pin itself to fct_orders v1.

Code.

-- models/marts/sales/agg_revenue_by_customer.sql
SELECT
    customer_id,
    sum(order_amount) AS total_revenue
FROM {{ ref('fct_orders', v=1) }}
GROUP BY customer_id;

Step-by-step explanation.

The ref('fct_orders', v=1) macro resolves to the physical table fct_orders_v1 — the v1 incarnation, with the old column name order_amount.
The mart's SELECT uses order_amount (the v1 name). It compiles and runs against v1's contract, which still declares order_amount.
When the mart team is ready, they change ref('fct_orders', v=1) → ref('fct_orders') (or v=2) and rename order_amount → order_amount_usd in their SELECT. One PR per consumer.
The producer team can drop v1 once every consumer has migrated and the deprecation_date has passed.

Output. The mart stays on v1 indefinitely (or until v1 is removed). The producer ships v2 in parallel. Consumers migrate at their pace.

dbt interview question on versioning a public model

Solution Using the publish-overlap-deprecate pattern

version: 2
models:
  - name: dim_customer
    config:
      access: public
      group: customer
    latest_version: 2
    versions:
      - v: 1
        config:
          contract: { enforced: true }
        deprecation_date: 2026-09-15  # 90 days from publish
        columns:
          - { name: customer_id, data_type: bigint }
          - { name: signup_at,   data_type: timestamp }   # renamed in v2
          - { name: email,       data_type: varchar }
      - v: 2
        config:
          contract: { enforced: true }
        columns:
          - { name: customer_id,   data_type: bigint }
          - { name: signed_up_at,  data_type: timestamp }  # the renamed column
          - { name: email,         data_type: varchar }

Step-by-step trace.

Date	Producer action	Consumer action	Effect
t=0	Publish v2 alongside v1; set `deprecation_date` 90 days out	Consumers continue on v1 by default until they migrate	both physical tables alive
t=0..30	Comms to consumers: "v2 published, 90-day window"	Forward-looking consumers migrate first	dbt warns on `ref('model', v=1)` after deprecation_date
t=30..75	Track v1 consumers via dbt selectors + query logs	Most consumers migrate; laggards get reminders	v1 traffic shrinks
t=75..90	Final reminder; sunset PR drafted	Last consumers migrate	v1 traffic approaches zero
t=90	Merge sunset PR — remove v1 from YAML and SQL	Any straggler `ref('model', v=1)` now fails to compile	clean state, only v2 alive

Output:

Stage	What exists	Who is affected	Cost
Publish v2	v1 + v2 both alive	nobody (consumers still on v1)	one PR for producer
Overlap window	v1 + v2 both alive	consumers migrate at own pace	storage cost of v1
Deprecation warnings	dbt compile warns on v=1	laggard consumers see warnings	none
Sunset v1	only v2 alive	nobody (everyone migrated)	one PR removing v1

Why this works — concept by concept:

Publish overlap as the migration safety net — v1 and v2 coexist for the deprecation window. Consumers migrate when they are ready, not when the producer demands. Zero coordination meetings required.
SemVer for data — the bump-or-not decision is a type decision (breaking → bump; non-breaking → same version). Once the team internalises the rule, every PR self-classifies and no one argues.
deprecation_date as the social contract — the date is the producer's promise to keep v1 alive that long. It is the consumer's deadline to migrate. dbt's warning at compile is the gentle nag that prevents the deadline from slipping unnoticed.
Cross-version ref() — the migration mechanism. Consumers explicitly pin to v1 with v=1; new consumers default to latest_version. The mechanism is the minimum coupling: one argument per ref.
Sunset PR as the final cut — removing v1 is one YAML edit + one SQL file delete. Any straggler consumer gets a clean compile error pointing at the removed version, not a silent break.
Cost — storage cost of the duplicate table during the overlap window. On most warehouses this is negligible for analytics-scale dims and facts. Compute cost is also low: v1 only loads what was already loading before; v2 loads in parallel.

Data modeling
Topic — slowly-changing-data
Slowly-changing-data problems (data modeling)

Practice →

5. Rollout and deprecation playbook

Coordinating dbt + BI + reverse-ETL on a single timeline — the four-phase rollout that retires v1 without drama

The four-phase playbook.

Phase 1 — Publish. Producer ships v2 in a single PR. v1 stays alive. deprecation_date is set on v1 (typically 30–90 days out). Comms go out: announcement, migration guide, FAQ, office hours.
Phase 2 — Overlap. Both versions run on every dbt build. Consumers migrate on their own schedule. Producer tracks adoption via dbt selectors and query logs. Comms cadence: weekly reminder, fortnightly tracker.
Phase 3 — Migrate. As deprecation_date approaches, producer surfaces remaining v1 consumers, opens tickets per team, runs office hours for stragglers. dbt compile warnings start firing.
Phase 4 — Sunset. After deprecation_date passes (with confirmation that v1 traffic is zero), producer ships a PR removing v1's YAML, SQL, and (eventually) the physical table.

The overlap window — how long?

30 days. Minimum for any non-trivial public model. Fine for internal teams with tight dbt slack channels.
60 days. A reasonable default for most production analytics orgs. Covers a typical sprint cadence and a vacation overlap.
90 days. For models used by many teams, by BI dashboards owned by non-engineers, or by external (partner-facing) consumers.
The pragmatic rule. Default to 60; bump to 90 if any consumer is non-technical or external; bump to 120 for regulated reporting where audit signoff is required.

Stakeholder comms template.

The announcement (day 0). Short Slack message + email: "We've published dim_customer_v2. v1 is deprecated as of today; sunset is YYYY-MM-DD (60 days). Migration guide: . Office hours: ."
The weekly reminder (day 7, 14, 21, ...). "v2 adoption: X/Y consumers migrated. Stragglers: . Office hours: ."
The pre-sunset warning (day -7). "Sunset in 7 days. Outstanding v1 consumers: . Please migrate or open a ticket for an extension."
The sunset PR (day 0 + window). "v1 removed. v2 is now the only version. Postmortem doc: ."

Tracking consumer migration.

dbt list --select +dim_customer_v1 — every model that downstream-references v1. The list shrinks as consumers migrate.
Query logs — warehouse query history filtered to dim_customer_v1 table name. Surfaces BI tools, reverse-ETL syncs, and ad-hoc consumers that dbt cannot see.
dbt exposures — declarative exposure: YAML blocks let you register BI dashboards, ML jobs, and external consumers as first-class graph nodes. dbt list --select +exposure:dim_customer_v1 then shows everything that depends on v1, including non-dbt artefacts.
The catalog / lineage tool — Atlan / Castor / Stemma surface upstream-downstream relationships including BI tiles. Often the most complete view.

Tying it all to CI.

PR CI. Run dbt build --defer --select state:modified+ on every PR — builds only the modified models (and downstream) against a baseline. Contracts and constraints catch interface changes at compile.
Slim CI. Use --defer against the prod state so the PR build doesn't need to rebuild every upstream model. Faster, cheaper, identical contract enforcement.
Block on contract violations. The contract failure is a build failure — make the PR check required for merge. No exceptions.
Deprecation warnings. Configure CI to fail (not just warn) when consumers reference a model past its deprecation_date. dbt 1.6+ has a --warn-error flag for this.

Coordinating with downstream BI and reverse-ETL.

Looker. Materialised LookML views referencing the dbt table by name need updating. Use a LookML view rename PR in the Looker repo when v2 is published.
Tableau. Live connections reference the table directly. Schedule a "Tableau update day" within the overlap window — extract → swap source → re-publish.
Hightouch / Census (reverse-ETL). Source models reference the dbt table by name. Update the source mapping when v2 is published.
Snowflake Share / BigQuery Authorised Views. External consumers see a view, not the underlying table. Re-create the share / authorised view against v2 during the overlap window so external consumers can migrate on their own schedule.

Postmortems for "contract broke prod" — what to add to the checklist.

Was the model marked contract.enforced: true? If not, why not.
Was the model marked access: public or group:? If not, why was it reachable from outside.
Was the change behind a version bump? If a breaking change shipped without a version, that is the primary root cause.
Did dbt CI catch it? If not, why — was state:modified+ not configured, was contract enforcement off in CI, was the test missing?
Did the comms go out? If not, why — and add to the playbook.
Was the rollback path documented? If not, add a "rollback PR" template.

Worked example — publishing v2 with a 60-day deprecation window

Question. Sketch the four PRs the producer ships during the rollout of dim_customer_v2.

Code — PR 1: Publish v2.

# models/marts/customer/dim_customer.yml — PR 1
version: 2
models:
  - name: dim_customer
    config: { access: public, group: customer }
    latest_version: 2
    versions:
      - v: 1
        deprecation_date: 2026-08-15
        config: { contract: { enforced: true } }
        columns:
          - { name: customer_id, data_type: bigint }
          - { name: signup_at,   data_type: timestamp }
          - { name: email,       data_type: varchar }
      - v: 2
        config: { contract: { enforced: true } }
        columns:
          - { name: customer_id,  data_type: bigint }
          - { name: signed_up_at, data_type: timestamp }
          - { name: email,        data_type: varchar }

Code — PR 2: Update one consumer (agg_revenue_by_customer).

-- agg_revenue_by_customer.sql — PR 2
SELECT
    c.customer_id,
    c.signed_up_at,                 -- was: c.signup_at
    sum(o.amount) AS total
FROM {{ ref('dim_customer') }} c    -- now resolves to v2
LEFT JOIN {{ ref('fct_orders') }} o
       ON o.customer_id = c.customer_id
GROUP BY 1, 2;

Code — PR 3: Track remaining v1 consumers.

# tracking script — PR 3 (CI cron job)
dbt list --select +dim_customer_v1 \
  --output name \
  > reports/v1_consumers.txt

# Plus warehouse query log scrape for BI tools
echo "Consumers still on dim_customer_v1:"
cat reports/v1_consumers.txt

Code — PR 4: Sunset v1 after the deprecation date.

# models/marts/customer/dim_customer.yml — PR 4 (after 2026-08-15)
version: 2
models:
  - name: dim_customer
    config: { access: public, group: customer }
    latest_version: 2
    versions:
      - v: 2
        config: { contract: { enforced: true } }
        columns:
          - { name: customer_id,  data_type: bigint }
          - { name: signed_up_at, data_type: timestamp }
          - { name: email,        data_type: varchar }
# v1 block removed; dim_customer_v1.sql file deleted

Step-by-step explanation.

PR 1 (day 0). Add v2, mark v1 deprecated. The PR is tiny: new YAML version block + new SQL file. CI verifies both versions contract-pass. Merged → both versions materialise on next build.
PR 2 (days 1–60). Each consumer team migrates in its own PR. The mart that owns agg_revenue_by_customer updates its SELECT and re-points ref('dim_customer') to the latest version (which is now v2). No coordination with other teams.
PR 3 (continuous). A CI job runs dbt list --select +dim_customer_v1 weekly and posts the shrinking list of remaining consumers to a Slack channel. Producer pings stragglers around day 45.
PR 4 (day 60+). Once dim_customer_v1 has zero remaining consumers, the producer removes the v1 block from YAML, deletes dim_customer_v1.sql, and (eventually, after one more clean build) drops the physical table.

Output.

Phase	Day	Producer	Consumer	Platform
Publish	0	PR 1 merged	—	both tables alive
Overlap	1–60	comms, tracking	migrate at own pace	shrinking v1 traffic
Sunset	60+	PR 4 merged	—	only v2 alive

Worked example — exposures as the BI/reverse-ETL visibility layer

Question. Show the YAML for an exposure: registering a Looker dashboard that depends on dim_customer, and explain how it surfaces during the rollout.

Code.

version: 2
exposures:
  - name: customer_360_dashboard
    type: dashboard
    maturity: high
    url: https://looker.internal/dashboards/customer-360
    description: "Marketing-ops Customer 360 dashboard."
    depends_on:
      - ref('dim_customer')           # latest_version
      - ref('fct_orders')
    owner:
      name: Marketing Analytics
      email: marketing-analytics@example.com

Step-by-step explanation.

The exposure: registers the dashboard as a downstream node. dbt list --select +dim_customer now includes exposure:customer_360_dashboard in the output.
During the rollout, the producer runs dbt list --select +dim_customer_v1 and immediately sees if the dashboard is still on v1. The exposure makes the BI tile visible to dbt for the first time.
The owner: block tells the producer who to message — automated comms can ping marketing-analytics@example.com directly.
When the dashboard migrates to v2, the owner updates the exposure to ref('dim_customer', v=2) (or leaves it at ref('dim_customer') to follow latest_version). dbt re-runs the list and the dashboard drops off the v1 consumer roster.

Worked example — the contract-broke-prod postmortem template

Question. Sketch the template structure for a "contract broke prod" postmortem.

Code — markdown template.

# Postmortem — dim_customer v1→v2 rollout incident

## Summary
[1-2 sentences: what broke, when, who noticed]

## Timeline
- t-7d  Publish v2 + 60-day deprecation_date on v1
- t-2d  Reminder ping in #data-platform
- t=0  v1 dropped (sunset PR merged)
- t+1h Looker tile X errors out; marketing-ops opens ticket
- t+2h Rollback PR re-introduces v1
- t+4h Resolution: tile migrated to v2 by hand; v1 dropped again

## Root cause
[Exact reason: e.g., exposure not registered for Looker tile X;
 weekly tracking script missed it; sunset PR proceeded with one
 unmigrated consumer]

## What worked
- contract.enforced caught two unrelated drift PRs during the overlap window
- Slack pings during weeks 4 and 6 surfaced 3 of 4 stragglers
- Rollback PR (re-add v1 block) restored service in ~30 minutes

## What didn't
- Looker tile X was not registered as an exposure
- Query-log scrape missed it because tile X uses an extract refreshed weekly

## Action items
- [ ] Register every Looker tile that reads marts/* as an exposure
- [ ] Extend rollout playbook with a "scrape extract schedules" step
- [ ] CI: fail (not warn) on compile when an unmigrated consumer references a deprecated version
- [ ] Update on-call runbook with "rollback PR" recipe

Step-by-step explanation.

Summary is the one-paragraph version a busy executive reads.
Timeline documents the events with t-relative times — easy to copy into other tooling.
Root cause names the specific gap (in this case: exposure not registered, weekly query-log scrape missed an extract).
What worked is the positive section — never skip it. Every postmortem needs to celebrate what the system did catch.
What didn't is the gap analysis. Be specific. "Comms were unclear" is not actionable; "Looker tile X was not registered as an exposure" is.
Action items are the playbook updates. Each one feeds back into the rollout checklist for the next release.

Output. A postmortem that teaches the next engineer. The playbook gets one new step. The CI gets one new check. The incident never happens the same way twice.

dbt interview question on the rollout playbook

Solution Using the four-phase rollout

# 60-day rollout — dim_customer v2

## Day 0 — Publish
- PR 1 merges: v2 alongside v1, contract.enforced on both, deprecation_date = day 60
- Comms: Slack announcement + email to data-platform-consumers@
- Migration guide: pinned in #data-platform
- Office hours: open every Friday for the next 8 weeks
- CI: contract enforcement on, deprecation warnings on (compile warning, no fail yet)

## Days 1-30 — Overlap (warning phase)
- Producer publishes weekly "v1 consumer count" Slack post
- Consumer teams migrate; each ships their own PR re-pointing `ref('dim_customer')`
- CI: continues to warn on `ref('dim_customer', v=1)` references
- Tracking: dbt list + warehouse query log + exposure metadata

## Days 31-60 — Migrate (escalation phase)
- Day 30: producer opens a JIRA ticket for each remaining v1 consumer team
- Day 45: producer pings each ticket owner directly
- Day 55: pre-sunset reminder Slack post + email
- Day 58: CI flip — `--warn-error` enabled for deprecation warnings; PRs that still reference v1 fail
- Day 60: deprecation_date reached

## Day 60+ — Sunset
- Confirm zero v1 traffic via query log for past 48 hours
- PR 4 merges: v1 YAML block removed; SQL file deleted
- After one clean dbt run, drop the physical v1 table
- Post-rollout note in #data-platform: "v1 sunset complete; v2 is now the only version"
- Postmortem only if anything went wrong; otherwise a brief retro

## Rollback paths
- During overlap: revert PR 1 (re-add v1 block if it was removed prematurely)
- After sunset: re-create v1 from the v2 SQL with a one-PR add-back if a critical consumer surfaces late

Step-by-step trace.

Day	Producer action	Consumer state	CI behaviour	Risk if skipped
0	publish v2; deprecation_date set	all on v1	contract pass; v=1 ref compiles cleanly	rollout has no anchor date
7	weekly tracker post	early adopters migrating	unchanged	no visibility into adoption pace
30	open per-team tickets	~50% migrated	warn on v=1 ref	stragglers never feel urgency
45	direct pings to laggards	~80% migrated	warn	last 20% slip past deadline
58	flip CI to fail on v=1 ref	~95% migrated	fail on v=1 ref	sunset breaks last consumers
60	sunset PR; remove v1	100% migrated	only v2 references compile	hard break if any consumer remains
60+	drop physical v1 table	done	done	storage cost only

Output:

Day	What exists in warehouse	What CI does	Risk profile
0	v1 + v2 alive	warn on v=1	low — overlap covers everyone
30	v1 + v2 alive	warn on v=1	low — half migrated
58	v1 + v2 alive	fail on v=1	medium — forces last migrations
60	v1 + v2 alive	fail on v=1	resolved — final cut
60+	only v2 alive	normal	clean steady state

Why this works — concept by concept:

Publish-overlap-migrate-sunset as four discrete phases — each phase has a clear start, a clear end, and a clear set of stakeholder actions. The producer is never "trying to figure out what to do next."
deprecation_date as the social contract — the date is fixed at publish time and visible in YAML. Everyone — producer, consumer, BI owner — sees the same deadline.
CI escalation from warn to fail — the gradual ratchet (warn for 58 days, fail for 2 days, sunset) gives consumers maximum runway with a final forcing function.
Per-team JIRA tickets at day 30 — turns the comms from "broadcast" to "directed." Each laggard team has an owner and a deadline.
Exposures as the BI visibility layer — without them, the query-log scrape is your only signal for non-dbt consumers. With them, every dashboard and reverse-ETL sync is a first-class graph node.
Postmortem only on incident — most rollouts are uneventful. Reserve the postmortem ritual for the times when something genuinely went wrong; otherwise a brief retro is enough.
Cost — producer time: ~4 hours over 60 days. Consumer time: ~30 min per team per migration. Storage cost: one duplicate table for 60 days. Compared to the cost of one broken-Monday incident, this is rounding error.

Data modeling
Topic — event-modeling
Event modeling problems (data modeling)

Practice →

Cheat sheet — dbt contract recipes

Mark a model public. Add config.contract.enforced: true, fill out the columns: block with name + data_type + constraints + description for every column, add config.access: public and config.group:. Ship as v: 1 in a versions: block from day one — saves a YAML refactor when you ship v: 2 later.
Add a column without breaking anyone. Append the new column at the end of the columns: block, ship the YAML + SQL in one PR, and do not bump the version. The change is non-breaking because no existing consumer named the new column.
Rename a column. Ship v: 2 alongside v: 1. Give v1 a 30–90 day deprecation_date. Update one consumer per PR. Drop v1 after the deprecation date and zero remaining traffic. Never edit v1 to rename in place.
Tighten a constraint (loose → strict). Verify the data already satisfies the strict form (run the test once against prod data). Edit the YAML to add not_null / check / unique. Ship as a non-breaking change if the data already satisfies it; otherwise bump the version because the change can fail consumers who insert NULLs.
Loosen a constraint (strict → loose). Treat as breaking. Removing not_null means downstream consumers that rely on the non-null contract may now crash. Ship as v: 2.
FK to a dim on Snowflake / BigQuery / Redshift. Declare the FK in the YAML (informational metadata + catalog + query-planner hint) and add a tests: relationships: test for the value-level audit. Belt and braces.
FK to a dim on Postgres. Declare the FK in YAML; index the referenced column for INSERT performance; add a tests: relationships: test as an audit layer. The DDL enforcement is real; the test is the cross-warehouse guarantee.
Composite primary key. Declare at the model level under constraints: with columns: [a, b]. Add a matching dbt_utils.unique_combination_of_columns test.
Check constraint. Add a check constraint with an expression: (e.g. "price >= 0"). Pair with a dbt_utils.expression_is_true or accepted_values test. The constraint enforces on Postgres; the test enforces on every warehouse.
Enforce at PR time. Configure CI to run dbt build --defer --select state:modified+ against the prod state. Make the contract-failure check required for merge. Use --warn-error to escalate deprecation warnings into failures.
Track who still consumes v1. Run dbt list --select +dim_customer_v1 --output name in a weekly CI cron. Scrape warehouse query logs for non-dbt consumers. Register every BI tile and reverse-ETL sync as a exposures: block.
Sunset v1 cleanly. Confirm zero traffic in the 48 hours before the cut. Ship a single PR that removes the v1 YAML block + deletes the v1 SQL file. Drop the physical table only after one clean build verifies nothing references it.
Roll back a breaking change. During the overlap window: revert the PR that removed v1 (re-add the YAML block and SQL file). After sunset: open a fresh PR that re-introduces v1 with the same shape. Both paths are quick because the v1 SQL is in git history.
Document intent in every column. Add description: to every column. Future-you (and every consumer) will thank you. Description edits are doc-only patches with no version bump.

Frequently asked questions

Are dbt constraints enforced by the warehouse?

Do I need contracts if I already have dbt tests?

When should I bump a model version?

Can I have contracts on incremental models?

How do I get foreign keys in Snowflake or BigQuery?

What's the difference between contracts and dbt-expectations?

Practice on PipeCode

Drill the data modelling practice library → for the schema-design, contract-readiness, and SCD interview surface.
Rehearse on dimensional modelling problems → for star-schema fact-and-dim contract design.
Tighten the schema-evolution muscles with slowly-changing-data drills → — versioning a public dim is the same problem class.
Stack the cardinality library → for "is this a 1:1, 1:N, or N:N relationship" probes that drive PK / FK / unique-constraint design.
Sharpen event-modelling problems → for the immutable-table contract patterns that show up in fact-table and event-source interview questions.
Layer the design problems library → for the broader "design this warehouse layer" interview surface.
For the broader DE surface, read top data engineering interview questions →.
Stack the prerequisites with the only 5 skills you need to become a data engineer →.
For long-form schema craft, work through data modelling for DE interviews →.
For the broader ETL design surface, take the ETL system design course →.

Practice data modeling now →
Dimensional modelling drills →

OpenLineage & OpenMetadata: Open Standards for Lineage and Cataloging

Gowtham Potureddi — Mon, 15 Jun 2026 13:20:18 +0000

openlineage openmetadata is the pair of words that quietly replaced the closed-catalog conversation in 2024 and 2025 — and by 2026, when an interviewer asks "how would you build lineage and a catalog across your stack?" the wrong answer is "we'd license Atlan" and the right answer starts with "OpenLineage as the wire format, OpenMetadata or DataHub as the backend." The shift is the same one that happened with Kubernetes versus proprietary container schedulers: the moment a credible open standard exists, every vendor either adopts it or argues itself into irrelevance.

This guide walks the two standards in production-engineering detail. It opens with why open standards for lineage and metadata matter at all (the cost of being trapped inside a closed metadata graph), then layers the OpenLineage event model (run, job, dataset, facets) on top of the OpenMetadata architecture (ingestion, metadata server, UI), and closes with the interop patterns that let you migrate off Atlan, Collibra, or Alation without a Big Bang cutover. Along the way it ties in Marquez and DataHub — the two most-mentioned reference backends — and shows the column-level lineage facet that makes a modern open data catalog actually useful for impact analysis. Every H2 ships at least one worked example with code, a step-by-step trace, an output table, and a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the ETL practice library →, rehearse on dimensional modeling problems →, and layer the data aggregation drills →.

On this page

Why open standards for lineage and metadata matter
The open standards stack
The OpenLineage event model
OpenMetadata architecture and entity model
Interop with proprietary vendors and migration patterns
Cheat sheet — open standards recipes
Frequently asked questions
Practice on PipeCode

1. Why open standards for lineage and metadata matter

Closed catalogs trap your metadata graph inside a vendor's billing model — open standards let lineage and entity definitions outlive the contract

The one-sentence invariant: lineage and metadata are the two most expensive things to backfill, so the format you choose to emit them in is a 5-to-10-year decision, and proprietary catalogs charge you forever to read back data you already paid to compute. Once you internalise that "the graph you build is more valuable than the UI you license," the case for openlineage plus openmetadata (or DataHub) over a closed product becomes the default architectural posture.

The lock-in tax of proprietary catalogs.

Per-asset pricing scales with success. Every catalog vendor invoices on "data assets" — tables, dashboards, columns, pipelines. The more your platform grows, the more you pay, even when the marginal user value of asset 50,001 is near zero.
Export is intentionally hard. Closed catalogs expose only narrow REST APIs (or paginated CSV exports) for the metadata you contributed. Lineage edges, column-level mappings, glossary tags, and ownership graphs are often not round-trippable — you can read them, but you cannot bulk-extract them in a form the next catalog will understand.
Connectors are the moat. A vendor's competitive edge is "we have 200 connectors." But those connectors emit into the vendor's internal metadata model. Switching means rebuilding every connector for the new tool — months of work for a platform team that wants to ship product instead.

The "every tool emits to its own black box" problem.

In a typical 2023-era stack, Airflow exposed lineage to its own DB, dbt exposed lineage to dbt Cloud, Spark exposed lineage to Spline (if anything), Atlan ingested from BigQuery, Monte Carlo ingested separately for observability, and Collibra ingested independently for governance. Each tool maintained its own copy of the same fact: job daily_orders reads raw_orders and writes fct_orders. That fact was duplicated five times, inconsistently, with each vendor's UI showing a slightly different graph.

What an open standard buys.

One emit, many consumers. Airflow emits an OpenLineage event once. Marquez, OpenMetadata, DataHub, Monte Carlo, Atlan, and Collibra can all receive it. The graph is single-sourced; the receivers compete on UX, not on data ownership.
Vendor portability. Move from Atlan to OpenMetadata? You point the OpenLineage transport at the new backend. Your emitters do not change. Your pipeline code does not change.
Community integrations. When the OpenLineage spec adds a columnLineage facet, every emitter and every receiver implements it on the same schedule, in the same shape. No more "Vendor X supports column lineage on Snowflake but not Postgres."
Schema review by committee. OpenLineage and OpenMetadata are governed by the LF AI & Data Foundation. Spec changes go through public RFC discussion. There is no surprise breaking change from a vendor changing strategy.

Lineage vs metadata vs catalog — separating the three concerns.

Lineage is the runtime fact of "this job read these inputs and wrote these outputs at this time." It is a stream of events emitted by the compute engine.
Metadata is the static description of an asset: its schema, owner, tags, description, freshness SLO, classification. It is rows in a catalog DB.
Catalog is the application layer — the UI, the search index, the REST API, the access policies — that lets humans browse and query the metadata graph.

OpenLineage targets the lineage problem. OpenMetadata targets the metadata + catalog problem. They are complementary, not competitors.

The current ecosystem.

OpenLineage — the wire-format standard. JSON Schema for runs, jobs, datasets, and extensible facets. Reference backend is Marquez.
OpenMetadata — the open catalog application. Self-hosted or managed via Collate. Ingests from databases, dashboards, pipelines, ML models. Defines its own entity schemas.
Marquez — the original OpenLineage backend. Simple Postgres + REST UI. Great when you only want lineage and do not yet need a full catalog.
DataHub — alternative open catalog, originally from LinkedIn. Slightly different entity model than OpenMetadata, stronger upstream metadata-event story.
Amundsen — earlier-generation open catalog from Lyft. Less actively developed in 2026; relevant mostly for historical context.

What interviewers listen for.

Do you say "OpenLineage is the wire format, not a catalog"? — senior signal.
Do you mention Marquez as the reference backend for OpenLineage? — senior signal.
Do you distinguish DataHub and OpenMetadata as two parallel open-catalog projects? — senior signal.
Do you propose a two-write migration when leaving a closed catalog? — senior signal.

Worked example — the lock-in cost of a closed catalog in one number

Detailed explanation. A platform team has 8,000 tables, 1,200 dashboards, and 400 dbt models. The closed catalog vendor invoices on data_assets. Migrating off the vendor requires re-emitting lineage from every pipeline; staying means paying forever. Pricing the two options surfaces why the open-standard answer is the default.

Question. Compute the three-year total cost of staying on a closed catalog at $0.50 per asset per month versus migrating to OpenMetadata + OpenLineage in a self-hosted footprint that costs $4,000 per month all-in (infra + 0.25 FTE). Assume asset count grows 25% per year.

Input.

Year	Assets (start)	Assets (end)	Avg assets
1	9,600	12,000	10,800
2	12,000	15,000	13,500
3	15,000	18,750	16,875

Code.

# Three-year cost model — closed vs open

closed_unit_cost_per_month = 0.50  # USD per asset per month
open_monthly_cost = 4_000          # USD per month, all-in self-hosted

avg_assets = [10_800, 13_500, 16_875]

closed_total = sum(a * closed_unit_cost_per_month * 12 for a in avg_assets)
open_total   = open_monthly_cost * 12 * 3

print(f"Closed 3-year cost:  ${closed_total:>10,.0f}")
print(f"Open   3-year cost:  ${open_total:>10,.0f}")
print(f"Savings:             ${closed_total - open_total:>10,.0f}")
print(f"Break-even assets:   {open_monthly_cost / closed_unit_cost_per_month:>10,.0f}")

Step-by-step explanation.

Closed-catalog cost is linear in asset count. At $0.50 per asset per month, 10,800 average assets year 1 means 10,800 * 0.50 * 12 = $64,800 for year 1.
Year 2 grows to 13,500 average assets → 13,500 * 0.50 * 12 = $81,000. Year 3 hits 16,875 average → $101,250.
Open-catalog cost is flat: $4,000 per month * 36 months = $144,000.
The break-even is open_monthly / closed_unit = 4000 / 0.50 = 8,000 assets. Above that asset count, OpenMetadata is cheaper at this infra budget.

Output.

Metric	Value
Closed 3-year cost	$247,050
Open 3-year cost	$144,000
Savings	$103,050
Break-even assets	8,000

Rule of thumb. Below ~5,000 assets, the cost case for self-hosting is weaker — the FTE overhead dominates. Above ~10,000 assets, the open-standard answer pays for itself within the first contract renewal, before counting the value of avoiding vendor lock-in.

Worked example — what "lineage as a stream of events" looks like end-to-end

Detailed explanation. OpenLineage's mental model is event-driven: every job run emits a START event when it begins and a COMPLETE event when it finishes (or FAIL / ABORT on error). Each event carries the run, the job, the input datasets, the output datasets, and any number of facets. Concatenated over time, these events are the lineage graph.

Question. Show the minimum two-event sequence that captures a daily Airflow run of dbt_run_orders which reads raw.orders and writes analytics.fct_orders. Identify which fields are mandatory and which are optional.

Input.

Field	Value
run id	`a3f1-2026-06-15-01`
job name	`analytics.dbt_run_orders`
inputs	`raw.orders`
outputs	`analytics.fct_orders`

Code.

// Event 1 — START
{
  "eventType": "START",
  "eventTime": "2026-06-15T01:00:00.000Z",
  "run":  { "runId": "a3f1-2026-06-15-01" },
  "job":  { "namespace": "analytics", "name": "dbt_run_orders" },
  "inputs":  [ { "namespace": "warehouse", "name": "raw.orders" } ],
  "outputs": [ { "namespace": "warehouse", "name": "analytics.fct_orders" } ],
  "producer": "https://github.com/OpenLineage/OpenLineage/tree/1.20.0/integration/airflow"
}

// Event 2 — COMPLETE
{
  "eventType": "COMPLETE",
  "eventTime": "2026-06-15T01:04:12.000Z",
  "run":  { "runId": "a3f1-2026-06-15-01" },
  "job":  { "namespace": "analytics", "name": "dbt_run_orders" },
  "inputs":  [ { "namespace": "warehouse", "name": "raw.orders" } ],
  "outputs": [ { "namespace": "warehouse", "name": "analytics.fct_orders" } ],
  "producer": "https://github.com/OpenLineage/OpenLineage/tree/1.20.0/integration/airflow"
}

Step-by-step explanation.

The START event arrives when the Airflow operator begins. The runId is a UUID stamped once per attempt — Airflow uses the DAG run's try_number plus task id to derive it.
Inputs and outputs are listed intentionally. OpenLineage does not infer them; the emitter is responsible for declaring what the job will read and write. dbt knows from its compiled manifest; Spark knows from its query plan; Airflow falls back to operator-specific hints.
The COMPLETE event arrives when the operator returns. It re-states the same run, job, inputs, and outputs — receivers reconcile the two events by runId. If a FAIL or ABORT event arrives instead, the receiver knows the lineage edge is attempted rather than successful.
The producer field is the URL of the emitter's source. Receivers use it to know "this event came from Airflow 1.20.0 integration" and apply version-specific facet handling.

Output (Marquez UI rendering).

Node type	Identifier	Edges
Job	`analytics.dbt_run_orders`	input from `warehouse.raw.orders`, output to `warehouse.analytics.fct_orders`
Dataset	`warehouse.raw.orders`	read by `analytics.dbt_run_orders`
Dataset	`warehouse.analytics.fct_orders`	written by `analytics.dbt_run_orders`
Run	`a3f1-2026-06-15-01`	status COMPLETE, duration 4m 12s

Rule of thumb. Think of OpenLineage as Prometheus for lineage: emitters push events; backends scrape and persist; UIs render. The wire format is small and stable; the value compounds over thousands of runs.

Worked example — the "every tool has its own graph" failure mode

Detailed explanation. Without an open standard, every tool keeps its own private graph and your platform team operates as the human consistency layer. When the dbt graph says model X depends on table Y but the Airflow graph says task A depends on table Z and the BI tool says dashboard D depends on column C, no one can answer "if I drop column C, what breaks?" in less than a half-day investigation.

Question. A finance dashboard breaks because the currency column was renamed in the source. Trace the four lookups a platform engineer must do without an open standard and the single lookup they would do with one.

Input (impacted assets in five tools).

Tool	Asset	Records column
Postgres source	`raw.invoices`	`ccy` (renamed from `currency`)
dbt	`int_invoices`, `fct_revenue`	references `currency`
Airflow	DAG `daily_revenue`	runs `dbt build`
BI	dashboard `finance.revenue_v2`	uses `fct_revenue.currency`
Catalog	`fct_revenue` lineage	last refreshed 6h ago

Code.

# Without OpenLineage / OpenMetadata — four siloed lookups
1. dbt docs: which models reference `currency`?
2. Airflow UI: which DAGs run those models?
3. BI tool: which dashboards depend on those tables?
4. Catalog: which downstream owners need notification?

# With OpenLineage + OpenMetadata — one query
GET /api/v1/lineage/table/warehouse.raw.invoices?upstreamDepth=0&downstreamDepth=4

Step-by-step explanation.

Without the open standard, each tool answers a slice of the question against its private graph. The engineer manually stitches answers — dbt says "models X, Y depend on currency"; Airflow says "DAG daily_revenue runs them"; the BI tool says "dashboards A and B depend on Y"; the catalog confirms ownership but is stale.
The stitching is error-prone: a dbt model invoked by an ad-hoc notebook (not Airflow) is invisible to the Airflow lookup. A dashboard that depends on a derived column via a join is invisible unless the BI tool indexed column lineage.
With OpenLineage emitters everywhere and OpenMetadata as the single sink, the question is one API call. The downstream graph is materialised continuously from the events; the answer is whichever assets currently sit downstream of warehouse.raw.invoices.
Time-to-impact-analysis drops from "half a day" to "30 seconds." That speed is the operational ROI of a unified metadata graph — and the strongest argument when the team's senior engineer asks "why are we spending two weeks adopting another standard?"

Output (impact-analysis table from a single OpenMetadata query).

Hop	Asset	Owner	Action required
1	`warehouse.raw.invoices.currency`	data-eng	rename mapping in dbt staging
2	`analytics.int_invoices`	data-eng	regenerate, redeploy
3	`analytics.fct_revenue`	analytics-eng	document column
4	`bi.dashboards.finance.revenue_v2`	finance-eng	update dashboard tile

Rule of thumb. The single best heuristic for "is our metadata stack mature?" is "can we answer the impact-analysis question in under a minute?" If no, the next architecture investment is OpenLineage emitters plus a single open backend.

Data engineering interview question on choosing between open and closed catalogs

A senior interviewer might frame this as: "Your CFO is asking why we should not just buy Atlan and be done. Defend the open-standards path in a 60-second answer that does not sound like an open-source zealot speech."

Solution Using a TCO + portability scorecard

# Decision matrix — closed vs open catalog
# Score each criterion 1-5 (higher = better for the option)

criteria = {
    "feature_velocity_today":        {"closed": 5, "open": 4},   # vendor ships polish
    "five_year_TCO_at_scale":         {"closed": 2, "open": 5},   # per-asset pricing scales painfully
    "vendor_portability":             {"closed": 1, "open": 5},   # OL means switching cost is near zero
    "control_over_metadata_graph":    {"closed": 2, "open": 5},   # self-hosted = your own DB
    "platform_team_FTE_required":     {"closed": 5, "open": 3},   # closed is cheaper in eng hours
    "integration_with_OSS_emitters":  {"closed": 3, "open": 5},   # OL emitters land on open backends day 1
}

closed_score = sum(c["closed"] for c in criteria.values())
open_score   = sum(c["open"]   for c in criteria.values())

print(f"Closed total: {closed_score}")
print(f"Open total:   {open_score}")

Step-by-step trace.

Criterion	Closed	Open	Comment
feature_velocity_today	5	4	vendors ship polish faster
five_year_TCO_at_scale	2	5	per-asset bill grows with success
vendor_portability	1	5	OL means switching cost near zero
control_over_metadata_graph	2	5	self-hosted = your own DB
platform_team_FTE_required	5	3	closed cheaper in eng hours
integration_with_OSS_emitters	3	5	OL emitters land everywhere day 1

The closed option leads on short-term polish and FTE economy; the open path leads on every multi-year axis (TCO, portability, control, integrations). For a platform expected to outlive any single vendor contract, the open path wins on every criterion you would care about three renewals out.

Output:

Path	Total
Closed catalog	18
Open standards (OL + OM/DH)	27

Why this works — concept by concept:

Total cost of ownership — vendor pricing is per-asset, and asset counts grow super-linearly with success; open infra is a flat-ish cost. Above ~10K assets, open wins on cash alone.
Portability premium — OpenLineage emitters survive backend changes; the cost of changing backends approaches the cost of pointing the OL transport at a new URL. That option value is real and grows with platform maturity.
Control over your own metadata graph — when the catalog DB is yours, you can run arbitrary queries against it: cardinality audits, governance dashboards, custom impact analyses. Closed APIs cap you at the vendor's imagination.
FTE realism — yes, self-hosted costs platform-engineering time. The fair comparison is not "free vs paid"; it is "X FTE-months vs Y dollars plus lock-in." The decision matrix surfaces this honestly.
OSS emitter integration — every new OpenLineage emitter (Snowflake, Trino, Materialize) lands on every open backend at the same time. Closed catalogs lag by a release cycle.
Cost — the analysis itself is one spreadsheet plus a back-of-envelope FTE estimate. The actual decision is bought back over years of avoided lock-in pain.

DE
Topic — ETL design
ETL & pipeline design problems

Practice →

2. The open standards stack

OpenLineage is the wire format, OpenMetadata is the catalog application — they sit at different layers of the same stack, and confusing them is the most-common interview mistake

The mental model in one line: OpenLineage defines what to emit (a JSON event); OpenMetadata defines where to store and query (a catalog application with REST APIs and a UI). Once you say "wire format versus application," every follow-up question about Marquez, DataHub, or whether to "use OpenLineage or OpenMetadata" answers itself: you almost always use both, at different layers.

The four-layer stack in one paragraph.

Layer 1 — Emitters. The things that produce lineage events: Airflow, dbt, Spark, Flink, Dagster, Prefect, custom Python apps. Each emitter has an OpenLineage integration that translates its native execution model into OL events.
Layer 2 — Wire format (OpenLineage). The JSON schema for the event itself: run, job, dataset, and an extensible facets slot. Versioned by the OpenLineage spec.
Layer 3 — Backends. The things that consume and persist the events: Marquez (reference backend, lineage-only), OpenMetadata (full catalog), DataHub (alternative catalog), and vendor receivers (Monte Carlo, Atlan, Bigeye, Collibra) when those products accept OL.
Layer 4 — Consumers. The humans and systems that read the persisted graph: catalog UIs, search indexes, impact-analysis services, governance dashboards, downstream alerting.

The "one event, many consumers" pattern.

A single OpenLineage event emitted by an Airflow task can simultaneously land in:

Marquez for the lineage graph UI used by data engineers.
OpenMetadata for the broader catalog with glossary and tags used by analysts and stewards.
Monte Carlo or Bigeye for observability and freshness anomaly detection.
A custom Kafka topic that downstream services subscribe to for "this table just changed" event-driven processing.

The OL spec includes an HTTP transport and a Kafka transport out of the box. Multi-cast is solved by either configuring multiple OPENLINEAGE_URL entries (newer integrations) or by running a small fan-out proxy that re-emits each event to N backends.

OpenLineage vs OpenMetadata — when each is the right answer.

You need	OpenLineage	OpenMetadata
The fact "job X read table Y at time T"	yes (emit + persist)	partial (ingests OL events)
A searchable UI of every table with owners and tags	no	yes
Column-level lineage facets	yes (in the event)	yes (renders the graph)
A glossary, classifications, PII tags	no	yes
Data quality test results	partial (facet)	yes (first-class entity)
Connectors for BigQuery, Snowflake, Tableau metadata	no	yes
A wire format other tools can also emit to	yes	no (it is an application)

Marquez and DataHub in one sentence each.

Marquez is the reference OpenLineage backend — Postgres for storage, REST API for ingest and query, a minimal lineage UI. Use when you want "OpenLineage and a graph viewer" and nothing else.
DataHub is an alternative open catalog (originally LinkedIn) that competes with OpenMetadata. It uses its own metadata-event model (MCE / MAE) but accepts OL events through an adapter. Use when you want strong upstream metadata propagation with Kafka under the hood.

Where vendors plug in.

As emitters. A vendor's product (e.g. a closed orchestrator) can ship native OL events instead of a proprietary metadata API. Increasingly common — even Databricks and Snowflake now have OL integration paths.
As backends. Monte Carlo, Bigeye, Atlan, and Collibra accept OL events as input. Your team emits once; the vendor enriches and visualises.
As ingestion sources for OpenMetadata. OpenMetadata's ingestion-framework runs as Airflow DAGs (or a Python container) and uses connectors to pull metadata from Snowflake, BigQuery, Tableau, Looker, Kafka. These connectors do not emit OL; they push entities directly into the OpenMetadata server.

Two paths in: events versus connectors.

OpenMetadata has two ingest paths. (1) Connectors that crawl source systems (Snowflake INFORMATION_SCHEMA, Tableau REST API, etc.) and push entity records via REST. (2) OpenLineage events that arrive via the OL endpoint and get converted into Pipeline entities + lineage edges. Many teams use both — connectors for the entity inventory, OL for the runtime lineage.

Common interview probes on the stack.

"Is OpenLineage a database?" — no. It is a wire format. Storage is the backend's job.
"Can I use OpenLineage without a catalog?" — yes. Marquez gives you lineage-only without the wider catalog surface.
"Can I use OpenMetadata without OpenLineage?" — yes. Connectors alone populate the catalog; lineage will then be limited to whatever the connectors infer from query history.
"Why not DataHub then?" — usually a tie. DataHub's metadata-event model is more event-native; OpenMetadata's connector library is broader. Pick by ecosystem fit, not by logo.

Worked example — sketching the stack as data flow

Detailed explanation. Drawing the four-layer stack with concrete tools at each layer is the fastest way to internalise where each project sits. The picture makes "OpenLineage versus OpenMetadata" stop being a question.

Question. Sketch a four-layer stack diagram for a team running Airflow, dbt, and Spark that wants both runtime lineage and a searchable catalog. Identify which projects sit at which layer and which transport carries events between them.

Input.

Tool	Role
Airflow	orchestration
dbt	transformation in warehouse
Spark	external transformation
Snowflake	warehouse
Marquez	wanted as lineage UI
OpenMetadata	wanted as catalog UI

Code.

LAYER 1 — Emitters
  Airflow (OL plugin)  dbt (OL adapter)  Spark (OL listener)

LAYER 2 — Wire format
  OpenLineage event (run, job, dataset, facets) over HTTP
    Endpoint: OPENLINEAGE_URL = http://oltransport:5000

LAYER 3 — Backends
  Marquez (lineage UI + Postgres)
  OpenMetadata (catalog UI + Elasticsearch + Postgres)
  Both subscribe via a fan-out proxy or dual OPENLINEAGE_URL

LAYER 4 — Consumers
  Marquez UI for "trace the job"
  OpenMetadata UI for "find the table, owner, tags"
  Custom Slack bot subscribed to FAIL events for on-call

Step-by-step explanation.

Each emitter is configured once to point at the OpenLineage transport URL. The team does not have to know which backends are subscribed downstream.
The transport is HTTP by default; Kafka is the production choice when you want backpressure and durability between emitters and backends.
The fan-out happens at the transport layer or with a small proxy (often a single FastAPI service) that POSTs each incoming event to every configured backend.
Marquez and OpenMetadata coexist happily. They consume the same OL events but render different parts of the metadata graph — Marquez focuses on the lineage graph; OpenMetadata adds catalog, glossary, and quality on top.

Output (the stack table).

Layer	Tool	What it does
1	Airflow, dbt, Spark	emit OL events on every run
2	OpenLineage JSON over HTTP	transport
3	Marquez, OpenMetadata	persist + render
4	Marquez UI, OpenMetadata UI, Slack bot	humans and downstream alerting

Rule of thumb. Draw this four-layer stack on a whiteboard before you write any code. The teams that get OL adoption wrong almost always conflated layer 2 with layer 3 ("we're going to use OpenLineage as our catalog") or layer 3 with layer 4 ("we'll just point everyone at Marquez UI").

Worked example — OpenMetadata's two ingest paths side by side

Detailed explanation. OpenMetadata accepts metadata via connectors (pull from source) and via OpenLineage events (push from emitter). Each path fills a different slot in the graph, and most teams need both for a complete picture.

Question. A platform team wants analytics.fct_orders in the OpenMetadata UI with its schema, owner, tags, and a lineage graph that shows the dbt model writing it. Outline which ingest path supplies which fields, and the order in which the paths should run.

Input.

Asset	Source of truth
Table schema (columns, types)	Snowflake `INFORMATION_SCHEMA`
Owner, tags, description	OpenMetadata UI + glossary
Lineage edge "dbt → fct_orders"	dbt run-time
Last-refreshed timestamp	dbt run-time

Code.

# 1) Connector ingest — runs as an Airflow DAG every hour
source:
  type: snowflake
  serviceName: warehouse_prod
  serviceConnection:
    config:
      type: Snowflake
      hostPort: acct.snowflakecomputing.com
      username: openmetadata_ro
      database: ANALYTICS
sink:
  type: metadata-rest
  config: {}

# 2) OpenLineage ingest — runs as a webhook the dbt CLI POSTs to
# Configure dbt to emit OL events to OpenMetadata's OL endpoint:
# OPENLINEAGE_URL=https://openmetadata.example.com
# OPENLINEAGE_ENDPOINT=/api/v1/openlineage

Step-by-step explanation.

The Snowflake connector enumerates every table in ANALYTICS, reads INFORMATION_SCHEMA for column types, and pushes Table entities into OpenMetadata. fct_orders appears in the UI but without lineage edges yet.
The dbt OL emitter fires on every dbt run and POSTs OL events to OpenMetadata's /api/v1/openlineage endpoint. OpenMetadata converts each event into a Pipeline entity and creates lineage edges from inputs to outputs.
After both paths have run, fct_orders appears in the UI with its full schema and the upstream edge from the dbt Pipeline. The user adds the owner and tags manually (or by API) — those metadata are catalog-native and not in any source system.
Order matters: the connector must run first so that the Table entity exists before the OL event tries to create the lineage edge. If the order is reversed, OpenMetadata creates a placeholder Table from the OL dataset reference and fills in real schema on the next connector pass.

Output (assembled OpenMetadata UI panel).

Field	Source
Name `analytics.fct_orders`	Snowflake connector
Columns + types	Snowflake connector
Tags `PII::masked`, `Domain::Finance`	Manual + glossary
Lineage upstream `dbt.run_fct_orders`	OpenLineage event
Last refresh timestamp	OpenLineage event

Rule of thumb. Run the connector hourly (or on a metadata-change CDC if available); run OpenLineage continuously (per task). Mixing the two cadences gives you both static asset inventory and live runtime lineage at the cost each path implies.

Data engineering interview question on partitioning the stack between OL and OM

A senior interviewer might say: "Walk me through which problems you would solve with OpenLineage and which with OpenMetadata if you were designing a metadata platform from scratch in 2026."

Solution Using a layered responsibility split

# Stack ownership matrix
LAYER                     OWNER PROJECT             ARTIFACT
emitters (per tool)       OpenLineage integration   one OL event per run
wire format               OpenLineage spec          JSON event with facets
transport                 OL HTTP / Kafka client    POST / produce
durable store             OpenMetadata or DataHub   Postgres + Elasticsearch
catalog entities          OpenMetadata schemas      Table, Pipeline, Dashboard
search + UI               OpenMetadata UI           browse, search, lineage view
governance                OpenMetadata Glossary     terms, classifications, PII
data quality              OM Test Suite             test cases + results entity
runtime lineage           OL events ingested by OM  edges populated from facets
freshness alerts          downstream consumer       Slack bot or vendor receiver

Step-by-step trace.

Concern	OpenLineage	OpenMetadata
run/job/dataset events	OL spec	consumes
schema + classifications	dataset facet (per event)	first-class entity
glossary + business terms	n/a	Glossary entity
lineage graph storage	n/a	yes
catalog search UI	n/a	yes
connectors to BI / Kafka	n/a	yes
extensible custom metadata	facets	extension API
transport multicast	yes (HTTP / Kafka)	n/a

The split makes the role of each project clear: OL owns the protocol and the runtime events; OM owns the application, the catalog entities, and the user experience. They meet at the OpenLineage endpoint where OM consumes events.

Output:

Decision	Project
What format do my emitters speak?	OpenLineage
Where do events live for the long term?	OpenMetadata (or DataHub)
Where do humans browse the catalog?	OpenMetadata UI
Where do I add a glossary or PII tags?	OpenMetadata
Which project do I configure transport on?	OpenLineage client

Why this works — concept by concept:

Separation of concerns — wire formats and applications evolve on different cadences; coupling them slows both. The four-layer stack is the architecture pattern that makes the metadata platform sustainable.
Backend portability — by treating OL as the protocol, you can replace Marquez with OpenMetadata, OpenMetadata with DataHub, or DataHub with a vendor without changing a single emitter.
Catalog ownership — OpenMetadata owns the entities (Table, Pipeline, Dashboard, MLModel, Glossary, Tag) and the policies that govern them; OL contributes the lineage edges between those entities.
Custom metadata via facets — anything you cannot express in the core OL schema goes into a custom facet. The receiver chooses whether to surface it. No forking required.
Transport choices — HTTP for simple setups, Kafka for high-volume production stacks where you want durability and replayability between emitters and backends.
Cost — protocol design plus catalog design happens once; daily operations are O(events) and dominated by Postgres + Elasticsearch in the backend. The architecture itself is cheap; the content is where the value lives.

DE
Topic — design
System design problems for data engineers

Practice →

3. The OpenLineage event model

`run`, `job`, `dataset`, `facets` — four nouns that capture every transformation in your stack, and the column-level facet is where the modern open data catalog gets its impact-analysis superpower

The mental model in one line: every OpenLineage event is a tuple (run, job, inputs, outputs, facets) describing one execution attempt of one unit of work. Once you can name the four nouns and the four run states (START, COMPLETE, FAIL, ABORT), the entire OL spec collapses to "fill in the right facets for your use case."

The four core entities.

run — a single execution attempt. Has a runId (UUID) plus optional facets for parent run, nominal time, error message.
job — the unit of work itself, independent of any single execution. Identified by (namespace, name). The job is stable across runs; runs come and go.
dataset — an input or output of the job. Identified by (namespace, name). Examples: (warehouse, raw.orders), (s3, bucket-name/path/prefix).
facets — optional, extensible blocks of typed metadata attached to runs, jobs, or datasets. The whole spec is extended through facets, not by changing the core schema.

Run states.

START — the run has begun. Receivers create an open run record.
COMPLETE — the run finished successfully. Receivers close the run and finalise edges.
FAIL — the run failed. Edges may be marked attempted; downstream consumers can alert.
ABORT — the run was killed (timeout, manual stop). Treated like FAIL by most receivers but the cause is different.

Standard facets you will use every day.

schemaFacet — attached to a dataset; lists columns and types. Lets a receiver know the shape of the data at the moment of the event.
sourceFacet — attached to a dataset; identifies the physical storage system (Snowflake, S3, Kafka topic). Helps backends group datasets by source.
sqlFacet — attached to a job; the exact SQL text the job ran. Powers query-level lineage for SQL engines.
columnLineageFacet — attached to an output dataset; maps each output column to the input columns it was derived from. The single most valuable facet for impact analysis.
dataQualityFacet — attached to a dataset; expected/actual stats (row count, null ratio, distinct count). Powers freshness and quality observability.
ownershipFacet — attached to a job or dataset; team or person responsible. Lets receivers route alerts.
parentRunFacet — attached to a run; reference to a parent run (e.g. an Airflow DAG run that contains a dbt task run). Lets the graph render hierarchically.

How Airflow, Spark, dbt, and Flink emit events.

Airflow. The OL Airflow plugin instruments every task. Each task emits a START on pre_execute and a COMPLETE / FAIL on post_execute. Operator-specific extractors fill in inputs and outputs (e.g. SnowflakeOperator knows what tables the SQL touches).
dbt. The OL dbt adapter wraps dbt run. After each model materialises, it emits an event with inputs (refs) and outputs (the model's relation). The sqlFacet carries the compiled SQL; the columnLineageFacet is derived from dbt's manifest.
Spark. The OL Spark listener hooks into the SparkSession. On each query execution, it walks the logical plan to extract input and output dataset references and emits a START / COMPLETE pair.
Flink. The OL Flink integration emits per-job events with stream sources and sinks as input / output datasets. Useful for keeping the streaming side of the graph aligned with the batch side.

Column-level lineage via the columnLineage facet.

The facet maps each output column to a list of (input dataset, input column, transformation type) tuples. Receivers render this as a column-level graph in the lineage UI. For a SQL job, the facet is computed by SQL-parsing the query plan (sqlglot, Calcite, or the engine's native parser). For a dbt model, the facet can be derived from dbt's manifest and ref() graph.

Custom facets — when and how.

When. You have metadata that does not fit the standard facets but is useful to your platform. Examples: a securityClassificationFacet, a costFacet (compute units consumed), a lineageQualityFacet (confidence score).
How. Declare a JSON Schema for the facet under a unique URI (e.g. https://your-org.com/openlineage/cost.json). Emit it inline. Receivers either render it or ignore it — no breaking changes either way.

The wire format itself in one paragraph.

Every event is a JSON object with mandatory fields eventType, eventTime, run.runId, job.namespace, job.name, plus optional inputs[], outputs[], and producer. Facets sit under run.facets, job.facets, or per-dataset facets. The schema is versioned via the top-level schemaURL. Receivers ignore facets they do not understand, which makes spec evolution painless.

Common interview probes on the event model.

"What is the difference between a job and a run?" — the job is the recipe; the run is one execution attempt. Multiple runs share a job; runs are immutable post-completion.
"Can I emit OL without inputs and outputs?" — yes (the event still describes the run), but the lineage edge is empty. You lose the main reason to emit at all.
"How does OL handle streaming jobs that never complete?" — periodic checkpoint events with a START at startup and intermittent COMPLETE markers (or no terminal event), with the Flink integration's convention being a long-lived run that receives status updates.
"What stops a facet from being misused?" — JSON Schema validation. Receivers validate facets against their declared schemas; malformed facets are dropped or quarantined.

Worked example — anatomy of a single dbt OL event

Detailed explanation. Reading one real event end-to-end is the fastest way to internalise the spec. The example below is the COMPLETE event for a dbt model that joins raw.orders to raw.customers and writes analytics.fct_orders.

Question. Annotate the event with which field powers which UI feature. Identify the four mandatory fields, the input/output datasets, the SQL facet, the schema facet, and the column-lineage facet.

Input.

Field	Value
job	`analytics.fct_orders`
run id	`c8b3-2026-06-15-01`
inputs	`raw.orders`, `raw.customers`
outputs	`analytics.fct_orders`

Code.

{
  "eventType": "COMPLETE",
  "eventTime": "2026-06-15T01:08:24.000Z",
  "run": { "runId": "c8b3-2026-06-15-01" },
  "job": {
    "namespace": "analytics",
    "name": "fct_orders",
    "facets": {
      "sql": {
        "_producer": "https://github.com/OpenLineage/OpenLineage/tree/1.20.0/integration/dbt",
        "_schemaURL": "https://openlineage.io/spec/facets/1-0-0/SqlJobFacet.json",
        "query": "select o.order_id, o.amount, c.country from raw.orders o join raw.customers c using (customer_id)"
      }
    }
  },
  "inputs": [
    { "namespace": "warehouse", "name": "raw.orders",
      "facets": {
        "schema": { "fields": [
          {"name": "order_id", "type": "BIGINT"},
          {"name": "customer_id", "type": "BIGINT"},
          {"name": "amount", "type": "NUMERIC"}
        ]}
      }
    },
    { "namespace": "warehouse", "name": "raw.customers",
      "facets": {
        "schema": { "fields": [
          {"name": "customer_id", "type": "BIGINT"},
          {"name": "country", "type": "STRING"}
        ]}
      }
    }
  ],
  "outputs": [
    { "namespace": "warehouse", "name": "analytics.fct_orders",
      "facets": {
        "schema": { "fields": [
          {"name": "order_id", "type": "BIGINT"},
          {"name": "amount", "type": "NUMERIC"},
          {"name": "country", "type": "STRING"}
        ]},
        "columnLineage": {
          "fields": {
            "order_id": { "inputFields": [
              {"namespace": "warehouse", "name": "raw.orders", "field": "order_id"}
            ]},
            "amount":   { "inputFields": [
              {"namespace": "warehouse", "name": "raw.orders", "field": "amount"}
            ]},
            "country":  { "inputFields": [
              {"namespace": "warehouse", "name": "raw.customers", "field": "country"}
            ]}
          }
        }
      }
    }
  ],
  "producer": "https://github.com/OpenLineage/OpenLineage/tree/1.20.0/integration/dbt"
}

Step-by-step explanation.

The mandatory fields eventType, eventTime, run.runId, job.namespace, and job.name define the run identity. Receivers reconcile START + COMPLETE pairs by runId.
inputs[] and outputs[] declare the lineage edge. The two inputs (raw.orders, raw.customers) feed the single output (analytics.fct_orders). Marquez and OpenMetadata draw this as two arrows into one node.
The sqlFacet on the job carries the compiled SQL. Catalog UIs render it as a clickable code block; impact-analysis tools can SQL-parse it to derive column lineage when the emitter does not provide it natively.
The schemaFacet on each dataset lists columns and types. Catalog UIs render it as the table's schema panel at the moment of the run.
The columnLineageFacet on the output dataset is the high-value payload: it maps output.order_id to inputs.raw.orders.order_id, output.amount to inputs.raw.orders.amount, and output.country to inputs.raw.customers.country. Downstream "if I drop country from raw.customers, what breaks?" queries traverse this map.

Output (UI features powered by this event).

UI feature	Field used
Run timeline	`run.runId` + START / COMPLETE pair
Lineage graph	`inputs[]`, `outputs[]`
Schema panel	dataset `schemaFacet`
Compiled SQL viewer	job `sqlFacet.query`
Column-level lineage view	output `columnLineageFacet.fields`
Producer attribution	top-level `producer`

Rule of thumb. When integrating a new tool, start with the mandatory fields plus schemaFacet. Add sqlFacet next (cheap to capture). columnLineageFacet last — it is the most valuable but the most work to compute correctly.

Worked example — dbt → Airflow → Spark chained run via parent facets

Detailed explanation. Real production lineage usually crosses tools. A scheduled Airflow DAG runs a dbt step that calls a Spark job. Each tool emits its own OL event; the chain is reconstructed via the parentRunFacet. The result is a single hierarchical graph spanning all three tools.

Question. Sketch the three OL events emitted when Airflow's DAG nightly schedules a dbt task dbt_run which kicks off Spark job etl_orders. Show the parent-run references that link the graph.

Input.

Tool	Run id	Parent run id
Airflow DAG `nightly`	`a-001`	—
dbt task `dbt_run`	`d-001`	`a-001`
Spark job `etl_orders`	`s-001`	`d-001`

Code.

// 1) Airflow DAG-level event
{
  "eventType": "START",
  "run": { "runId": "a-001" },
  "job": { "namespace": "airflow", "name": "nightly" }
}

// 2) dbt task event, parent = airflow DAG
{
  "eventType": "START",
  "run": {
    "runId": "d-001",
    "facets": {
      "parent": {
        "run": { "runId": "a-001" },
        "job": { "namespace": "airflow", "name": "nightly" }
      }
    }
  },
  "job": { "namespace": "analytics", "name": "dbt_run" }
}

// 3) Spark job event, parent = dbt task
{
  "eventType": "START",
  "run": {
    "runId": "s-001",
    "facets": {
      "parent": {
        "run": { "runId": "d-001" },
        "job": { "namespace": "analytics", "name": "dbt_run" }
      }
    }
  },
  "job": { "namespace": "etl", "name": "etl_orders" }
}

Step-by-step explanation.

Airflow emits the outer event for the DAG run with id a-001. This becomes the top-level node in the lineage graph.
dbt's emitter knows it was invoked from inside an Airflow task — the integration reads the OPENLINEAGE_PARENT_* environment variables to construct the parentRunFacet, pointing back to a-001.
Spark's emitter, when invoked from a dbt python model or external task, similarly reads the parent context and constructs a facet pointing back to d-001.
Receivers reconstruct the tree: a-001 is the root; d-001 is a child of a-001; s-001 is a grandchild via d-001. The Marquez and OpenMetadata UIs render this hierarchically with collapsible sub-runs.

Output (graph structure).

Level	Run id	Job
root	`a-001`	`airflow.nightly`
child	`d-001`	`analytics.dbt_run`
grandchild	`s-001`	`etl.etl_orders`

Rule of thumb. Always propagate parent run context through environment variables when one tool launches another. Without it, the graph fragments into disconnected islands and the "what triggered this job?" question becomes hard to answer.

Worked example — emitting a custom facet for compute cost

Detailed explanation. Sometimes the standard facets do not cover a metric your team needs. The OL spec lets you declare a custom facet under your own URI. The receiver either renders it or ignores it — both are safe behaviours.

Question. Define a computeCostFacet that carries CPU seconds and dollar cost for each run, and attach it to a Spark COMPLETE event. Show the facet payload and the receiver's options for displaying it.

Input.

Field	Value
CPU seconds	124.5
Estimated cost USD	0.42
Cluster id	`spark-cluster-prod-01`

Code.

{
  "eventType": "COMPLETE",
  "run": {
    "runId": "s-001",
    "facets": {
      "computeCost_dataeng_example_com": {
        "_producer": "https://github.com/dataeng-example/openlineage-cost",
        "_schemaURL": "https://dataeng.example.com/openlineage/computeCost.json",
        "cpu_seconds": 124.5,
        "estimated_cost_usd": 0.42,
        "cluster_id": "spark-cluster-prod-01"
      }
    }
  },
  "job": { "namespace": "etl", "name": "etl_orders" }
}

Step-by-step explanation.

The facet key is namespaced (computeCost_dataeng_example_com) so it cannot collide with any future standard facet.
The mandatory facet fields _producer and _schemaURL let receivers identify the source and validate the payload. Receivers that do not recognise the schema simply ignore the facet — no breaking change.
The payload itself is arbitrary JSON conforming to the schema at _schemaURL. The schema lives in your org's repo and is referenced by URL — receivers can fetch and validate at runtime, or trust the producer.
Receivers like OpenMetadata render unknown facets either as raw JSON in a "raw facets" panel or, with a custom plugin, as a typed widget. Marquez stores them in its facets table for later querying.

Output (UI surfaces).

Receiver	Treatment
Marquez	persisted in facets table; queryable via REST
OpenMetadata	rendered in raw facets panel; surfaced via custom widget if installed
Monte Carlo	ignored (does not know the schema)
Custom Spark cost dashboard	consumes via Kafka feed; renders as a chart

Rule of thumb. Use custom facets sparingly. If three teams independently invent the same facet, lobby for it to become a standard. The OpenLineage community has accepted multiple originally-custom facets into the spec over the past two years.

Data engineering interview question on minimum-viable lineage instrumentation

A senior interviewer might frame this as: "You have an existing Airflow + dbt + Spark stack and zero lineage today. Walk me through the smallest first deployment of OpenLineage that delivers useful lineage in two weeks."

Solution Using emitters-first, single-backend rollout

WEEK 1
- Stand up Marquez via docker-compose in staging.
  POSTGRES + the marquez-web container.
- Install the OL Airflow plugin on the staging Airflow.
  Set OPENLINEAGE_URL=http://marquez:5000.
- Verify lineage events arrive by running one staging DAG.
- Add the dbt OL adapter to the dbt project.
  Run `dbt build` against staging; confirm events appear.

WEEK 2
- Add the OL Spark listener to staging Spark cluster config.
  spark.extraListeners=io.openlineage.spark.agent.OpenLineageSparkListener
- Run the most important production DAG once in staging
  with realistic data; capture the full lineage graph.
- Promote OL configuration to production for one team's pipelines
  with feature flag, monitor Marquez for two days.
- Plan week 3 for OpenMetadata or DataHub as second consumer.

Step-by-step trace.

Day	Action	Outcome
1	docker-compose up Marquez	running locally
2	install Airflow OL plugin	events arriving from staging Airflow
3	install dbt OL adapter	events arriving from staging dbt
4	run end-to-end staging DAG	lineage graph visible in Marquez UI
5–7	iterate, fix missing extractors	graph passes peer review
8	enable Spark listener	Spark jobs join the graph
9	flag one prod team	prod lineage flowing
10–14	monitor, fix gaps	stable for one team

By week three, the team can choose to layer OpenMetadata or DataHub as a second consumer of the same OL events without touching the emitters. The migration cost is configuring the new backend's OL endpoint, not re-instrumenting the pipelines.

Output:

Milestone	Week
Marquez running, Airflow events flowing	1
dbt events flowing	1
Spark events flowing	2
One prod team fully instrumented	2
Second backend (OpenMetadata) as additional consumer	3

Why this works — concept by concept:

Marquez first, catalog second — Marquez is the cheapest credible OL backend. Standing it up validates the emitters before you spend weeks on OpenMetadata schemas and connectors.
Per-tool integration — each emitter (Airflow, dbt, Spark) plugs into its native lifecycle. No code changes to pipelines; the integration owns the event generation.
Feature flag in prod — emitter overhead is small but real (one HTTP call per task). Roll out by team so any regression is contained.
Two-week MVP — the metric that matters is "first useful lineage graph visible to humans." Everything beyond that (column lineage, facets, glossary) layers on without touching the foundation.
Backend swap is cheap — by week three, switching from Marquez to OpenMetadata is "point the OL URL at the new endpoint." This is exactly the portability the standard buys you.
Cost — staging infra plus ~3 engineer-weeks for the MVP; prod onboarding is incremental per team thereafter.

DE
Topic — event modeling
Event modeling problems for lineage and audit

Practice →

4. OpenMetadata architecture and entity model

OpenMetadata is a catalog application with three layers — Ingestion, Metadata Server, UI — and a unified entity model spanning tables, dashboards, pipelines, topics, and ML models

The mental model in one line: OpenMetadata is "a single catalog DB plus a REST API plus a UI plus a connector framework," and every metadata concern (lineage, governance, quality, glossary, classification) is a first-class entity in that DB. Once you internalise that "everything is an entity," the API surface and the UI both make obvious sense.

Three-layer architecture.

Ingestion. Connectors run as Airflow DAGs (or as standalone Python apps) and push entity records into the metadata server. The ingestion framework is open and pluggable — adding a connector is writing a Python class that conforms to the source / sink interface.
Metadata server. The heart. A Java service (Dropwizard) exposing a REST API; backed by Postgres or MySQL for storage and Elasticsearch (or OpenSearch) for search. Defines the entity schemas, the policies, and the lineage graph queries.
UI. A React app that calls the REST API. Renders entity pages, search, the lineage graph, the glossary, data quality results, and admin pages.

The unified entity model.

OpenMetadata models everything as an entity. The same patterns (versioning, tagging, ownership, lineage) apply across:

Database, Schema, Table. Tables across Snowflake, BigQuery, Postgres, MySQL, etc., live as Table entities under a DatabaseService → Database → DatabaseSchema → Table hierarchy.
Pipeline. Airflow DAGs, dbt projects, Dagster pipelines — each becomes a Pipeline entity. Lineage edges connect Pipelines to Tables (read / write).
Dashboard, Chart. Looker / Tableau / Metabase dashboards become Dashboard entities, with each tile or chart as a Chart sub-entity.
Topic. Kafka / Pulsar / Kinesis topics become Topic entities with schema and ownership.
MLModel, Container. ML models (MLflow / SageMaker) become MLModel entities; storage containers (S3 / GCS / Azure) become Container entities.
Glossary, GlossaryTerm. Business vocabulary lives in Glossary and GlossaryTerm entities, which can be linked as tags on any other entity.
Tag, Classification. PII tags, data sensitivity classifications, and domain tags all live as Tag entities under Classification parents.
TestSuite, TestCase, TestCaseResult. Data quality is first-class: TestCase definitions and their run results are entities that the UI renders alongside the table.

Ingestion framework in detail.

Connectors. One per source (Snowflake, BigQuery, Postgres, MySQL, Trino, Redshift, Tableau, Looker, PowerBI, Kafka, Airflow, dbt, MLflow). Each connector reads from the source via its native API and yields OpenMetadata entity records.
Workflow types. Metadata (entities + schema), Lineage (edges from query history), Profiler (column stats), Data Quality (test runs), Usage (query history for popularity), dbt (parses manifest.json), Application Settings (admin).
Scheduling. Workflows run as Airflow DAGs that come pre-bundled with OpenMetadata's ingestion image. Production teams typically point them at their own Airflow.

The metadata server's data model.

Postgres stores entity rows, versions, and relationships.
Elasticsearch stores the search index for each entity type plus the autocomplete index.
REST API at /api/v1/* exposes every entity type. Filtering, search, and lineage queries all live here.

UI features.

Search. Full-text plus typed filters (entity type, service, tier, owner, tag).
Lineage graph. Bidirectional graph view with table-level and column-level depth controls.
Glossary. Hierarchical business vocabulary; terms can be assigned to tables, columns, dashboards.
Data quality. Test results render inline with each table; failing tests can route to Slack.
Profiling. Column-level statistics (null %, distinct %, distributions) computed by the Profiler workflow.
Roles, policies. Fine-grained access — who can read / edit / delete which entity types.
PII tagging. Auto-classification of columns based on data and naming patterns; manual override via the UI.

How OL events flow into OpenMetadata.

OpenMetadata exposes an OpenLineage endpoint at /api/v1/openlineage. Each arriving event is translated into:

A Pipeline entity (created if absent, looked up by namespace + name).
Lineage edges from the listed input datasets to the listed output datasets.
Column-level lineage edges if the event carries a columnLineageFacet.
Pipeline status entries reflecting the run's success or failure.

The translation is convention-driven (dataset namespace warehouse maps to the DatabaseService named warehouse_prod, etc.) and configurable via the connector settings.

Self-hosted vs Collate.

Self-hosted. Docker / Kubernetes Helm chart; you operate Postgres, Elasticsearch, and the metadata server. Cost is infra plus part-time platform-engineering work.
Collate. Commercial managed offering from the same team. Hosted multi-tenant; eliminates the operational burden in exchange for per-asset pricing similar to other vendors.

Common interview probes on OpenMetadata.

"What is the difference between a Table and a Topic entity?" — both are dataset-like, but Table maps to a relational warehouse and Topic to an event-stream; lineage edges treat them the same.
"Where does PII classification come from?" — automatic classifiers run during the metadata or profiler workflow; manual overrides via the UI. Both produce Tag entities attached to the column.
"How does OpenMetadata handle column-level lineage?" — column edges live as part of the Table entity; the UI renders them as a sub-graph inside the lineage panel.
"Can OpenMetadata be the source of truth for ownership?" — yes — the ownership field on each entity is canonical and propagates to downstream alerting via webhooks.

Worked example — modeling a Snowflake table with full metadata

Detailed explanation. Walking through the entity payload for one table makes the model concrete. Below is the JSON shape stored for analytics.fct_orders after the connector ingests it.

Question. Build the Table entity for warehouse_prod.ANALYTICS.fct_orders with three columns, an owner team, a Finance domain tag, a PII tag on one column, and a glossary term link. Identify which fields are connector-supplied and which are user-curated.

Input.

Field	Value	Source
Name	`fct_orders`	Snowflake
Database	`WAREHOUSE_PROD.ANALYTICS`	Snowflake
Columns	`order_id`, `amount`, `customer_email`	Snowflake
Owner	team `analytics-eng`	Manual
Domain tag	`Domain.Finance`	Manual
PII tag on `customer_email`	`PII.Sensitive`	Auto-classifier
Glossary term	`Finance.GMV` linked	Manual

Code.

{
  "name": "fct_orders",
  "fullyQualifiedName": "warehouse_prod.ANALYTICS.fct_orders",
  "service": "warehouse_prod",
  "database": "WAREHOUSE_PROD",
  "databaseSchema": "ANALYTICS",
  "columns": [
    {"name": "order_id", "dataType": "BIGINT"},
    {"name": "amount",   "dataType": "NUMERIC"},
    {"name": "customer_email",
     "dataType": "STRING",
     "tags": [{"tagFQN": "PII.Sensitive", "labelType": "Automated"}]}
  ],
  "owner": {"id": "team-analytics-eng", "type": "team"},
  "tags": [{"tagFQN": "Domain.Finance", "labelType": "Manual"}],
  "glossaryTerms": [{"id": "gt-finance-gmv", "type": "glossaryTerm"}]
}

Step-by-step explanation.

The Snowflake connector populates name, fullyQualifiedName, service, database, databaseSchema, and the column list with names and types. Connector-supplied fields are versioned and re-synced on every ingestion run.
The auto-classifier (part of the metadata workflow) inspects column names and sample data. It tags customer_email with PII.Sensitive and records labelType: Automated so reviewers can distinguish auto from manual labels.
A platform admin (or a steward in the UI) assigns the team owner. Owner propagates to all downstream alerts: failing tests, freshness violations, and OpenLineage FAIL events route to the team.
The domain tag Domain.Finance and the glossary term link are manual. They make the table discoverable via filtered search ("show me every Finance table") and tie business vocabulary to physical assets.

Output (rendered Table entity page).

Panel	Content
Schema	`order_id BIGINT`, `amount NUMERIC`, `customer_email STRING (PII)`
Owner	`analytics-eng`
Tags	`Domain.Finance`, `PII.Sensitive` (on column)
Glossary	`Finance.GMV`
Lineage	upstream from `dbt.fct_orders`, downstream to BI
Quality	last 3 test runs and freshness metric

Rule of thumb. Let the connector own everything mechanical (names, types, sizes, freshness timestamps); let humans own everything contextual (owner, domain, glossary). Auto-classifiers sit in the middle — let them propose, let stewards approve.

Worked example — converting an OL event into an OpenMetadata Pipeline + lineage

Detailed explanation. When an OpenLineage event arrives at OpenMetadata's /api/v1/openlineage endpoint, the server converts it into one Pipeline entity plus lineage edges. Walking the conversion makes the integration tangible.

Question. Trace the conversion for the dbt event from Section 3 (analytics.fct_orders reading raw.orders and raw.customers). Identify which entities are created and which edges are upserted.

Input.

OL field	Becomes in OM
`job.namespace + job.name`	Pipeline FQN
`inputs[]`	source nodes for edges
`outputs[]`	target nodes for edges
`columnLineageFacet`	column-level edges
`run.runId` + `eventType`	PipelineStatus entry

Code.

Incoming OL event
  job = analytics.fct_orders
  inputs = [warehouse.raw.orders, warehouse.raw.customers]
  outputs = [warehouse.analytics.fct_orders]
  columnLineage = {order_id: [raw.orders.order_id], ...}

Conversion
  ensure Pipeline entity "analytics.fct_orders" exists
  ensure Table "warehouse_prod.raw.orders" referenced
  ensure Table "warehouse_prod.raw.customers" referenced
  ensure Table "warehouse_prod.analytics.fct_orders" referenced
  upsert lineage edge: raw.orders -> analytics.fct_orders
  upsert lineage edge: raw.customers -> analytics.fct_orders
  upsert column-level edge: raw.orders.order_id -> analytics.fct_orders.order_id
  upsert column-level edge: raw.orders.amount   -> analytics.fct_orders.amount
  upsert column-level edge: raw.customers.country -> analytics.fct_orders.country
  append PipelineStatus: runId=c8b3-2026-06-15-01, state=Successful

Step-by-step explanation.

The server looks up the Pipeline by (service, namespace, name). If absent, it is created with the OL producer as the service hint. Subsequent events update the same Pipeline rather than create duplicates.
The input and output datasets are mapped to Table entities by FQN convention. Datasets that do not yet exist (because the table connector has not run) are created as placeholder Tables and enriched later when the connector pass arrives.
The lineage edges are upserted. Re-running the same event is idempotent — no duplicate edges. This is critical: every COMPLETE event in production carries the same edges, and the storage must collapse them.
The column lineage facet drives the column-level edges. The UI renders them as a sub-graph inside the table-level edge; users toggle "column lineage" to drill in.
The PipelineStatus entry records the run's outcome with timestamps. The Pipeline page displays a run history; failing runs annotate the connected tables with "last run failed."

Output.

Entity	Action
Pipeline `analytics.fct_orders`	upserted
Table `raw.orders`	placeholder upserted
Table `raw.customers`	placeholder upserted
Table `analytics.fct_orders`	placeholder upserted
Lineage edge (table)	2 upserted
Lineage edge (column)	3 upserted
PipelineStatus	1 appended

Rule of thumb. Run the database connectors before expecting OL ingestion to fill the catalog. The connectors give you the entity inventory; OL gives you the lineage edges. Run them in the right order and your catalog is complete on day one.

Worked example — wiring data quality results into the catalog

Detailed explanation. OpenMetadata's TestSuite and TestCase entities make data quality first-class — every table can carry a list of tests, each test has a definition (e.g. "row count > 0"), and each test run produces a TestCaseResult that the UI surfaces inline. The same model accepts results from external tools via REST.

Question. Define a TestSuite for analytics.fct_orders with three tests (row count, distinct customer count, freshness), and show how a test runner posts results to the catalog.

Input.

Test	Expectation
row_count_min	rows > 0
distinct_customers_min	unique `customer_id` > 100
freshness	data updated within 24h

Code.

// 1) Create the TestSuite
POST /api/v1/dataQuality/testSuites
{
  "name": "fct_orders_quality",
  "entity": {"id": "table-id-fct-orders", "type": "table"}
}

// 2) Define a TestCase
POST /api/v1/dataQuality/testCases
{
  "name": "row_count_min",
  "entityLink": "<#E::table::warehouse_prod.ANALYTICS.fct_orders>",
  "testDefinition": "tableRowCountToBeBetween",
  "parameterValues": [
    {"name": "minValue", "value": "1"}
  ],
  "testSuite": "fct_orders_quality"
}

// 3) After running the test, post the result
POST /api/v1/dataQuality/testCases/testResults
{
  "testCaseFQN": "warehouse_prod.ANALYTICS.fct_orders.row_count_min",
  "result": "Success",
  "timestamp": 1718492400000,
  "testResultValue": [{"name": "rowCount", "value": "248913"}]
}

Step-by-step explanation.

The TestSuite is the container for tests on one entity. Each table can have one TestSuite that aggregates its tests; failing tests on any case roll up to a suite-level health indicator.
The TestCase definition references a testDefinition (a built-in or custom test type) plus parameters. The platform ships a library of definitions like tableRowCountToBeBetween, columnValuesToBeUnique, tableFreshnessSLA, plus a custom SQL test.
The result is posted by whoever runs the test — OpenMetadata's own profiler workflow, an external dbt test run, a Great Expectations run, or a custom script. The same REST API accepts results from any source.
The UI surfaces the latest result inline on the table page, with a colour-coded badge (green / amber / red). Failing tests can trigger webhooks to Slack or PagerDuty via OpenMetadata's alerting system.

Output (table page UI).

Test	Last result	Last run
row_count_min	Success	2026-06-15 03:00
distinct_customers_min	Success	2026-06-15 03:01
freshness	Failed	2026-06-15 03:02

Rule of thumb. Treat the test results as another lineage signal — a failing freshness test on a source table is exactly the information a downstream consumer needs before reading. Surface them inline in the lineage graph, not on a separate dashboard.

Data engineering interview question on adopting OpenMetadata across a 50-team org

A senior interviewer might frame this as: "You have OpenMetadata running for one team. How do you scale it to 50 teams without it becoming a dumping ground of stale entities?"

Solution Using domain-scoped ingestion + steward ownership

SCALING PLAN

1. Domain-scope the catalog
   - Each business domain (Finance, Marketing, Product, Platform)
     gets its own DatabaseService prefix and Glossary scope.
   - Tags use Domain.* hierarchy so search is domain-filterable.

2. Steward per domain
   - Every domain nominates a data steward.
   - Stewards own glossary terms, tag policies, and PII reviews
     for assets in their domain.

3. Connector cadence by tier
   - Tier-1 assets (production warehouse, dashboards): hourly
   - Tier-2 (staging, lab): daily
   - Tier-3 (sandboxes): weekly or on-demand
   - Tier classification is itself a Tag entity.

4. Lineage from OL is continuous
   - Airflow + dbt + Spark + Flink emit OL events.
   - Per-team OL endpoints converge in one OM instance.

5. Quality tests gated by tier
   - Tier-1 tables MUST have row_count + freshness + uniqueness
   - Tier-2 SHOULD have at least one custom test
   - Tier-3 OPTIONAL.

6. PII review SLA
   - Auto-classifier proposes; steward approves within 14 days.
   - Unreviewed PII tags flagged on the steward dashboard.

7. Stale asset reaping
   - Assets without ingestion for 30 days auto-archived
     unless explicitly pinned.

Step-by-step trace.

Step	Owner	Cadence	Output
1 domain scope	platform	once	DatabaseService + Glossary roots
2 stewards	data leadership	once + on join	named steward per domain
3 connector cadence	platform + team	continuous	per-tier ingestion DAGs
4 OL emitters	each team	continuous	runtime lineage
5 tier-gated tests	each team	per release	failing tests block deploy
6 PII review	steward	14-day SLA	tags approved or rejected
7 archive	platform	weekly	clean catalog

The result is a catalog where every entity has a known owner, a known tier, and a known refresh expectation. Search returns relevant assets first because tier and domain are filterable.

Output:

Health metric	Target
Tier-1 coverage by tests	100%
Domain assignment completeness	> 95%
Stale entities (no refresh in 30d)	< 2%
PII auto-tags unreviewed > 14 days	0
OL events per minute (steady state)	proportional to pipeline count

Why this works — concept by concept:

Domain scoping — Domain.* tag hierarchy gives the catalog a top-down structure that mirrors how the org thinks about data, and lets stewards own their slice without blocking each other.
Steward ownership — putting humans at the leaf of every policy decision (glossary, PII, classification) is the only way a catalog survives at scale. Auto-classification proposes; humans dispose.
Tier-driven cadence — not every asset deserves hourly metadata. Tiering keeps the ingestion pipeline cheap and the catalog signal-to-noise high.
Continuous OL ingestion — runtime lineage is the always-fresh part of the graph; static connectors fill in the shape; together they keep the catalog accurate.
Stale-asset reaping — a catalog that grows monotonically becomes useless. Archive policies keep search focused on assets that still matter.
Cost — connectors scale O(assets); OL events scale O(pipeline runs). Postgres + Elasticsearch sized to those rates plus an FTE fraction per few hundred TB of source metadata.

DE
Topic — dimensional modeling
Dimensional modeling problems for warehouses

Practice →

5. Interop with proprietary vendors and migration patterns

OpenLineage is the migration off-ramp from Atlan, Collibra, Alation, and Monte Carlo — emit once, route to whichever backend wins this quarter, and use the two-write pattern to stage the cutover

The mental model in one line: as long as OpenLineage events leave your pipelines, the choice of backend is a configuration change, not an architecture change. Once your team can quote that invariant, the conversation with the closed-catalog vendor on renewal day becomes very different — and the migration plan can be incremental rather than Big Bang.

Where vendors plug into the OpenLineage event stream.

Monte Carlo. Accepts OL events as a lineage input. Layers freshness, volume, and schema-change anomaly detection on top of the same graph your open backend sees.
Atlan. Has a documented OL adapter; ingests events into the Atlan graph and renders them inline with vendor-curated metadata.
Bigeye. Similar to Monte Carlo — OL events feed the observability layer.
Collibra. Accepts OL events for technical lineage; business-glossary side stays inside Collibra's model. Most teams keep Collibra for governance and use OL to keep its lineage panel current.
Alation. Accepts OL through a plugin; the business catalog stays vendor-owned while runtime lineage is single-sourced from OL.

Emit OpenLineage from Airflow / dbt / Spark and forward to vendor X.

The integration pattern is identical regardless of which vendor receives:

emitter (Airflow / dbt / Spark)
   |
   v
OPENLINEAGE_URL = http(s)://vendor-endpoint/openlineage
   |
   v
vendor receiver ingests, renders, alerts

The emitter does not know it is talking to a vendor. The vendor does not know it is reading a community-format event. The standard makes both sides plug-and-play.

Multi-cast to two or more receivers.

When you want both an open backend and a vendor receiver during a migration, configure multi-cast:

Newer OL integrations accept a comma-separated OPENLINEAGE_URL.
Older integrations require a small proxy: a single FastAPI service that POSTs each event to N configured URLs.
Kafka transport turns multi-cast into "multiple consumer groups on one topic."

This is the two-write pattern: events flow to the old backend and the new one for the duration of the migration, so the new backend builds historical context before you turn the old one off.

Replace a closed catalog with OpenMetadata gradually.

A 90-day migration timeline that has worked for multiple platform teams:

Day 1–14. Stand up OpenMetadata in staging. Run connectors against the same sources the old catalog covers. Verify entity completeness against the old catalog's asset list.
Day 15–30. Enable OpenLineage emitters in production with multi-cast: events flow to both the old vendor and to OpenMetadata. Both catalogs now show identical runtime lineage.
Day 31–60. Migrate business metadata (glossary, ownership, tags) into OpenMetadata. Most vendors have an export API or a CSV bulk download; the import can be scripted via OpenMetadata's REST API.
Day 61–80. Switch primary user UI to OpenMetadata. Old vendor stays read-only as a fallback.
Day 81–90. Decommission the old vendor. The OL multi-cast configuration drops the vendor endpoint. The renewal is not signed.

DataHub vs OpenMetadata — when to pick which.

Both are credible open catalogs with active communities and similar feature surface. The choice usually comes down to ecosystem fit.

Pick OpenMetadata when — you want a broader out-of-the-box connector library, tighter integration with OpenLineage as a native ingest path, a more polished UI for end-users, or a managed offering (Collate) on the same stack.
Pick DataHub when — you want an event-native architecture under the hood (the Metadata Change Event / Metadata Audit Event model on Kafka), strong upstream propagation for downstream services, or your existing stack already has heavy Kafka investment.
Either way — OL events flow into both. The wire-format standard means you can change your mind later without re-instrumenting pipelines.

Governance integrations.

Glossary and business terms. OpenMetadata models Glossary and GlossaryTerm as entities; terms can be linked to tables, columns, dashboards. DataHub uses the GlossaryNode / GlossaryTerm model. Both let you bulk-import terms from a CSV or an external governance tool.
Data classification. Both support hierarchical tags (PII.Sensitive, PII.Email, Finance.Revenue). OpenMetadata's auto-classifier proposes tags; admins approve. DataHub uses Glossary Terms similarly.
Access policies. Role + Policy model in both: a Policy lists allowed actions on entity types matched by a rule. Roles bundle policies. Users / Teams are assigned roles.
Compliance reporting. Glossary + Tag + Classification combine into a queryable matrix: "show every column tagged PII that touches a Finance domain dashboard." Both catalogs support this via search filters; OpenMetadata also exposes the query as a REST call.

Cost picture — self-hosted vs vendor.

Stack	Year 1	Year 3	Notes
Closed vendor at $0.50/asset/mo, 10K assets	$60,000	~$225K cumulative	Grows with asset count
OpenMetadata self-hosted (4 vCPU, 16GB, 200GB DB)	~$25K infra + 0.25 FTE	~$100K cumulative	Flat-ish; FTE is bulk
Collate managed (similar to vendor)	$0.40/asset/mo	similar to vendor	Less ops overhead
Vendor receiver (Monte Carlo / Bigeye) — additive on top of any catalog	$20–60K/year typical	similar	Pays only for the observability layer, not the catalog

Long-term bets.

The OL spec is converging on column-level lineage as the default. Within two years, "OL without column lineage" will be considered a half-instrumented stack.
Vendor receivers are becoming OL-first. New observability tools launch with OL ingestion as the recommended path, not as an afterthought.
OpenMetadata and DataHub will likely both survive. They serve different architectural tastes; neither is going away.
Marquez stays the reference backend. Useful as a sanity check during migrations and as a lightweight first deployment.

Common interview probes on interop and migration.

"Can I send OpenLineage to Monte Carlo?" — yes. Configure OPENLINEAGE_URL to Monte Carlo's OL endpoint, or multi-cast.
"What is the two-write pattern?" — emit events to both the old and new backend during migration; cut over when the new backend has parity.
"How do I migrate business metadata (glossary, owners) into OpenMetadata?" — export from the old vendor (REST or CSV), import via OpenMetadata's REST API. Scriptable in a day for most orgs.
"Is column lineage automatic?" — only when the emitter produces the columnLineageFacet. dbt and Spark do; Airflow does for the operators that have extractors; custom Python is on you.

Worked example — the two-write pattern in configuration

Detailed explanation. Two-write is the safest migration shape: send every event to both backends, verify parity, then drop the old one. The configuration cost is tiny; the safety it buys is real.

Question. Configure a dbt project to emit OpenLineage events to both Atlan (the old catalog) and OpenMetadata (the new catalog) during a 60-day migration. Show the env vars or proxy required.

Input.

Endpoint	Role
Atlan	old catalog, read-only by day 60
OpenMetadata	new catalog, gaining context

Code.

# Option A — multi-URL (newer OL integrations)
export OPENLINEAGE_URL="https://atlan.example.com/openlineage,https://openmetadata.example.com/api/v1/openlineage"
export OPENLINEAGE_API_KEY_ATLAN="..."
export OPENLINEAGE_API_KEY_OM="..."

# Option B — fan-out proxy (older OL integrations)
# proxy posts every incoming event to both URLs
export OPENLINEAGE_URL="http://ol-proxy.internal:5000"

# Minimal fan-out proxy (FastAPI) — Option B
from fastapi import FastAPI, Request
import httpx

app = FastAPI()
TARGETS = [
    "https://atlan.example.com/openlineage",
    "https://openmetadata.example.com/api/v1/openlineage",
]

@app.post("/")
async def fanout(request: Request):
    body = await request.body()
    headers = {"Content-Type": "application/json"}
    async with httpx.AsyncClient(timeout=5.0) as client:
        for url in TARGETS:
            try:
                await client.post(url, content=body, headers=headers)
            except Exception:
                pass  # never block the producer
    return {"status": "ok"}

Step-by-step explanation.

Option A is the cleanest path when the OL integration supports comma-separated URLs (Airflow OL >= 1.18, dbt OL >= 1.16, Spark OL >= 1.20 with the OpenLineageClient transports config). Each URL receives every event.
Option B works with any integration. The proxy is a single ~20-line FastAPI service. It POSTs each event to every configured target, swallowing per-target failures so the producer never blocks.
The producer's view never changes during the migration. Pipelines do not know they are now talking to two backends; they POST once to the OL URL.
On migration day 60, drop one URL from the list (Option A) or remove one TARGET entry (Option B). No code change anywhere else.

Output (during the migration window).

Backend	Events received	UI status
Atlan	100%	primary (days 0–45), read-only (days 46–60)
OpenMetadata	100%	secondary (days 0–45), primary (days 46–60)

Rule of thumb. Run the two-write window for at least 30 days. The new backend needs a meaningful history before you trust it as the primary UI.

Worked example — migrating glossary terms from Collibra to OpenMetadata

Detailed explanation. Business metadata does not flow over OpenLineage — it lives in the catalog itself. Migrating it is an export + transform + import job. OpenMetadata's REST API makes the import scriptable.

Question. Migrate 500 Collibra business terms (each with a name, description, and domain) into OpenMetadata as GlossaryTerm entities under a Finance glossary. Show the script outline.

Input.

Field	Example
name	`Gross Merchandise Value`
description	`Total value of goods sold over a period.`
domain	`Finance`

Code.

import csv, requests

OM_URL = "https://openmetadata.example.com/api/v1"
OM_TOKEN = "...JWT..."
HDR = {"Authorization": f"Bearer {OM_TOKEN}",
       "Content-Type": "application/json"}

# 1) Ensure parent Glossary exists
glossary = {"name": "Finance",
            "displayName": "Finance",
            "description": "Finance domain business vocabulary"}
requests.put(f"{OM_URL}/glossaries", json=glossary, headers=HDR)

# 2) For each Collibra term, POST as GlossaryTerm
with open("collibra_export.csv") as f:
    for row in csv.DictReader(f):
        term = {
            "name": row["name"].replace(" ", "_"),
            "displayName": row["name"],
            "description": row["description"],
            "glossary": "Finance"
        }
        r = requests.put(f"{OM_URL}/glossaryTerms", json=term, headers=HDR)
        r.raise_for_status()

Step-by-step explanation.

The Collibra export is a CSV with name, description, domain columns. Standard Collibra "Export Asset List" feature.
The script ensures the parent Glossary entity exists in OpenMetadata. PUT is idempotent — re-running the script does not duplicate the Glossary.
For each row, the script POSTs (or PUTs, depending on whether you want create-or-update) a GlossaryTerm. The name field cannot contain spaces in OpenMetadata FQNs; displayName keeps the original.
Each term lands in the Finance Glossary. The terms can now be linked from tables, columns, and dashboards via the UI or programmatically.

Output.

Imported	Count
Glossary	1 (`Finance`)
GlossaryTerm	500
Links to tables	0 (next migration phase)

Rule of thumb. Migrate the glossary first, then the table-to-term links. Linking is the part that benefits most from human review — let stewards approve sample links rather than bulk-import them blindly.

Worked example — multi-cast to Marquez, OpenMetadata, and Monte Carlo

Detailed explanation. Some teams want the lineage UI of Marquez (fast to render), the catalog of OpenMetadata (governance), and the observability of Monte Carlo (anomaly detection). The OL standard makes this trivial: each backend is just another URL.

Question. Configure the fan-out proxy to deliver every OL event to Marquez, OpenMetadata, and Monte Carlo. Show the resulting graph experience for the end user.

Input.

Backend	Role
Marquez	lineage graph UI for engineers
OpenMetadata	catalog + glossary + governance
Monte Carlo	observability + freshness alerts

Code.

TARGETS = [
    "http://marquez.internal:5000/api/v1/lineage",
    "https://openmetadata.example.com/api/v1/openlineage",
    "https://api.getmontecarlo.com/openlineage",
]
# same fan-out logic as Worked example above

Step-by-step explanation.

The proxy accepts one event per task / model / job and POSTs it to all three URLs in parallel. Latency is bounded by the slowest receiver.
Marquez renders the lineage graph immediately. Engineers use it for "trace the job" deep-dives during incidents.
OpenMetadata creates a Pipeline entity and lineage edges, plus updates the affected tables. Analysts and stewards use this view.
Monte Carlo cross-references the event against learned baselines — table appeared, schema changed, row count dropped. It alerts on anomalies; the alert pages the on-call.
All three views show the same underlying facts because the source-of-truth event is the OL payload from the pipeline.

Output (per persona).

Persona	Tool	Reason
Data engineer	Marquez	clean lineage graph for debugging
Analytics engineer	OpenMetadata	catalog browsing, glossary, owners
Analyst	OpenMetadata	search, find tables, see freshness
Steward	OpenMetadata	governance, PII review
On-call	Monte Carlo	freshness / schema-change alerts

Rule of thumb. The right number of OL consumers is "however many distinct user personas you have, minus the ones whose needs overlap entirely." The marginal cost of adding a receiver is configuration; the marginal value is the persona it serves.

Data engineering interview question on planning a closed-catalog exit

A senior interviewer might frame this as: "Your CFO has asked for a plan to leave Vendor X at renewal in six months. Walk me through it from week one to cutover, in enough detail that the platform team can execute without me."

Solution Using a six-month phased migration plan

MONTH 1 — Stand up
- Deploy OpenMetadata in staging via Helm.
- Configure Postgres + Elasticsearch in dedicated VMs / managed services.
- Run all warehouse connectors (Snowflake, BigQuery, Postgres) once.
- Sanity check: entity count vs vendor's reported asset count.

MONTH 2 — Lineage
- Enable OL emitters on staging Airflow + dbt + Spark.
- Multi-cast OL events to both the vendor and OpenMetadata.
- Verify table-level + column-level lineage parity for top-20 tables.
- Document gaps; file integration bugs upstream where needed.

MONTH 3 — Business metadata
- Export glossary + owner + tags from vendor (CSV or API).
- Script the import into OpenMetadata GlossaryTerm / Tag / owner.
- Stewards review sample of 50 imports; fix mapping issues.

MONTH 4 — Quality and policy
- Define top TestSuite per Tier-1 table.
- Migrate or re-author data quality tests (dbt tests + custom SQL).
- Replicate Role / Policy model — admin / steward / read-only.

MONTH 5 — UX cutover
- Switch internal documentation links from vendor to OpenMetadata.
- Vendor UI moves to read-only mode; team is told "use OM going forward."
- Monitor support tickets, fix UX gaps, train teams.

MONTH 6 — Renewal day
- Drop vendor URL from OL multi-cast.
- Cancel vendor contract.
- Capture lessons learned for the next standards adoption (e.g. DataHub
  as second open option, or a managed Collate as an upgrade path).

Step-by-step trace.

Month	Headline deliverable	Risk	Mitigation
1	OM running, connectors green	infra sizing	start with x86-large VMs + 200GB Postgres
2	OL multi-cast in prod	emitter overhead	feature flag per team
3	Business metadata imported	term mapping errors	steward review sample
4	Quality tests live	test coverage gaps	tier-gated requirements
5	UX cutover	user pushback	early demos, training
6	Vendor decommissioned	sign-off blocking	written acceptance from each domain

The plan is designed to fail safely: at every month, if the new stack is not ready, the old vendor is still receiving events and serving as the source of truth. Cutover only happens when parity is real, not when the calendar says.

Output:

Month	Status
1	OpenMetadata running in staging
2	Lineage two-write in prod
3	Glossary imported, owners assigned
4	Quality tests + roles parity
5	UX cutover, vendor read-only
6	Vendor decommissioned at renewal

Why this works — concept by concept:

Two-write everywhere — the migration never has a "one-night switch" risk because events flow to both backends throughout. Either side can be the primary at any moment.
Connector-first, lineage-second — entities give you the inventory; OL gives you the edges. Stand them up in that order so the OL graph has nodes to attach to.
Steward review of business metadata — automated import handles 80%; humans handle the 20% with judgement calls. Stewards are the only durable defence against junk metadata.
Tier-gated quality — every Tier-1 table must have a test suite; lower tiers are optional. This keeps quality investment proportional to business impact.
UX cutover before renewal — the team must actively prefer the new UI before renewal day. If they do not, the plan slips by a month — better than slipping the renewal.
Cost — six months of platform-engineering attention (~0.5 FTE) plus ~$40K infra annually. The vendor renewal usually exceeds that within a year for any non-trivial asset count.

DE
Topic — data aggregation
Data aggregation problems for catalog metrics

Practice →

Cheat sheet — open standards recipes

"I want lineage without a catalog yet." OpenLineage emitters in every tool + Marquez as the backend. One docker-compose stack; rendered lineage graph in 30 minutes.
"I want a full open catalog." OpenMetadata (broader connector library, polished UI) or DataHub (event-native, Kafka-friendly). Pick by ecosystem fit, not by feature checklist alone.
"dbt + Airflow + Spark stack." OpenLineage emitters in all three (dbt OL adapter, Airflow OL plugin, Spark OL listener), single OPENLINEAGE_URL, one backend behind it. Promote per team via feature flag.
"Migrating off Collibra / Alation / Atlan." OpenMetadata in parallel; OL multi-cast for 60–90 days; import glossary via REST; cut over once user-facing parity is real.
"Need column-level lineage." Enable the columnLineageFacet end-to-end. dbt computes it from its manifest; Spark from query plans; SQL engines via sqlglot. Render in OpenMetadata or DataHub.
"Want governance + glossary." OpenMetadata's Glossary + GlossaryTerm + Tag + Classification entities, plus the Role / Policy model. Stewards own approval; auto-classifiers propose.
"Need to stream lineage into a vendor." Configure the vendor's OL endpoint as one of the multi-cast targets. Monte Carlo, Bigeye, Atlan, and Collibra all accept OL events.
"Production transport — HTTP or Kafka?" HTTP for setups under a few thousand events per minute and one backend. Kafka when you need durability, replay, or multiple downstream consumer groups.
"How do I cross tool boundaries?" Use the parentRunFacet. Airflow → dbt → Spark events all carry parent links; receivers reconstruct the hierarchical graph automatically.
"Custom metadata that the spec does not cover." Custom facet with your org's URI. Receivers either render it or ignore it. Lobby for promotion to standard if the use case generalises.
"OpenMetadata vs DataHub — quick decision." Want the deepest connector library and the most polished UI? OpenMetadata. Want event-native with Kafka under the hood? DataHub. Both accept OL events natively.
"Cost back-of-envelope." Closed catalog: ~$0.50/asset/mo, grows linearly. Self-hosted OpenMetadata: ~$2–4K infra + 0.25 FTE. Crossover around 8–10K assets. Add ~$20–60K/year for a vendor observability layer if needed.
"What about Marquez in production?" Fine for lineage-only at moderate scale. Lacks the catalog surface (glossary, tags, classification) — pair with OpenMetadata or DataHub if you need those.

Frequently asked questions

Is OpenLineage a catalog?

No — OpenLineage is a wire format for emitting lineage events; it is not a catalog application. It defines the JSON schema (run, job, dataset, facets) and reference clients in Python and Java, but storage and UI are the backend's job. The reference backend is Marquez (Postgres + a minimal lineage UI). For a full catalog you pair OpenLineage with OpenMetadata or DataHub. The most common interview mistake is conflating the standard with a backend — "we'll use OpenLineage as our catalog" is the wrong sentence; "we'll emit OpenLineage and store it in OpenMetadata" is the right one.

Should I use OpenMetadata or DataHub?

Both are credible open catalogs with active communities, similar feature surfaces, and OpenLineage support. Pick OpenMetadata when you want a broader out-of-the-box connector library, a polished end-user UI, native OL ingestion as a first-class path, or a managed offering (Collate) on the same code base. Pick DataHub when you want an event-native architecture with Kafka under the hood, strong upstream propagation to downstream services via the MCE / MAE model, or your existing stack already has heavy Kafka investment. Either way, your OL emitters do not change — you can switch later by pointing the transport at the new endpoint.

Does OpenLineage support column-level lineage?

Yes — the columnLineageFacet is a standard facet that maps each output column to the input columns it was derived from. dbt's OL adapter generates it from the compiled manifest; Spark's listener derives it from query plans; SQL engines via parsers like sqlglot or Calcite can compute it from the SQL text. Receivers (OpenMetadata, DataHub, Marquez) render column-level edges as a sub-graph inside the table-level lineage view. Column-level lineage is the high-value payload for impact analysis ("if I drop column C, what dashboards break?") — make sure your emitters produce the facet end-to-end.

Can I send OpenLineage events to Monte Carlo or Bigeye?

Yes — both vendors document OpenLineage endpoints. Configure OPENLINEAGE_URL to the vendor's OL endpoint (or include it in the comma-separated list for multi-cast) and the vendor receives every event your pipelines emit. Monte Carlo and Bigeye layer freshness, volume, and schema-change anomaly detection on top of the same graph your open backend sees, so you can keep one observability vendor while running an open catalog underneath. Atlan and Collibra also accept OL events for the lineage half of their products. The standard is the shared interface; the vendors compete on UX and analytics, not on data ownership.

Is Marquez production-ready?

For lineage-only workloads at small-to-medium scale (~100K events / day, ~10K datasets), yes — Marquez has been in production at multiple companies since 2019. It is the reference backend for OpenLineage, so spec changes land there first, and the Postgres + REST + minimal UI architecture is easy to operate. Marquez does not include the broader catalog surface (no glossary, no tag classification, no role / policy model). If you need that, pair Marquez with OpenMetadata or DataHub — or skip Marquez entirely and use OpenMetadata as both lineage backend and catalog. Many teams use Marquez during the OL adoption phase (weeks 1–4) and migrate to OpenMetadata as the second consumer once the catalog needs surface.

How does OpenMetadata compare to Atlan and Collibra?

OpenMetadata and the vendors converge on the same feature set (entity model, lineage graph, glossary, classification, data quality) but diverge on ownership and pricing. With Atlan or Collibra you license the product per asset and the metadata graph lives inside the vendor's database; switching vendors means rebuilding connectors and re-ingesting metadata. With OpenMetadata you self-host (or pay for the managed Collate variant), the metadata DB is yours, and the OL emitters that feed it also feed every other open or vendor receiver. Atlan and Collibra still win on out-of-the-box polish and vendor support; OpenMetadata wins on portability, cost at scale, and the option value of swapping backends without re-instrumenting pipelines. The honest answer is "both are credible; pick the trade-off your platform can actually live with for the next five years."

Practice on PipeCode

Drill the ETL practice library → for end-to-end pipeline problems where lineage and catalog instrumentation actually pay off.
Rehearse on dimensional modeling problems → to sharpen the entity instincts you need for catalog schema design.
Sharpen the event modeling library → for the runtime side of lineage emitters.
Layer the data aggregation drills → for the catalog metrics and coverage reports senior interviewers love.
Stack the aggregation library → for the COUNT-style queries that drive every "how many assets / pipelines / tests do we have?" question.
For the broader surface, read top data engineering interview questions →.
Stack the prerequisites with the only 5 skills you need to become a data engineer →.
Sharpen the design axis with the ETL system design for data engineering interviews course →.
For long-form schema craft, work through data modelling for DE interviews →.

Pipecode.ai is Leetcode for Data Engineering — every OpenLineage and OpenMetadata recipe above ships with hands-on practice rooms where you wire the emitters, design the entity model, and write the SQL behind the catalog metrics against real graded inputs. PipeCode pairs every reading with 450+ DE-focused problems and a real-time scoring engine, so you never have to wonder whether your column-lineage facet actually round-trips between Marquez, OpenMetadata, and a vendor receiver in the same way it will on interview day.

Practice ETL design now →
Dimensional modeling drills →

DEV Community: Gowtham Potureddi

ClickHouse for Real-Time Analytics: MergeTree, Materialized Views & Sharding

1. Why ClickHouse for sub-second analytics

Columnar storage and vectorised execution are the two ideas that make a billion-row aggregation feel instant

Worked example — measuring the columnar speed-up on a single aggregate

Worked example — the workloads ClickHouse does NOT love

Worked example — vectorised execution by hand

Senior interview question on the ClickHouse latency model

Solution Using the four-layer latency model

2. ClickHouse's role in the modern stack

ClickHouse sits between the stream and the dashboard — the sub-second serving tier that a batch warehouse cannot reach

Worked example — the canonical Kafka → ClickHouse → dashboard pipeline

Worked example — choosing between lambda and kappa

Worked example — multi-tenant table layout

Senior interview question on real-time stack design

Solution Using a four-component pipeline

3. The MergeTree family — the heart of ClickHouse

MergeTree is one engine, six personalities — the variant you pick is the variant your write pattern needs

Worked example — choosing MergeTree vs ReplacingMergeTree for a CDC sink

Worked example — SummingMergeTree for a pre-aggregated counter table

Worked example — CollapsingMergeTree for row-level updates

Worked example — ReplicatedMergeTree for production HA

Senior interview question on choosing the right MergeTree variant

Solution Using a ReplicatedReplacingMergeTree plus an AggregatingMergeTree roll-up

4. Materialized views — incremental aggregation engine

ClickHouse materialized views are insert-time triggers, not refresh-on-schedule — the difference is the whole story

Worked example — a 1-minute pre-aggregation MV

Worked example — cascading MVs (1-minute → 1-hour → 1-day)

Worked example — backfilling a new MV without POPULATE

Worked example — the MV-source-join pitfall

Senior interview question on materialized-view design

Solution Using an AggregatingMergeTree roll-up with HyperLogLog uniq state

5. Sharding and replication at scale

Distributed table + Replicated*MergeTree is the production pattern — shards scale write throughput, replicas scale HA and read concurrency

Worked example — a 3-shard × 2-replica reference cluster

Worked example — choosing the sharding key

Worked example — GLOBAL IN for cross-shard subqueries

Worked example — measuring replication lag

Senior interview question on cluster scaling

Solution Using a four-step horizontal scale-out plan

Cheat sheet — ClickHouse recipes

Frequently asked questions

What is ClickHouse used for?

What is the difference between MergeTree and ReplacingMergeTree?

How do materialized views work in ClickHouse?

How does ClickHouse handle updates and deletes?

ClickHouse vs Snowflake — which one for real-time analytics?

Do I need ZooKeeper to run ClickHouse?

Practice on PipeCode

Trino vs Presto vs Athena: Federated SQL Engines for the Modern Lakehouse

1. Why federated SQL became the lakehouse default

Federated SQL is the answer to "the data lives in seven places" — the lakehouse needs a query engine, not another database

Worked example — when federated SQL is the wrong choice

Worked example — the "before federated SQL" pipeline

SQL interview question on federated SQL fundamentals

Solution Using the federated-vs-MPP decision frame

2. The Presto → Trino → Athena lineage

One Facebook project became three engine names — the lineage is the most-asked Trino interview question

Worked example — picking the right engine name in an interview answer

Worked example — what the fork unlocked

SQL interview question on the Trino / Presto / Athena lineage

Solution Using a SQL dialect probe across the family

3. Architecture compared — coordinator, workers, connectors

Every federated SQL engine is a coordinator + N workers + N connectors — the differences live in execution and ops, not topology

Worked example — sizing a Trino cluster on EKS

Worked example — Athena workgroup vs self-hosted Trino — same SQL, different ops

SQL interview question on engine architecture

Solution Using a stage-by-stage trace

4. Connector ecosystem & federation patterns

Connectors are how a federated SQL engine reaches the lakehouse — and how cross-source joins actually work

Worked example — federated join: Iceberg events × Postgres customers

Worked example — predicate pushdown check with EXPLAIN

Worked example — cross-source join gotcha: shipping rows over the network

SQL interview question on connector ecosystem

Solution Using three-connector federation with selective pushdown

5. Cost, performance & when to pick which

Engine choice is a utilisation question, not a benchmark question — Athena, Trino, and Starburst each win at a different cluster utilisation band

Worked example — Athena cost reduction by partition pruning

Worked example — choosing Athena vs Trino by utilisation

Worked example — Trino fault-tolerant execution for a 90-minute ETL

Worked example — choosing `MergeTree` vs `ReplacingMergeTree` for a CDC sink

Worked example — `SummingMergeTree` for a pre-aggregated counter table

Worked example — `CollapsingMergeTree` for row-level updates

Worked example — `ReplicatedMergeTree` for production HA

Solution Using a `ReplicatedReplacingMergeTree` plus an `AggregatingMergeTree` roll-up

Solution Using an `AggregatingMergeTree` roll-up with HyperLogLog `uniq` state

Worked example — `GLOBAL IN` for cross-shard subqueries