DEV Community: Gowtham Potureddi

Prophecy: Visual, Git-Backed Low-Code Spark & SQL Pipelines for the Enterprise

Gowtham Potureddi — Sun, 27 Sep 2026 17:47:57 +0000

Prophecy low-code data engineering is a visual, drag-and-drop pipeline builder that does something most low-code tools do not: instead of running your work inside a proprietary black box, it compiles the canvas to open-source code — readable PySpark or Scala Spark for data pipelines, or dbt-style SQL models for warehouse pipelines — and commits that code to your own Git repository. You wire a few boxes together on a graph, and Prophecy generates the exact Spark job a senior engineer would have hand-written, tests and all.

That is a genuinely different shape from the two options enterprise data teams reached for before it: a legacy visual ETL suite (Informatica, Alteryx, Ab Initio, SSIS) whose logic is trapped in an opaque metadata format only that vendor can execute, or a purely hand-coded Spark stack that only your most senior engineers can safely touch. Prophecy sits in the middle — visual enough that analysts and junior engineers are productive, but code-first enough that the output is standard, version-controlled, runnable-without-the-vendor Spark and SQL. This guide walks through the five ideas an interviewer or an architecture-review board will actually probe — the gem-and-pipeline model, the visual↔code round-trip on Git, execution on Spark and Databricks via Fabrics, and reusable subgraphs with tests and column-level lineage — and pairs each with a Solution-Tail interview answer: code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the Spark SQL practice library →, rehearse the extract-and-load shapes on the ETL practice set →, and sharpen the transform logic Prophecy generates on the data-transformation practice set →.

On this page

Why Prophecy changes enterprise pipeline building in 2026
Gems, pipelines & the visual canvas
The visual ↔ code round-trip on Git
Running on Spark & Databricks with Fabrics
Reusable subgraphs, tests & column-level lineage
Cheat sheet — Prophecy recipes
Frequently asked questions
Practice on PipeCode

1. Why Prophecy changes enterprise pipeline building in 2026

Prophecy is a code generator wearing a visual UI — that one fact decides where it fits

The one-sentence invariant: Prophecy is not a runtime you send data through; it is a visual editor that emits ordinary Spark and SQL code, which then runs on your own engine. Everything that makes Prophecy defensible to a platform team follows from that. There is no proprietary execution engine to license per row, no metadata format only the vendor can read, and no "export to a dead PDF of a diagram" migration cliff — the artifact Prophecy produces is a Git repo full of PySpark files or SQL models you could keep running if Prophecy vanished tomorrow.

The low-code-without-lock-in split — what Prophecy does and deliberately does not do.

Authoring. Prophecy gives you a drag-and-drop canvas of gems (transform nodes) that snap into a pipeline graph, with per-gem data previews and a schema that flows through the graph as you build.
Code generation. Every visual pipeline compiles to standard code: PySpark or Scala for Spark projects, dbt-style SQL models for SQL projects. That code is the source of truth and lives in Git.
Execution is out of scope on purpose. Prophecy does not run your data on its own cluster; the generated Spark runs on Databricks (or any Spark), and the generated SQL runs on your warehouse via dbt. Prophecy stays a design-and-generate layer, which is exactly why the output has no lock-in.

Where Prophecy sits against the alternatives.

vs hand-written Spark. A raw PySpark codebase is maximally flexible but gates every change behind a senior engineer and a code review. Prophecy lets an analyst build the same pipeline visually while still producing the same reviewable PySpark — you widen the contributor pool without lowering the artifact quality.
vs legacy Informatica / Alteryx / Ab Initio. Those visual suites also let non-experts build pipelines, but the logic is trapped in a vendor metadata format and runs only on the vendor engine. Prophecy's visual output is open-source Spark/SQL in your repo, so there is no proprietary runtime to keep paying for and no black box to reverse-engineer during an audit.
vs Fivetran / Airbyte. Managed connectors solve extract-load for sources that already have a connector. Prophecy solves transform — the joins, aggregations, and business logic downstream of ingestion — and generates the Spark/SQL that expresses it.

What interviewers listen for.

Do you say "Prophecy compiles the canvas to code, it is not a runtime" in the first sentence? — senior signal.
Do you frame the value as "low-code authoring, code-first artifact" rather than "no-code magic"? — required framing.
Do you reach for Prophecy when the goal is "migrate Informatica to Spark without hand-rewriting 4,000 mappings" or "let analysts contribute reviewable Spark", not as a Fivetran replacement? — senior signal.
Do you mention Git, tests, and column-level lineage as built-ins, not afterthoughts? — the enterprise point.

Worked example — three boxes that become a real Spark job

Detailed explanation. The canonical Prophecy "hello world" is a three-gem pipeline — a Source that reads a table, a Reformat that derives a column, and a Target that writes the result. It looks like a toy flowchart, and that is the point: the same three boxes that transform a five-row table generate the same PySpark structure that a production job would, because Prophecy always emits one function per gem and a main that wires them together. No notebook, no boilerplate SparkSession glue that you maintain by hand.

Question. Read an orders table, add a total = qty * price column, and write to orders_enriched. Show the shape of the Spark code Prophecy generates from the three gems.

Input.

order_id	qty	price
1	2	10.0
2	3	4.0

Code.

from pyspark.sql import DataFrame, SparkSession
from pyspark.sql import functions as F

def Source(spark: SparkSession) -> DataFrame:
    return spark.read.table("sales.orders")

def Reformat(spark: SparkSession, in0: DataFrame) -> DataFrame:
    return in0.withColumn("total", F.col("qty") * F.col("price"))

def Target(spark: SparkSession, in0: DataFrame) -> None:
    in0.write.mode("overwrite").saveAsTable("sales.orders_enriched")

def pipeline(spark: SparkSession) -> None:
    df_src = Source(spark)
    df_ref = Reformat(spark, df_src)
    Target(spark, df_ref)

Step-by-step explanation. The Source gem becomes a function that returns a DataFrame from sales.orders; it declares where data comes from and nothing else. The Reformat gem becomes a pure DataFrame → DataFrame function that adds the total column with withColumn. The Target gem becomes a sink function that writes the table. The pipeline driver wires them in dependency order — exactly the DAG you drew on the canvas — so the visual graph and the code are the same object viewed two ways.

Output.

order_id	qty	price	total
1	2	10.0	20.0
2	3	4.0	12.0

Rule of thumb. If you can draw the pipeline as boxes and arrows, Prophecy can generate it as functions and a driver — the toy example and the 200-gem production pipeline differ only in how many gems there are, never in the code shape.

2. Gems, pipelines & the visual canvas

A gem is one transform node, a pipeline is a DAG of gems, and each gem compiles to a function

Prophecy has one authoring primitive you compose endlessly — the gem — and an interviewer who asks "how is a Prophecy pipeline structured?" wants the gem-to-pipeline-to-code chain in order. Get this vocabulary crisp and the whole tool snaps into focus.

The core objects.

Gem — one transform node. A gem is a single box on the canvas that performs one operation. Built-in gems cover the standard Spark verbs: Source and Target (read/write), Reformat (select/derive columns), Aggregate (group-by), Join, Filter, OrderBy, Deduplicate, SetOperation (union/intersect), and FlattenSchema for nested data. Each gem carries its own configuration and its own output schema.
Pipeline — a DAG of gems. Wiring gem outputs to gem inputs builds a directed acyclic graph. Data flows as typed Spark DataFrames along the edges; the schema at each edge is known at design time, so Prophecy can validate column references before you ever run.
Project — a Git-backed collection of pipelines. A project bundles pipelines, subgraphs, tests, and generated code into one repository. It is the unit that maps to a Git repo.

What every gem gives you.

A generated function. Each gem compiles to exactly one function in the project code — Reformat(spark, in0), Aggregate(spark, in0), and so on — so the code is as modular as the diagram.
A visual expression builder. Inside a gem you write column expressions in SQL-like syntax (qty * price, upper(name)), and Prophecy translates them into the target dialect (Spark functions calls for PySpark, raw SQL for SQL projects).
An interactive preview. You can run the pipeline up to any gem and see a sample of the DataFrame at that point, which is the low-code equivalent of a breakpoint.

Why the gem is the powerful unit.

Because each gem is an isolated function, a pipeline is trivially testable gem-by-gem and readable diff-by-diff in a pull request.
Because gems carry schema, Prophecy catches "column does not exist" at design time, not at 3 a.m. in production.
Because the gem set is extensible, a platform team can ship custom gems (via the Gem Builder) that encode company standards, and every analyst gets them as drag-and-drop boxes.

Worked example — a Filter gem and a Reformat gem in one pipeline

Detailed explanation. Real pipelines chain several gems. Here a two-gem transform keeps only paid orders (a Filter gem) and then standardizes the customer name (a Reformat gem). Each gem becomes its own function, and the pipeline driver composes them — the canvas order is the call order.

Question. Build a pipeline that filters orders to status = 'paid' and uppercases customer, and show the two generated gem functions plus the driver.

Input. Four orders, two of them paid, returned by the Source gem.

Code.

from pyspark.sql import DataFrame, SparkSession
from pyspark.sql import functions as F

def FilterPaid(spark: SparkSession, in0: DataFrame) -> DataFrame:
    return in0.filter(F.col("status") == "paid")

def StandardizeCustomer(spark: SparkSession, in0: DataFrame) -> DataFrame:
    return in0.withColumn("customer", F.upper(F.col("customer")))

def pipeline(spark: SparkSession) -> None:
    src   = spark.read.table("sales.orders")
    paid  = FilterPaid(spark, src)
    clean = StandardizeCustomer(spark, paid)
    clean.write.mode("overwrite").saveAsTable("sales.orders_clean")

Step-by-step explanation. The Filter gem becomes FilterPaid, a DataFrame → DataFrame function applying one predicate. The Reformat gem becomes StandardizeCustomer, applying one withColumn. The driver reads the source, threads it through the two gem functions in canvas order, and writes the sink. Because each gem is a separate function, a reviewer reading the pull request sees exactly two logical changes, cleanly named.

Output.

order_id	customer	status
1	ADA	paid
3	LINUS	paid

Rule of thumb. One gem = one function = one reviewable unit. Keep each gem doing a single verb and your generated code stays as clean as your diagram.

Prophecy interview question on the gem/pipeline model

Question. An interviewer describes a Reformat gem that should derive three columns — total = qty * price, year = year(order_ts), and a normalized email = lower(trim(email)) — from an orders DataFrame. Write the single Spark transform this gem generates, keeping it one projection so the physical plan stays flat.

Solution Using a single projection with column expressions

Code.

from pyspark.sql import DataFrame, SparkSession
from pyspark.sql import functions as F

def Reformat(spark: SparkSession, in0: DataFrame) -> DataFrame:
    return in0.select(
        "order_id",
        "qty",
        "price",
        (F.col("qty") * F.col("price")).alias("total"),
        F.year(F.col("order_ts")).alias("year"),
        F.lower(F.trim(F.col("email"))).alias("email"),
    )

Step-by-step trace.

step	expression applied	column produced
1	`qty * price`	`total`
2	`year(order_ts)`	`year`
3	`lower(trim(email))`	`email` (overwritten)
4	one `select` projection	flat plan

A Reformat gem is a projection: it maps input columns to output columns in a single select, so all derivations land in one plan node.
total multiplies two columns; year extracts the year from a timestamp; email is normalized by nesting lower around trim.
Because everything is one select, Spark's Catalyst optimizer fuses the expressions into a single project stage — no extra shuffle or scan.
Naming each derived column with .alias(...) is what makes the gem's output schema explicit and lets the next gem reference total and year by name.

Output:

order_id	qty	price	total	year	email
1	2	10.0	20.0	2026	ada@x.io

Why this works — concept by concept:

Projection gem — a Reformat compiles to one select, so N derived columns cost one plan node, not N chained withColumn calls that stack projections.
Column expressions — writing qty * price in the gem and letting Prophecy emit F.col("qty") * F.col("price") keeps the visual and the code semantically identical.
Explicit aliases — every derived column is named, so the downstream schema is deterministic and design-time validation can catch typos.
Optimizer-friendly — a single flat projection lets Catalyst fuse expressions, which is why "one gem, one select" is both cleaner and faster than a pile of withColumns.
Cost — the transform is O(rows) with no shuffle; a projection is a narrow transformation, so it adds no stage boundary.

Spark SQL
Topic — spark-sql
Spark SQL transform and projection problems

Practice →

ETL Topic — etl ETL pipeline-design problems

Practice →

3. The visual ↔ code round-trip on Git

Every visual edit regenerates readable code, and the whole project lives in your Git repo

The feature that sells Prophecy to a platform team is the round-trip: the canvas and the code are two views of the same artifact, kept in sync bidirectionally, and both are stored in ordinary version control. Edit a gem and the code regenerates; edit the code and the canvas re-parses. This is the difference between a low-code tool you cannot audit and one that behaves like a normal software repository.

How the round-trip works.

Visual → code. Every change on the canvas — adding a gem, editing an expression, rewiring an edge — immediately regenerates the corresponding function in the project's source files. There is no separate "export" step; the code is the save format.
Code → visual. Because Prophecy parses the generated code back into the graph, an engineer can open pipeline.py in their IDE, edit it, push, and see the canvas reflect the change. The code and diagram cannot drift apart.
Standard, readable output. The generated PySpark/SQL is not obfuscated machine output; it reads like code a person wrote, with named functions per gem, so it survives a human code review.

Git is the backbone, not a bolt-on.

Projects are repositories. A Prophecy project maps to a Git repo (GitHub, GitLab, Bitbucket, Azure DevOps, or Prophecy-managed Git). Pipelines, subgraphs, tests, and configs are all files.
Branch, commit, PR, merge. You develop on a branch, commit visual changes as normal diffs, open a pull request, get a review, and merge — the full software lifecycle, driven from a visual tool.
CI/CD like any codebase. Because the artifact is code, you plug the repo into your existing CI to run tests and into your existing CD to deploy, with no Prophecy-specific pipeline server in the critical path.

Why "no lock-in" is a technical claim, not a slogan.

The generated Spark job runs on any Spark cluster with spark-submit; the generated SQL runs through dbt. Neither needs Prophecy at execution time.
If you stopped using Prophecy, you keep a working, readable, tested Spark/SQL codebase — the exit cost is "you lose the visual editor," not "your pipelines stop running."

Worked example — an Aggregate gem and its Git diff

Detailed explanation. The clearest way to see the round-trip is to add one Aggregate gem and read the diff it produces. The gem groups orders by customer and sums the total; the generated function is a plain groupBy(...).agg(...), and the Git diff a reviewer sees is a single new named function — reviewable in seconds.

Question. Add an Aggregate gem that computes revenue = sum(total) and orders = count(*) per customer, and show the generated function that lands in the commit.

Input.

customer	total
ada	20.0
ada	5.0
linus	12.0

Code.

from pyspark.sql import DataFrame, SparkSession
from pyspark.sql import functions as F

def AggregateByCustomer(spark: SparkSession, in0: DataFrame) -> DataFrame:
    return (
        in0.groupBy("customer")
           .agg(
               F.sum("total").alias("revenue"),
               F.count(F.lit(1)).alias("orders"),
           )
    )

Step-by-step explanation. The Aggregate gem's group-by key (customer) becomes groupBy("customer"), and each aggregate expression becomes one .agg(...) entry with an alias. Prophecy writes this as a single new function; the commit diff is + def AggregateByCustomer(...) plus one wiring line in the driver. A reviewer reads the diff exactly as they would review a hand-written Spark change — because it is one.

Output.

customer	revenue	orders
ada	25.0	2
linus	12.0	1

Rule of thumb. If a visual change does not produce a clean, reviewable code diff, you are drawing something the code cannot express cleanly — split it into more gems until each diff is one idea.

Prophecy interview question on aggregation transforms

Question. An interviewer asks you to compute, per customer, the total revenue and the number of distinct order days, then keep only customers with revenue over 100. Write the Spark aggregation this gem would generate.

Solution Using groupBy with a filtered aggregate

Code.

from pyspark.sql import DataFrame, SparkSession
from pyspark.sql import functions as F

def CustomerRollup(spark: SparkSession, in0: DataFrame) -> DataFrame:
    return (
        in0.groupBy("customer")
           .agg(
               F.sum("total").alias("revenue"),
               F.countDistinct(F.to_date("order_ts")).alias("active_days"),
           )
           .filter(F.col("revenue") > 100)
    )

Step-by-step trace.

customer	rows in	sum(total)	distinct days	kept?
ada	3	140.0	2	yes
linus	2	60.0	2	no
grace	4	220.0	3	yes

groupBy("customer") collapses each customer's rows into one group, triggering one shuffle keyed on customer.
sum("total") accumulates revenue and countDistinct(to_date(order_ts)) counts unique calendar days, both computed in the same aggregation pass.
The post-aggregation filter(revenue > 100) is a HAVING clause: it runs on the grouped rows, dropping linus while keeping ada and grace.
Ordering matters — filtering after the aggregate means the predicate sees revenue, a column that only exists post-group; a Filter gem placed before the Aggregate could not reference it.

Output:

customer	revenue	active_days
ada	140.0	2
grace	220.0	3

Why this works — concept by concept:

Group-by shuffle — the aggregation repartitions rows by customer so all of one customer's rows meet on one executor; this is the single shuffle the whole transform costs.
Multiple aggregates, one pass — sum and countDistinct are computed together, so two metrics cost one scan and one shuffle, not two.
HAVING via post-filter — filtering after agg is the Spark equivalent of SQL HAVING; the predicate references the aggregated revenue, which is only defined after the group.
countDistinct semantics — counting distinct to_date(order_ts) answers "how many active days," a different question from count(*); picking the right counter is the correctness crux.
Cost — O(rows) to scan plus one O(rows) shuffle keyed on customer; the final filter is O(groups), typically tiny.

ETL
Topic — etl
ETL aggregation and rollup problems

Practice →

Transform Topic — data-transformation Group-by and HAVING transform problems

Practice →

4. Running on Spark & Databricks with Fabrics

A Fabric binds a pipeline to where it runs — Spark on a cluster, or SQL on a warehouse

Authoring produces code; something has to run it. In Prophecy that something is a Fabric — the named execution environment that holds the connection, credentials, and cluster or warehouse settings your pipeline runs against. The same canvas can point at different Fabrics (dev, staging, prod), and the project language decides what the generated code targets: Spark projects emit PySpark/Scala that runs on a cluster, SQL projects emit dbt models that run on a warehouse.

What a Fabric is.

The execution environment. A Fabric bundles where and how a pipeline runs: a Databricks workspace and cluster, a Spark-on-EMR/Dataproc/Livy endpoint, or a SQL warehouse (Databricks SQL, Snowflake, BigQuery) for SQL projects.
Connection + credentials + compute. It carries the workspace URL, auth token/secret, and cluster size, so switching from a small dev cluster to a large prod cluster is a Fabric swap, not a code change.
Interactive and scheduled. During development the Fabric backs the live per-gem preview; for production you deploy the pipeline as a scheduled Databricks Job or an Airflow DAG that calls the same generated code.

Spark projects vs SQL projects.

Spark project → PySpark/Scala. Gems compile to DataFrame transformations; the job is submitted to a Spark cluster. Choose this for large-scale distributed transforms, complex joins, and ML feature pipelines.
SQL project → dbt. Gems compile to SQL models with ref()/source() dependencies that run in-warehouse via dbt. Choose this when the data already lives in the warehouse and you want warehouse-native, dbt-tested models.
Same authoring, different target. The gem canvas looks the same either way; the project language decides the dialect and the engine, which is how one tool serves both Spark and SQL teams.

What interviewers probe about execution.

Development ≠ deployment. Interactive runs use an attached cluster for fast feedback; production runs are scheduled jobs on right-sized compute. Conflating the two is a classic junior mistake.
Cluster sizing is a Fabric concern. Spilling, out-of-memory, and shuffle-partition tuning are properties of the Fabric's compute, not the visual pipeline — the generated Spark obeys the same spark.sql.shuffle.partitions you would tune by hand.

Worked example — the same join as Spark and as SQL

Detailed explanation. Because the project language chooses the dialect, the same visual Join gem generates PySpark in a Spark project and SQL in a SQL project. Seeing both makes the "same canvas, different engine" point concrete: an orders-to-customers inner join is one gem, two outputs.

Question. Join orders to customers on customer_id to attach region. Show what the Join gem generates in a Spark project versus a SQL project.

Input.

order_id	customer_id
1	7
2	8

Code.

## Spark project (PySpark) — generated Join gem
def JoinCustomers(spark, orders, customers):
    return orders.join(customers, on="customer_id", how="inner") \
                 .select("order_id", "customer_id", "region")

-- SQL project (dbt model) — generated Join gem
select o.order_id, o.customer_id, c.region
from {{ ref('orders') }} o
join {{ ref('customers') }} c
  on o.customer_id = c.customer_id

Step-by-step explanation. In the Spark project the Join gem emits orders.join(customers, on="customer_id", how="inner") and a projection selecting region. In the SQL project the identical gem emits an ANSI JOIN with dbt ref() calls that wire model dependencies. The canvas, the join key, and the join type are the same; only the target dialect and engine differ, decided by the project language and the Fabric.

Output.

order_id	customer_id	region
1	7	EU
2	8	US

Rule of thumb. Pick the project language by where the data lives and how big it is — Spark for distributed scale, SQL/dbt for warehouse-native models — then let the same gem canvas target either.

Prophecy interview question on deduplicating with Spark SQL

Question. Orders arrive with duplicate rows per order_id (retries), each with an updated_at. Keep exactly one row per order_id — the latest updated_at — using Spark SQL, the way a Deduplicate gem would generate it. How do you make it deterministic?

Solution Using ROW_NUMBER over a partition

Code.

with ranked as (
  select
    *,
    row_number() over (
      partition by order_id
      order by updated_at desc, ingest_seq desc
    ) as rn
  from orders
)
select * except (rn)
from ranked
where rn = 1

Step-by-step trace.

order_id	updated_at	ingest_seq	rn	kept
7	12:30	2	1	yes
7	12:30	1	2	no
8	11:00	1	1	yes
8	10:00	1	2	no

partition by order_id groups the duplicate rows for each order together so ranking is per-key.
order by updated_at desc puts the freshest row first; the tie-break ingest_seq desc makes the winner deterministic when two rows share the same updated_at.
row_number() assigns 1 to the winner of each partition, 2..N to the rest.
where rn = 1 keeps exactly one row per order_id, and except (rn) drops the helper column so the output schema matches the input.

Output:

order_id	updated_at	ingest_seq
7	12:30	2
8	11:00	1

Why this works — concept by concept:

Window partition — partition by order_id turns "dedupe per key" into a ranking problem solved inside each partition, no self-join required.
Deterministic tie-break — adding ingest_seq desc to the order by removes the nondeterminism of ties, so re-running yields the same survivor every time.
row_number vs rank — row_number guarantees a single winner (no ties at rank 1), which is exactly the "keep one" requirement; rank could keep several.
except (rn) — projecting away the helper column keeps the deduplicated output schema-identical to the source, so downstream gems are unaffected.
Cost — one shuffle to partition by order_id plus an in-partition sort, O(rows log rows) per partition; a Deduplicate gem generates exactly this plan.

Spark SQL
Topic — spark-sql
Window-function and dedup problems in Spark SQL

Practice →

Optimization Topic — optimization Shuffle and partition tuning problems

Practice →

5. Reusable subgraphs, tests & column-level lineage

Package gems into reusable subgraphs, unit-test them, and trace every column

The three features that make Prophecy an enterprise tool rather than a personal one are reuse, testing, and lineage. A reusable subgraph turns a proven set of gems into a shareable component; gem-level unit tests pin behaviour in CI; and column-level lineage answers "if I change this field, what breaks?" across every pipeline in the org. These are the capabilities a legacy visual ETL suite either lacks or hides behind an opaque metadata store.

Reusable subgraphs.

A subgraph is a group of gems as one node. You select several gems — say, the five that standardize a customer record — and package them into a reusable subgraph with defined input and output ports. It appears on the canvas as a single box.
Build once, reuse everywhere. Every pipeline that needs "standardize customer" drops in the same subgraph; a fix to the subgraph propagates to all consumers, so business logic is DRY instead of copy-pasted across 40 pipelines.
Parameterizable. Subgraphs (and custom gems built with the Gem Builder) take configuration, so one component adapts to slightly different inputs without forking.

Tests that run in CI.

Gem-level unit tests. You attach sample input rows and expected output rows to a gem; Prophecy generates a real unit test (pytest for PySpark, dbt tests for SQL) that runs in your CI on every pull request.
Fail the build, not production. Because tests are code in the repo, a transform regression fails the PR check — the same guardrail a hand-written Spark codebase has, now automatic from the visual definition.
Data-quality expectations. Beyond unit tests, expectation checks (not-null, uniqueness, accepted ranges) can gate a load so bad data fails loudly instead of corrupting a table.

Column-level lineage.

Field-to-field tracing. Prophecy captures how each output column was derived from input columns across gems and pipelines, so you can trace revenue back through every transform to its source columns.
Impact analysis. Before renaming or dropping a source column, lineage shows every downstream pipeline and report that depends on it — the difference between a safe change and a Monday-morning outage.

Worked example — a reusable "standardize customer" subgraph

Detailed explanation. The everyday reuse pattern is a subgraph that encapsulates a few cleaning gems behind one input and one output port. Its generated code is a function that takes a DataFrame and returns a DataFrame, so any pipeline can call it — the visual box and the reusable function are, again, the same thing.

Question. Package "trim + lowercase email, uppercase name, drop test rows" into a reusable subgraph and show the generated function that pipelines call.

Input. A raw customers DataFrame with mixed-case and test rows.

Code.

from pyspark.sql import DataFrame, SparkSession
from pyspark.sql import functions as F

def StandardizeCustomer(spark: SparkSession, in0: DataFrame) -> DataFrame:
    return (
        in0.withColumn("email", F.lower(F.trim(F.col("email"))))
           .withColumn("name", F.upper(F.col("name")))
           .filter(~F.col("email").endswith("@test.local"))
    )

Step-by-step explanation. The subgraph's three gems become three chained transformations inside one function: normalize email, uppercase name, and drop internal test accounts. Because it is exposed as a single DataFrame → DataFrame function with one input port, every pipeline reuses it by wiring one box — and a fix here updates all of them at once.

Output.

name	email
ADA	ada@x.io
GRACE	grace@y.io

Rule of thumb. If the same three-or-more gems appear in two pipelines, promote them to a reusable subgraph — you trade a one-time packaging step for org-wide consistency and a single place to fix bugs.

Prophecy interview question on testing a transform

Question. You must guarantee that the "standardize customer" transform always lowercases email and never lets a @test.local row through. Write the unit test a Prophecy gem test generates so this is enforced in CI, and explain what a failure would catch.

Solution Using a gem unit test with expected output

Code.

from pyspark.sql import SparkSession
from mypipeline.gems import StandardizeCustomer

def test_standardize_customer(spark: SparkSession):
    src = spark.createDataFrame(
        [("Ada", " ADA@X.IO "), ("Bot", "bot@test.local")],
        ["name", "email"],
    )
    out = StandardizeCustomer(spark, src)
    rows = {r["name"]: r["email"] for r in out.collect()}

    assert rows == {"ADA": "ada@x.io"}          # Bot dropped, Ada normalized
    assert out.count() == 1                       # test row filtered out

Step-by-step trace.

input (name, email)	after normalize	after filter	in expected?
Ada, " ADA@X.IO "	ADA, ada@x.io	kept	yes
Bot, bot@test.local	BOT, bot@test.local	dropped	n/a

The test builds a tiny DataFrame with one good row and one @test.local row, then calls the exact gem function under test.
StandardizeCustomer trims and lowercases the email, uppercases the name, and filters the @test.local row out.
The assertion pins the whole expected output — one row, {"ADA": "ada@x.io"} — so any regression (a missing lower, a broken filter) fails the assertion.
out.count() == 1 independently guards the filter, so a change that lets test rows through fails even if the surviving row still looks right.

Output:

check	result
normalized email	`ada@x.io` ✓
test row filtered	count == 1 ✓

Why this works — concept by concept:

Gem-level unit test — testing one gem function in isolation makes failures point at exactly one transform, not a whole pipeline run.
Expected-output assertion — pinning the full output (not just "no error") is what turns a smoke test into a real regression guard.
CI enforcement — because the test is code in the repo, it runs on every pull request, so a visual edit that breaks the contract fails the build, not production.
Independent invariants — asserting both the value and the row count catches two distinct failure modes (bad normalization vs a broken filter) separately.
Cost — a two-row test runs in milliseconds locally, so it is cheap enough to attach to every gem that carries business rules.

Transform
Topic — data-transformation
Reusable transform and cleaning problems

Practice →

Quality Topic — data-quality Unit-test and data-validation problems

Practice →

Cheat sheet — Prophecy recipes

Minimal Spark pipeline (Source → Reformat → Target).

def pipeline(spark):
    src = spark.read.table("sales.orders")
    out = src.withColumn("total", F.col("qty") * F.col("price"))
    out.write.mode("overwrite").saveAsTable("sales.orders_enriched")

Generated gem function shape (one function per gem).

def Reformat(spark, in0):
    return in0.select("id", (F.col("qty") * F.col("price")).alias("total"))

Aggregate gem → groupBy/agg.

def Aggregate(spark, in0):
    return in0.groupBy("customer").agg(F.sum("total").alias("revenue"))

SQL project (dbt model) gem.

select o.order_id, c.region
from {{ ref('orders') }} o
join {{ ref('customers') }} c on o.customer_id = c.customer_id

Reusable subgraph signature.

def StandardizeCustomer(spark, in0):   # one input port, one output port
    return in0.withColumn("email", F.lower(F.trim(F.col("email"))))

Gem unit test skeleton.

def test_gem(spark):
    src = spark.createDataFrame([(1, 2, 10.0)], ["id", "qty", "price"])
    out = Reformat(spark, src)
    assert out.collect()[0]["total"] == 20.0

Choosing project language.

Situation	Project type
Large distributed transforms, complex joins, ML features	Spark (PySpark/Scala)
Data already in the warehouse, want dbt-tested models	SQL (dbt)
Migrating Informatica/Alteryx mappings to open code	Spark or SQL by target engine
Analysts contributing reviewable pipelines	either — both emit Git-tracked code

Frequently asked questions

What is Prophecy?

Prophecy is a low-code data engineering platform: a visual, drag-and-drop canvas for building data pipelines that compiles to open-source code. You wire together gems (transform nodes) on a graph, and Prophecy generates readable PySpark or Scala Spark for Spark pipelines, or dbt-style SQL models for warehouse pipelines, and commits that code to your own Git repository. It is a design-and-generate layer, not a runtime you send data through.

Does Prophecy lock me into its runtime?

No — that is the central design choice. The artifact Prophecy produces is standard Spark or SQL code in your Git repo; the Spark runs on any Spark cluster (Databricks, EMR, Dataproc) via spark-submit, and the SQL runs through dbt on your warehouse. Neither needs Prophecy at execution time, so if you stopped using the visual editor you would still have a working, readable, tested codebase.

What are gems in Prophecy?

Gems are the building blocks of a pipeline — each gem is one transform node on the canvas. Built-in gems cover the standard operations (Source, Target, Reformat, Aggregate, Join, Filter, OrderBy, Deduplicate, SetOperation, FlattenSchema), and each gem compiles to exactly one function in the generated code. Platform teams can also build custom gems with the Gem Builder to encode company-specific logic as reusable drag-and-drop boxes.

How does the visual-to-code round-trip work?

Every edit on the visual canvas immediately regenerates the corresponding code, and because Prophecy also parses code back into the graph, an engineer can edit the generated file in an IDE and see the canvas update. The two views cannot drift apart. Since the code is the save format and lives in Git, you get branches, commits, pull requests, reviews, and CI/CD exactly like any software repository.

Does Prophecy support SQL as well as Spark?

Yes. A Spark project compiles gems to PySpark/Scala DataFrame transformations that run on a Spark cluster, while a SQL project compiles the same style of gem canvas to dbt SQL models that run in-warehouse (Databricks SQL, Snowflake, BigQuery). The authoring experience is the same; the project language and the Fabric (execution environment) decide the dialect and engine.

Can Prophecy migrate legacy Informatica or Alteryx pipelines?

Yes — a common enterprise use case is converting legacy visual ETL (Informatica, Alteryx, Ab Initio, SSIS, DataStage) into open Spark or SQL. Prophecy's migration tooling maps the legacy mappings/workflows onto gems and generates the equivalent PySpark or dbt code in Git, so you exit the proprietary runtime while keeping a visual editor and a modern, testable, version-controlled codebase.

Practice on PipeCode

Pipecode.ai is Leetcode for Data Engineering — every Prophecy idea above, from the projection Reformat gem to the group-by rollup, the ROW_NUMBER dedup, and the unit-tested reusable subgraph, maps to a hands-on practice room where you write the Spark or SQL against real graded inputs. PipeCode pairs each reading with 450+ DE-focused problems and a real-time scoring engine, so your answer to "what code would that gem generate?" holds up under a senior interviewer's depth probes.

Practice Spark SQL problems now →
ETL pipeline drills →

Bruin: SQL + Python Pipelines in One Framework With Built-In Quality Checks

Gowtham Potureddi — Sun, 27 Sep 2026 17:43:16 +0000

bruin is the open-source data pipeline framework that collapses four tools into one: it runs your SQL transforms, executes your Python jobs, ingests data from external sources, and enforces data-quality checks — all from a single Go binary and a folder of files you keep in version control. You do not wire dbt to Airbyte to Great Expectations to an orchestrator and hope the seams hold. You write assets — SQL files, Python files, or small YAML files — each carrying its own metadata in a comment block, and you type bruin run.

That is a different shape from the modern stack most teams assembled over the last five years, where transformation, ingestion, testing, and scheduling are four separate products glued together with CI scripts. This guide walks through the four ideas an interviewer will actually probe — the asset model with YAML-in-comment metadata, materialization strategies, built-in and custom quality checks, and ingestion plus the dependency-driven DAG — and pairs each with a Solution-Tail interview answer: code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the pipeline-design practice library →, rehearse the load-shape decisions on the ETL practice set →, and harden your test suite on the data-quality practice set →.

On this page

Why Bruin puts the whole pipeline in one framework
Assets: SQL, Python & YAML-in-comment metadata
Materialization: how a SELECT becomes a table
Built-in & custom quality checks
Ingestion with ingestr & multi-destination DAGs
Cheat sheet — Bruin recipes
Frequently asked questions
Practice on PipeCode

1. Why Bruin puts the whole pipeline in one framework

Bruin is one tool for SQL, Python, ingestion, and quality — that single fact decides where it fits

The one-sentence invariant: Bruin treats every step of a pipeline — extract, transform, and test — as an asset in one project, so you stop stitching four products together and run the whole graph with bruin run. Everything that makes Bruin attractive to a data engineering team follows from that. There is no separate ingestion service, no separate test framework, no separate scheduler required to make a pipeline correct; a Bruin project is a folder of asset files plus a pipeline.yml that the CLI reads, orders, executes, and validates.

What lives inside one Bruin project.

Ingestion. An ingestr asset pulls data from an external source (an API, a database, a bucket) into a destination, with no connector code to write.
Transformation. A SQL asset (bq.sql, sf.sql, duckdb.sql, pg.sql, and more) or a Python asset runs your business logic and materializes a table or view.
Quality. Column-level and custom checks are declared in the same file as the asset they guard, and run right after it produces data.
Orchestration. Bruin reads each asset's depends list, builds a DAG, and runs assets in topological order — no external orchestrator needed for a single pipeline.

Where Bruin sits against the alternatives.

vs dbt. dbt is transform-only: it compiles and runs SQL (and, more recently, Python models) but leaves ingestion and orchestration to other tools. Bruin covers the same SQL-modelling ground plus native Python assets, ingestr-based ingestion, and a built-in runner. If your answer to "how does raw data arrive?" is "a different tool," Bruin folds that step in.
vs dbt + Airbyte + Great Expectations + Airflow. The classic modern stack is four products and the glue between them. Bruin puts ingestion, transformation, quality, and single-pipeline scheduling behind one CLI and one config language, so there are fewer moving parts to break.
vs a hand-rolled Python stack. A pile of scripts has no schema for metadata, no dependency graph, no first-class checks, and no consistent run semantics. Bruin gives you all four without a platform to operate.

What interviewers listen for.

Do you say "Bruin is SQL + Python + ingestion + quality in one framework" rather than "another dbt"? — senior signal.
Do you place dbt as "transform-only" and Bruin as "end-to-end assets" unprompted? — required framing.
Do you mention that quality checks live next to the asset and can block downstream — not in a separate suite? — the whole point.
Do you note that bruin run needs no external orchestrator for a single pipeline, while still deploying to Airflow when you want it? — senior signal.

Worked example — one asset that materializes a table

Detailed explanation. The canonical Bruin "hello world" is a single SQL file whose top comment block is YAML metadata and whose body is a SELECT. Bruin reads the metadata to learn the asset's name, platform, and how to materialize it, then wraps your query in the right DDL. You never write CREATE TABLE — you declare materialization: type: table and Bruin does it.

Question. Write one Bruin SQL asset that turns a two-row SELECT into a DuckDB table dashboard.hello, with no CREATE TABLE written by you.

Input.

what you write	what Bruin uses it for
`@bruin` comment block	asset metadata (name, type, materialization)
`SELECT` body	the query whose result becomes the table

Code.

/* @bruin
name: dashboard.hello
type: duckdb.sql
materialization:
  type: table
@bruin */

select 1 as id, 'ada' as name
union all
select 2 as id, 'linus' as name

Step-by-step explanation. The block between /* @bruin and @bruin */ is parsed as YAML: name: dashboard.hello becomes the target schema.table, type: duckdb.sql tells Bruin to run the body as DuckDB SQL, and materialization: type: table tells it to persist the result. When you run bruin run assets/hello.sql, Bruin generates a create+replace statement (the default table strategy), executes your SELECT, and stores the rows in dashboard.hello. The SQL body stays a plain SELECT — all the persistence logic lives in the metadata.

Output.

Bruin created	value
table	`dashboard.hello`
columns	`id`, `name` (inferred from the `SELECT`)
rows	2
strategy applied	`create+replace` (table default)

Rule of thumb. If a step can be expressed as "metadata plus a body," it is a Bruin asset — a SQL query, a Python script, or an ingestion spec differ only in the body, never in the project shape.

2. Assets: SQL, Python & YAML-in-comment metadata

The asset — code plus its YAML metadata in one file — is Bruin's entire mental model

Bruin has exactly one core abstraction you compose, and an interviewer who asks "walk me through Bruin's model" wants it crisp: everything is an asset, and an asset is code with a block of YAML metadata attached to it. Learn the three file shapes and where the metadata goes, and the whole framework snaps into focus.

The three asset file shapes.

SQL asset (.sql). The metadata lives between /* @bruin and @bruin */ markers at the top of the file; the SQL query body follows underneath. Definition and query stay in one file — Bruin will not let you split them across a sibling .asset.yml.
Python asset (.py). The metadata lives between """@bruin and @bruin""" docstring markers; the Python script follows. Same file, same idea.
YAML asset (<name>.asset.yml). A standalone YAML file with only metadata and no code body — used for asset types that have no inline code, such as ingestr, sensor, and seed. The .asset.yml suffix is required; a plain .yml is treated as config and ignored.

The metadata keys that matter.

name. The asset's identity, following the schema.table convention (dots separate segments). It is optional — if omitted, Bruin infers it from the file path relative to assets/, so assets/analytics/orders.sql becomes analytics.orders.
type. How the asset executes: bq.sql, sf.sql, pg.sql, duckdb.sql, python, ingestr, r, and more. The type binds the asset to a platform.
depends. The list of upstream assets this one waits for. This list is the edge set of the DAG — Bruin runs an asset only after every asset in its depends has succeeded.
materialization, columns, custom_checks. How the result is persisted, the typed columns with their quality checks, and any SQL-expressed custom checks — all covered in the next sections.

Why the asset is the powerful unit.

Because metadata sits next to the code, everything about a table — its owner, its dependencies, its tests — is readable in one file and versioned in one commit.
Because SQL and Python are both just assets, a single pipeline can mix a duckdb.sql staging model and a python enrichment job in the same DAG, with a dependency edge between them.
Because name follows schema.table, the asset graph mirrors your warehouse layout, and Bruin's VS Code extension can render lineage straight from the depends edges.

Worked example — a Python asset that depends on a SQL asset

Detailed explanation. Real pipelines mix languages. Here a SQL asset builds a staging.players table, and a Python asset depends on it to compute something SQL is awkward at. The depends edge is what makes Bruin run the SQL asset first and the Python asset second, in one command.

Question. Define a duckdb.sql asset staging.players and a python asset analytics.player_features that depends on it, so bruin run executes them in the right order.

Input. Two files under assets/.

Code.

/* @bruin
name: staging.players
type: duckdb.sql
materialization:
  type: table
@bruin */

select 'ada' as name, 42 as rating
union all
select 'linus' as name, 51 as rating

"""@bruin
name: analytics.player_features
type: python
depends:
  - staging.players
@bruin"""

print("computing features from staging.players")
result = None  # a real asset would read the upstream table and write a result

Step-by-step explanation. The SQL file's @bruin block names it staging.players and materializes it as a table. The Python file's docstring block names it analytics.player_features, sets type: python, and lists staging.players under depends. When you run bruin run, Bruin parses both files, sees the edge staging.players → analytics.player_features, topologically sorts them, and executes the SQL asset before the Python asset. Neither file references the other's code — the depends metadata alone encodes the order.

Output.

asset	type	runs when
`staging.players`	`duckdb.sql`	first (no upstream)
`analytics.player_features`	`python`	after `staging.players` succeeds

Rule of thumb. One file = one asset. Put the language in type, put the order in depends, and let Bruin, not a scheduler you configure separately, decide execution order.

Bruin interview question on the asset model

Question. An interviewer gives you a folder of SQL and Python files and asks: how does Bruin know an asset's table name, which platform runs it, and what must complete before it — without any central manifest listing all of this? Show a SQL asset that answers all three, and explain where each answer comes from.

Solution Using inline YAML metadata and path-based name inference

Code.

/* @bruin
name: analytics.daily_revenue
type: bq.sql
owner: data-team@acme.com
depends:
  - staging.orders
  - staging.refunds
materialization:
  type: table
  strategy: create+replace
@bruin */

select
  order_date,
  sum(amount) as gross,
  sum(refund) as refunds,
  sum(amount) - sum(refund) as net
from staging.orders o
left join staging.refunds r using (order_id, order_date)
group by order_date

Step-by-step trace.

question	answer	where it comes from
table name?	`analytics.daily_revenue`	the `name` key (or the file path if omitted)
which platform?	BigQuery	`type: bq.sql`
what runs first?	`staging.orders`, `staging.refunds`	the `depends` list

Bruin scans every file under assets/ and parses each file's inline @bruin block — there is no central manifest; the metadata is distributed into the files themselves.
name: analytics.daily_revenue sets the target schema.table; had it been omitted, Bruin would have inferred it from the path (assets/analytics/daily_revenue.sql).
type: bq.sql binds the asset to the BigQuery connection so the body runs as BigQuery SQL.
Each entry in depends becomes an inbound edge, so Bruin runs staging.orders and staging.refunds before this asset and fails fast if either upstream fails.

Output:

resolved property	value
target	`analytics.daily_revenue` (BigQuery)
upstream edges	`staging.orders`, `staging.refunds`
execution position	after both upstreams succeed

Why this works — concept by concept:

Inline metadata — because the YAML lives in the same file as the code, an asset is self-describing; there is no separate registry to keep in sync with the files.
Name inference — the schema.table convention (explicit or path-derived) means the asset graph mirrors the warehouse layout without redundant configuration.
Type binds platform — type alone routes the body to BigQuery, Snowflake, DuckDB, or Python, so one project can span engines.
depends is the DAG — the dependency list is the single source of execution order; Bruin never guesses, and lineage is exact.
Cost — parsing is O(assets) at startup; the DAG build is O(V + E) topological sort, negligible next to query runtime.

Pipelines
Topic — pipelines
Pipeline-design and DAG-ordering problems

Practice →

ETL Topic — etl Extract-transform-load asset problems

Practice →

3. Materialization: how a SELECT becomes a table

Bruin wraps your SELECT in the load strategy you choose — from full refresh to merge and SCD2

The feature that turns Bruin from "a SQL runner" into "a warehouse-loading framework" is materialization: you write a plain SELECT, and Bruin applies the DDL and DML needed to persist that result the way you asked. Say it in one breath: type decides table-or-view, strategy decides how rows hit the table. Pick wrong and you get a data-quality bug; pick right and you never hand-write a MERGE again.

The two materialization types.

type: table. Persist the query result as a table. This is the workhorse and the one that carries a strategy.
type: view. Persist the query as a view — no data is copied, the query re-runs on read.

The strategies that matter (all on type: table).

create+replace (default). Overwrite the whole table with the new result every run. A full refresh — simple and correct for small tables, expensive for large ones.
delete+insert. Incremental: needs an incremental_key; Bruin loads the query into a temp table, deletes the target rows whose key values appear in the new batch, then inserts. Good for reprocessing a partition.
truncate+insert. Full replacement that keeps the existing table's schema, permissions, and indices — TRUNCATE then INSERT, no DROP.
append. Only add the new rows, never overwrite — the correct choice for immutable event logs.
merge. Upsert: mark columns with primary_key: true and Bruin updates matched rows and inserts new ones. Use update_on_merge to mark which columns get overwritten on a match, or merge_sql for a custom expression like GREATEST(target.col, source.col).
time_interval. Incrementally load a time window; needs incremental_key and time_granularity (date or timestamp), and you pass --start-date / --end-date.
ddl. Create an empty table from the columns definition only — no query body — when you want the structure created exactly once.
scd2_by_column / scd2_by_time. Maintain full Slowly-Changing-Dimension history with automatic _valid_from, _valid_until, and _is_current columns, tracking changes by column diff or by a time-based incremental key.

Worked example — an incremental merge on a primary key

Detailed explanation. The most common non-trivial materialization is a merge: you have a dimension keyed by an id, rows change, and you want to update the changed ones and insert the new ones without duplicating. Bruin generates the MERGE for you from the columns metadata — you mark the key and the updatable columns and write a plain SELECT.

Question. Materialize a dim.customers table that upserts on customer_id, overwriting email and tier when a customer already exists.

Input.

customer_id	email	tier
5	ada@x	gold
6	linus@x	silver

Code.

/* @bruin
name: dim.customers
type: bq.sql
materialization:
  type: table
  strategy: merge

columns:
  - name: customer_id
    type: integer
    primary_key: true
  - name: email
    type: string
    update_on_merge: true
  - name: tier
    type: string
    update_on_merge: true
@bruin */

select customer_id, email, tier
from staging.customers

Step-by-step explanation. strategy: merge tells Bruin to generate a MERGE statement instead of overwriting the table. Bruin reads the columns block: customer_id is primary_key: true, so it becomes the match key; email and tier carry update_on_merge: true, so they are the columns overwritten with source values when a row matches. On each run, Bruin runs your SELECT as the source, matches on customer_id, updates the matched rows' email/tier, and inserts the unmatched rows. Rows deleted at the source stay in the target — merge never deletes, unlike delete+insert.

Output.

customer_id	result on next run
5 (exists)	`email`/`tier` updated in place
6 (exists)	`email`/`tier` updated in place
9 (new)	inserted

Rule of thumb. Use merge for mutable dimensions keyed by an id; use append for immutable events; reach for delete+insert only when the source truly removes rows and the target must mirror that.

Bruin interview question on tracking history

Question. Product prices change over time and the business needs to answer "what was this product's price on any past date," not just its current price. Which Bruin materialization strategy gives you that with no hand-written history SQL, and what columns does it add?

Solution Using the scd2_by_column strategy

Code.

/* @bruin
name: dim.product_catalog
type: bq.sql
materialization:
  type: table
  strategy: scd2_by_column

columns:
  - name: id
    type: integer
    primary_key: true
  - name: name
    type: string
  - name: price
    type: float
@bruin */

select id, name, price
from staging.products

Step-by-step trace.

run	id 1 price	action taken	resulting rows for id 1
1 (full refresh)	29.99	insert current version	1 current row
2	39.99	expire old, insert new	1 historical + 1 current
3	39.99 (unchanged)	no change detected	unchanged

strategy: scd2_by_column makes Bruin compare each incoming row against the current stored version, keyed by the primary_key column id.
On the first run (--full-refresh), each product is inserted as a current version with _is_current = true, _valid_from = now, and _valid_until = 9999-12-31.
When price changes on run 2, Bruin marks the old row _is_current = false and sets its _valid_until to the change time, then inserts a new current row with the updated price.
A row that disappears from the source is expired (_is_current = false); an unchanged row is left alone. You never wrote a line of history-tracking SQL.

Output:

id	price	_is_current	_valid_from	_valid_until
1	29.99	false	2024-01-01	2024-01-02
1	39.99	true	2024-01-02	9999-12-31

Why this works — concept by concept:

scd2_by_column — a declarative strategy that detects changes in any non-key column and versions the row, so history is a config choice, not bespoke SQL.
Reserved validity columns — _valid_from, _valid_until, and _is_current are added and maintained by Bruin, giving you point-in-time queries with a simple where _valid_from <= d and _valid_until > d.
primary_key drives detection — the key identifies the entity across runs; changes to non-key columns trigger a new version, unchanged rows are skipped.
Idempotent history — re-running with the same source inserts nothing new, because no change is detected — retries are safe.
Cost — the change comparison is O(rows in the batch) against the current-version slice of the target, far cheaper than rebuilding a history table by hand.

ETL
Topic — etl
Load-mode and incremental-materialization problems

Practice →

Transform Topic — data-transformation Upsert, merge and SCD2 history problems

Practice →

4. Built-in & custom quality checks

Quality checks live in the asset file and run right after it — a blocking failure stops the DAG

The feature that sells Bruin to a data engineer who has been burned by silent data corruption is that tests are part of the asset, not a separate suite you hope someone wired up. You declare column-level checks and SQL-based custom checks in the same @bruin block as the query, and Bruin runs them immediately after the asset produces data. A failing check with blocking: true fails the asset and prevents its downstream from running — the bad data never propagates.

Built-in column checks.

Existence and uniqueness. not_null (no nulls in the column) and unique (no value appears twice) — the two you attach to almost every key.
Sign and range. positive, negative, non_negative, and min / max with a value threshold (numbers or dates) constrain the domain.
Membership and shape. accepted_values with a value list restricts to an enum; pattern with a regex enforces a format.
Referential integrity. relationships verifies every non-null value exists in a parent column named by the column's foreign_key metadata — a foreign-key test with no join written by you.

Custom checks — for logic the built-ins can't express.

A custom_checks entry is a SQL query plus an expected value. Bruin runs the query and passes the check only if the returned integer equals value.
Encode business invariants. "Row count is greater than zero," "client X has exactly 15 credits for June," "no order has a negative net after refunds" — anything you can phrase as a query returning a number.
blocking and description. Each check can set blocking (whether a failure stops downstream) and a human description for the report.

How checks run and gate the pipeline.

After the asset, not before. Checks execute once the asset has produced its table, then validate the produced data.
blocking defaults to true. A blocking failure marks the asset failed and stops its downstream; set blocking: false to record the failure without blocking.
retries. A check can retry on failure independently, resolving through the chain check → asset → pipeline.
Run checks alone. bruin run --only checks assets/my_asset.sql re-validates without recomputing the table.

Worked example — column checks on a keyed table

Detailed explanation. The everyday quality pattern is a handful of column checks that encode what "correct" means for a table: the key is unique and never null, a count is positive, a status is one of a fixed set. You attach them in the columns block and Bruin runs them after the asset.

Question. Add checks to a dataset.player_stats table so that name is unique and not null, player_count is not null and positive, and status is one of active or inactive.

Input.

columns:
  - name: name        # unique, not null
  - name: player_count  # not null, positive
  - name: status      # accepted_values: active/inactive

Code.

/* @bruin
name: dataset.player_stats
type: duckdb.sql
materialization:
  type: table
depends:
  - dataset.players

columns:
  - name: name
    type: string
    checks:
      - name: not_null
      - name: unique
  - name: player_count
    type: integer
    checks:
      - name: not_null
      - name: positive
  - name: status
    type: string
    checks:
      - name: accepted_values
        value: [active, inactive]
@bruin */

select name, count(*) as player_count, 'active' as status
from dataset.players
group by name

Step-by-step explanation. Bruin first runs the query and materializes dataset.player_stats. Then it runs the checks column by column: not_null and unique on name, not_null and positive on player_count, and accepted_values on status. Each check is a generated SQL query counting violations; zero violations passes. Because blocking defaults to true, any failing check fails the asset and stops anything that depends on it — so a downstream dashboard model never reads a table with a duplicate key.

Output.

column	check	passes when
`name`	`not_null`, `unique`	no nulls, no duplicates
`player_count`	`not_null`, `positive`	no nulls, all `> 0`
`status`	`accepted_values`	every value in `{active, inactive}`

Rule of thumb. Attach not_null + unique to every key as a reflex; add accepted_values and positive wherever the domain is known — the checks cost one query each and catch the drift that silently corrupts downstream tables.

Bruin interview question on business-rule validation

Question. A finance table tier2.client_credits must satisfy a rule the built-in checks can't express: for June 2024, client X should have exactly 15 rows where credits_spent = 1. If it doesn't, the run must fail before any downstream asset reads the table. How do you encode that in Bruin?

Solution Using a blocking custom check

Code.

/* @bruin
name: tier2.client_credits
type: bq.sql
materialization:
  type: table

custom_checks:
  - name: Client X has 15 credits for June 2024
    description: Guards against the ACME-1234 miscount regression.
    value: 15
    blocking: true
    query: |
      SELECT count(*)
      FROM `tier2.client_credits`
      WHERE client = 'client_x'
        AND date_trunc(start_date, month) = '2024-06-01'
        AND credits_spent = 1
@bruin */

select client, start_date, credits_spent
from staging.credits

Step-by-step trace.

stage	what runs	outcome
1	asset query materializes `tier2.client_credits`	table built
2	custom check query runs, returns a count	e.g. `15`
3	Bruin compares count to `value: 15`	equal → pass; not equal → fail
4	on fail with `blocking: true`	asset marked failed, downstream stopped

Bruin materializes the table first, then executes the custom_checks query against the just-written data.
The query returns a single integer — the number of matching credit rows for client X in June 2024.
Bruin passes the check only if that integer equals the declared value of 15; any other number fails.
Because blocking: true, a failure marks tier2.client_credits as failed and prevents every asset that depends on it from running, so the miscount never reaches a report.

Output:

returned count	check result	downstream
15	pass	runs
14 or 16	fail (blocking)	stopped

Why this works — concept by concept:

Custom check — an arbitrary SQL query plus an expected value lets you encode any business invariant the built-in checks can't express.
Expected-value equality — the pass condition is "query returns exactly value," which turns a fuzzy business rule into a deterministic gate.
blocking gate — blocking: true converts a failed check into a hard stop, so corrupt data cannot flow downstream — a failed run is recoverable, a silently wrong report is not.
Co-located with the asset — the check lives in the same file as the table it guards, so the invariant is versioned and reviewed alongside the query that must satisfy it.
Cost — each check is one aggregate query, O(rows scanned by the query); use partitioned predicates to keep the scan cheap on large tables.

Quality
Topic — data-quality
Built-in and custom data-quality-check problems

Practice →

Validation Topic — data-validation Referential-integrity and constraint-validation problems

Practice →

5. Ingestion with ingestr & multi-destination DAGs

An ingestr asset lands raw data, SQL and Python transform it, and bruin run executes the whole DAG

The last piece that makes Bruin end-to-end is ingestion: instead of a separate connector platform, you declare an ingestr asset — a small YAML file — and Bruin pulls data from a source into a destination as the first node of the same DAG that transforms and tests it. ingestr is Bruin's open-source ingestion CLI, and one line of config replaces a hand-written extract job.

The ingestr asset.

type: ingestr in a .asset.yml. No code body — just parameters naming the source and destination.
parameters. source_connection and source_table say where data comes from; destination says where it lands. The connections themselves live in .bruin.yml.
Any source to any destination. ingestr covers databases, SaaS APIs, and file sources, so "there is no connector for this" is rarely the blocker it is with a hand-rolled script.

The runner and its flags.

bruin run. Execute the whole pipeline — every asset, in dependency order.
bruin run assets/x.sql. Run a single asset (and, with --downstream, everything that depends on it).
bruin run --tag daily. Run only assets carrying a tag — the same graph, sliced by schedule or domain.
bruin run --full-refresh. Drop and rebuild instead of incrementally updating, for a clean reload.
bruin run --start-date … --end-date …. Bound the window for time_interval and ingestr assets.
bruin validate. Statically check every asset — parse the metadata, verify types and dependencies — without running anything, so CI catches a broken pipeline before it touches data.

The DAG and multi-destination pipelines.

depends builds the graph. Bruin reads every asset's depends, assembles a DAG, and runs it in topological order — ingest first, then staging, then marts, then checks.
One pipeline can span platforms. An ingestr asset can land data in Postgres, a duckdb.sql asset can transform it, and a bq.sql asset can publish it — Bruin routes each asset to its own connection by type.
No external orchestrator for a single pipeline. bruin run is the scheduler for one graph; when you need cluster-scale scheduling, Bruin can deploy the same assets to Airflow.

Worked example — an ingestr asset feeding a SQL transform

Detailed explanation. The everyday ingestion pattern is a two-asset chain: an ingestr asset lands a raw table, and a SQL asset depends on it to build a clean model. Running the pipeline executes them in order without you sequencing anything by hand.

Question. Ingest a profiles table from a chess source into DuckDB, then build a dataset.player_stats aggregate that depends on it.

Input. Two files under assets/.

Code.

name: dataset.players
type: ingestr
parameters:
  destination: duckdb
  source_connection: chess-default
  source_table: profiles

/* @bruin
name: dataset.player_stats
type: duckdb.sql
materialization:
  type: table
depends:
  - dataset.players
@bruin */

select name, count(*) as player_count
from dataset.players
group by name

Step-by-step explanation. The .asset.yml file declares an ingestr asset: it reads the chess-default connection's profiles table and lands it in DuckDB as dataset.players. The SQL asset lists dataset.players under depends, so Bruin knows the ingestion must finish before the aggregate runs. bruin run parses both, orders them ingest → transform, executes ingestr to load the raw table, then runs the SELECT to build dataset.player_stats.

Output.

asset	type	produces
`dataset.players`	`ingestr`	raw `profiles` landed in DuckDB
`dataset.player_stats`	`duckdb.sql`	aggregated counts (runs second)

Rule of thumb. Model ingestion as an asset, not a pre-step: put the source in an ingestr .asset.yml and let every downstream model depends on it, so the whole extract-transform chain runs and validates in one bruin run.

Bruin interview question on orchestrating a mixed pipeline

Question. You have four assets — an ingestr load, a SQL staging model, a Python enrichment job, and a final SQL mart with quality checks. An interviewer asks: how does Bruin run these in the correct order across two platforms in one command, and how would you verify the graph is valid before it touches production data?

Solution Using depends-driven topological execution and bruin validate

Code.

/* @bruin
name: mart.daily_active
type: bq.sql
materialization:
  type: table
  strategy: merge
depends:
  - raw.events          # ingestr asset
  - staging.sessions    # duckdb.sql asset
  - features.user_score # python asset

columns:
  - name: user_id
    type: integer
    primary_key: true
    checks:
      - name: not_null
      - name: unique
@bruin */

select user_id, active_date, score
from staging.sessions s
join features.user_score f using (user_id)

Step-by-step trace.

asset	type	depends on	run position
`raw.events`	ingestr	—	1
`staging.sessions`	duckdb.sql	`raw.events`	2
`features.user_score`	python	`staging.sessions`	3
`mart.daily_active`	bq.sql	all three	4, then checks

Bruin parses all four assets and builds the DAG from their depends edges: ingest → staging → features → mart.
bruin validate first runs statically — it parses every @bruin block, checks that referenced dependencies exist, and verifies types and column definitions, all without executing a single query.
On bruin run, Bruin executes in topological order, routing each asset to its platform by type (ingestr, DuckDB, Python, BigQuery) — one command, two-plus platforms.
After mart.daily_active materializes via merge, its not_null and unique checks on user_id run; a blocking failure would stop anything downstream of the mart.

Output:

step	command	effect
pre-flight	`bruin validate`	graph + metadata verified, nothing run
execute	`bruin run`	four assets in order, then checks

Why this works — concept by concept:

depends-driven DAG — the union of every asset's depends list is the graph; Bruin topologically sorts it, so order is derived, never hand-maintained.
Cross-platform routing — type sends each asset to its own connection, so a single DAG spans ingestr, DuckDB, Python, and BigQuery in one run.
validate before run — static validation catches missing dependencies and metadata errors in CI, before any query touches production data.
Checks as the final gate — quality checks on the mart run after materialization and block downstream on failure, closing the loop from ingest to validated output.
Cost — the DAG sort is O(V + E); total runtime is dominated by the assets themselves, and bruin run adds no orchestrator overhead for a single pipeline.

ETL
Topic — etl
Ingestion and extract-load pipeline problems

Practice →

Pipelines Topic — pipelines DAG-orchestration and multi-platform pipeline problems

Practice →

Cheat sheet — Bruin recipes

Minimal SQL asset.

/* @bruin
name: dashboard.hello
type: duckdb.sql
materialization:
  type: table
@bruin */

select 1 as id

Python asset with a dependency.

"""@bruin
name: analytics.features
type: python
depends:
  - staging.players
@bruin"""

print("build features here")

ingestr asset (any source to any destination).

name: raw.profiles
type: ingestr
parameters:
  destination: bigquery
  source_connection: chess-default
  source_table: profiles

Merge / upsert materialization.

/* @bruin
name: dim.customers
type: bq.sql
materialization:
  type: table
  strategy: merge
columns:
  - name: id
    type: integer
    primary_key: true
  - name: email
    type: string
    update_on_merge: true
@bruin */

select id, email from staging.customers

Column + custom quality checks.

/* @bruin
name: dataset.orders
type: duckdb.sql
materialization:
  type: table
columns:
  - name: order_id
    type: integer
    checks:
      - name: not_null
      - name: unique
custom_checks:
  - name: table is not empty
    query: SELECT count(*) FROM dataset.orders
    value: 1
    blocking: true
@bruin */

select order_id from staging.orders

Common bruin commands.

bruin init default my-pipeline      # scaffold a project
bruin validate                      # static-check every asset, run nothing
bruin run                           # run the whole DAG in order
bruin run assets/orders.sql --downstream
bruin run --tag daily --full-refresh
bruin run --only checks assets/orders.sql

Materialization picker.

Situation	Strategy
Small, re-pullable table	`create+replace`
Append-only events	`append`
Mutable rows keyed by an id	`merge` + `primary_key`
Reprocess a partition	`delete+insert` + `incremental_key`
Full change history required	`scd2_by_column` / `scd2_by_time`

Frequently asked questions

What is Bruin?

Bruin is an open-source data pipeline framework — a single Go CLI plus a VS Code extension — that runs SQL transforms, Python jobs, ingestion, and data-quality checks from one project. You define assets (SQL, Python, or YAML files) whose metadata lives in an inline @bruin comment block, declare dependencies with depends, and run the whole DAG with bruin run. It is a tool you install and version in your repo, not a hosted platform.

How is Bruin different from dbt?

dbt is transform-only: it compiles and runs SQL (and Python) models but leaves ingestion, testing infrastructure, and scheduling to other tools. Bruin covers the same SQL-modelling ground and adds native Python assets, ingestr-based ingestion, built-in column and custom quality checks, and a runner that executes the DAG without a separate orchestrator. In short, Bruin aims to be SQL + Python + ingestion + quality in one framework, where dbt is the transform layer of a larger stack.

How do Bruin quality checks work?

You declare checks in the same @bruin block as the asset. Column checks like not_null, unique, positive, accepted_values, pattern, min/max, and relationships attach to a column; custom_checks are SQL queries with an expected integer value. Checks run right after the asset materializes, and a failing check with blocking: true (the default) fails the asset and stops its downstream, so bad data never propagates. You can re-run just the checks with bruin run --only checks.

What materialization strategies does Bruin support?

For type: table, Bruin supports create+replace (full overwrite, the default), append (immutable insert), delete+insert and time_interval (incremental with an incremental_key), truncate+insert (replace data, keep the table), merge (upsert on a primary_key), ddl (create an empty typed table), and scd2_by_column / scd2_by_time for full history with _valid_from / _valid_until / _is_current columns. You write a plain SELECT and Bruin generates the DDL and DML for the strategy you chose. type: view persists the query as a view instead.

What is ingestr and how does Bruin use it?

ingestr is Bruin's open-source ingestion CLI that copies data from any supported source (databases, SaaS APIs, files) into any supported destination. In Bruin you use it by declaring an ingestr asset — a .asset.yml file with parameters naming the source_connection, source_table, and destination, with the connection secrets kept in .bruin.yml. That asset becomes the first node of your DAG, so ingestion, transformation, and quality checks all run in the same bruin run.

Do I need an orchestrator to run Bruin?

Not for a single pipeline. bruin run reads the depends edges, builds the DAG, and executes assets in topological order itself, so it is the scheduler for one graph — you can run it locally, in CI, or in a container. When you need cluster-scale scheduling across many pipelines, Bruin can deploy the same assets to Airflow, and settings like rerun_cooldown translate to Airflow retry semantics. Use bruin validate in CI to catch a broken graph before it runs.

Practice on PipeCode

Pipecode.ai is Leetcode for Data Engineering — every Bruin idea above, from the depends-driven DAG to the merge materialization and the blocking custom check, maps to a hands-on practice room where you build the pipeline against real graded inputs. PipeCode pairs each reading with 450+ DE-focused problems and a real-time scoring engine, so your answer to "how would you gate this pipeline on data quality?" holds up under a senior interviewer's depth probes.

Practice pipeline problems now →
Data-quality drills →

A Local Lakehouse on Your Laptop: DuckDB + Iceberg + Trino for Zero-Cloud Dev

Gowtham Potureddi — Sun, 27 Sep 2026 17:39:50 +0000

local lakehouse means the whole modern data stack — an open table format, object storage, and one or more query engines — running on your machine with nothing rented from a cloud provider. The pieces that used to require an S3 bucket, a Glue catalog, and a Spark cluster now install as a Python wheel, a single binary, and a Docker container. You land Parquet on your own disk, register it as an Apache Iceberg table in a catalog you host, and query it from DuckDB in the same process or from Trino over a socket — all offline, all free, all deterministic.

That shape matters because the slowest part of data engineering is rarely the query; it is the loop. Every change you make to a transform, a partition spec, or a merge key normally has to round-trip through a shared dev warehouse: push, wait for the orchestrator, burn warehouse credits, discover the schema drifted, repeat. A lakehouse you can spin up on a laptop collapses that loop to seconds and makes it reproducible in CI. This guide is a hands-on walkthrough of the four moving parts an interviewer — or your own future test suite — will actually probe: DuckDB reading and writing Parquet, a local Iceberg catalog on the filesystem or MinIO, Trino querying the exact same tables, and partitioning plus compaction done locally. Each section pairs the concepts with a Solution-Tail interview answer: code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the ETL practice library →, tune scans and file layout on the optimization practice set →, and rehearse table layout on the partitioning practice set →.

On this page

Why a local lakehouse beats cloud dev loops in 2026
DuckDB: reading and writing Parquet on the local filesystem
A local Iceberg catalog on the filesystem or MinIO
Trino querying the same Iceberg tables
Partitioning and compaction on your laptop
Cheat sheet — local lakehouse recipes
Frequently asked questions
Practice on PipeCode

1. Why a local lakehouse beats cloud dev loops in 2026

A lakehouse is three swappable layers — table format, storage, engine — and every one of them now runs on a laptop

The one-sentence invariant: a lakehouse is an open table format over plain files, plus a catalog that names those files as tables, plus any engine that speaks the format — and none of those three layers requires a cloud account to run. Once you internalise that separation, "run it locally" stops being a hack and becomes the obvious default for development.

The three layers, decoupled.

Table format (Apache Iceberg). Iceberg is a specification, not a service: a table is a tree of JSON and Avro metadata files (metadata.json → manifest list → manifests) that point at immutable Parquet data files. Nothing about that tree needs the cloud; it can sit in any directory.
Storage. The data and metadata files live somewhere with a path. That "somewhere" can be your local filesystem (file:///Users/you/warehouse) or a local S3-compatible server like MinIO (s3://warehouse). The format does not care which.
Engine. DuckDB, Trino, Spark, and pyiceberg are all clients that read the same Iceberg metadata and Parquet files. You pick the engine per task — DuckDB for a fast in-process scan, Trino for distributed SQL, pyiceberg for writes — and they agree because they share one catalog.

The zero-cloud loop.

Land Parquet on disk. A generator, a pandas/pyarrow frame, or a COPY statement writes columnar files to a folder. No upload, no credentials.
Register and query in-process. DuckDB reads those files with zero server; pyiceberg or Trino registers them as an Iceberg table so schema, snapshots, and partitioning become first-class.
Iterate at memory speed. Because everything is local, a full write-query-verify cycle is milliseconds to seconds, not the minutes a shared warehouse round-trip costs.

Why interviewers and CI both care.

Determinism. A fixed local dataset plus Iceberg snapshot isolation means a test reads exactly the rows it expects, every run, with no "someone else's job changed the shared table" flakiness.
Offline and free. CI runners need no cloud secrets, no network egress, and no per-query bill; the lakehouse spins up inside the job and is torn down with it.
Fidelity. Because the format is identical to production Iceberg, a local test exercises the real partition spec, the real merge semantics, and the real compaction — not a sqlite stand-in that behaves differently.

What an interviewer listens for.

Do you separate "table format vs storage vs engine" cleanly, rather than treating "the lakehouse" as one product? — senior signal.
Do you place DuckDB as an in-process reader and Trino as the distributed engine, using the same tables? — required framing.
Do you reach for a local Iceberg catalog for tests instead of mocking the warehouse? — the whole point of this post.

Worked example — the whole loop in two DuckDB statements

Detailed explanation. Before any catalog or engine setup, the smallest possible proof that a lakehouse loop needs no cloud is a single DuckDB session that writes Parquet to local disk and reads it straight back. DuckDB runs in-process — there is no server to start, no port to bind, no credentials — so this is the innermost loop every later layer wraps.

Question. Using only DuckDB, write two order rows to a local Parquet file and read them back as an aggregate, with no server and no cloud storage.

Input.

order_id	customer	amount	order_date
1	ada	42.50	2026-01-05
2	linus	17.00	2026-02-11

Code.

-- duckdb (in-process; `duckdb` CLI or the Python module — no server)
COPY (
    SELECT * FROM (VALUES
        (1, 'ada',   42.50, DATE '2026-01-05'),
        (2, 'linus', 17.00, DATE '2026-02-11')
    ) AS t(order_id, customer, amount, order_date)
) TO 'lake/orders.parquet' (FORMAT PARQUET);

SELECT customer, sum(amount) AS total
FROM 'lake/orders.parquet'
GROUP BY customer
ORDER BY total DESC;

Step-by-step explanation. COPY (...) TO 'lake/orders.parquet' (FORMAT PARQUET) materialises the query result as a single columnar Parquet file on your local filesystem — DuckDB creates the lake/ directory content and writes typed columns with statistics. The second statement reads that file directly by naming its path in the FROM clause; DuckDB's Parquet reader opens the file, applies the GROUP BY, and returns rows. There is no CREATE TABLE, no connection string, and no object store — the file is the table.

Output.

customer	total
ada	42.50
linus	17.00

Rule of thumb. If a step in your pipeline can be expressed as "read Parquet, transform, write Parquet," it can run locally with zero cloud — the catalog and Trino layers you add later only make those files behave like managed tables.

2. DuckDB: reading and writing Parquet on the local filesystem

`read_parquet`, `COPY ... PARTITION_BY`, and the `iceberg` extension make DuckDB the fast local reader and Parquet producer

DuckDB is the engine you reach for first in a local lakehouse because it is an embedded, columnar SQL engine with a first-class Parquet reader and writer and an optional iceberg extension. It has no server, starts in milliseconds, and pushes filters and projections down into Parquet so a scan touches only the bytes it needs.

Reading Parquet.

Glob paths. read_parquet('lake/orders/**/*.parquet') reads a whole tree; you can also just put the path string in FROM. DuckDB unifies the files into one relation.
Projection pushdown. Selecting three columns from a hundred-column file reads only those column chunks — Parquet is columnar, and DuckDB honours it.
Predicate pushdown + statistics. WHERE amount > 100 uses each row group's min/max stats to skip groups that cannot match, so a filtered scan reads a fraction of the file.

Writing Parquet.

Single file. COPY tbl TO 'out.parquet' (FORMAT PARQUET) writes one file with a chosen compression (CODEC 'zstd').
Hive-partitioned dataset. COPY tbl TO 'lake/orders' (FORMAT PARQUET, PARTITION_BY (event_date)) writes lake/orders/event_date=2026-02-11/data_0.parquet, encoding the partition value in the directory name.
Row-group sizing. ROW_GROUP_SIZE controls how many rows per group, which trades scan granularity against file overhead.

The iceberg extension.

Install once. INSTALL iceberg; LOAD iceberg; adds Iceberg support to a DuckDB session.
Scan a table. SELECT * FROM iceberg_scan('lake/warehouse/lake.db/orders', allow_moved_paths => true) reads an Iceberg table by pointing at its metadata directory, resolving the current snapshot and its Parquet data files.
Attach a REST catalog. Recent DuckDB can ATTACH '' AS ice (TYPE ICEBERG, ENDPOINT 'http://localhost:8181/catalog') and then SELECT * FROM ice.lake.orders, so DuckDB reads the same catalog Trino uses. Iceberg writes from DuckDB are still newer/preview, so most local setups author tables with pyiceberg or Trino and use DuckDB as the fast reader.

Worked example — write a partitioned Parquet dataset, then read one partition

Detailed explanation. The everyday DuckDB pattern is to write a Hive-partitioned Parquet directory and then read back only the partitions you need. Because the partition value is encoded in the folder path, a filter on that column lets DuckDB skip whole directories without opening their files — the local equivalent of partition pruning in a cloud warehouse.

Question. Write three orders partitioned by event_date, then read back only 2026-02-11 and show that DuckDB pruned the other partition.

Input.

order_id	amount	event_date
10	42.5	2026-01-05
11	17.0	2026-02-11
12	9.9	2026-02-11

Code.

-- write a Hive-partitioned dataset
COPY (
    SELECT * FROM (VALUES
        (10, 42.5, DATE '2026-01-05'),
        (11, 17.0, DATE '2026-02-11'),
        (12,  9.9, DATE '2026-02-11')
    ) AS t(order_id, amount, event_date)
) TO 'lake/orders' (FORMAT PARQUET, PARTITION_BY (event_date));

-- read back only one partition
SELECT count(*) AS rows, sum(amount) AS revenue
FROM read_parquet('lake/orders/**/*.parquet', hive_partitioning => true)
WHERE event_date = DATE '2026-02-11';

Step-by-step explanation. The COPY ... PARTITION_BY (event_date) writes two directories — event_date=2026-01-05/ and event_date=2026-02-11/ — each holding the rows for that date. The read uses hive_partitioning => true, so DuckDB reconstructs event_date from the directory names as a real column. The WHERE event_date = DATE '2026-02-11' is evaluated against those directory values first, so DuckDB never opens the January file — it prunes the partition before any Parquet is read.

Output.

rows	revenue
2	26.9

Rule of thumb. Partition by the column you filter on most (usually a date), read with hive_partitioning => true, and DuckDB turns a directory layout into free partition pruning — the same idea Iceberg formalises with hidden partitioning.

DuckDB interview question on Parquet pushdown

Question. You have a directory lake/events of Parquet files partitioned by event_date, one hundred columns wide and billions of rows. An analyst needs sum(amount) for a single month, 2026-02. Write the DuckDB query and explain, concretely, which bytes DuckDB avoids reading and why.

Solution Using hive partitioning with projection and predicate pushdown

Code.

INSTALL parquet;   -- bundled; explicit for clarity
LOAD parquet;

SELECT sum(amount) AS feb_revenue
FROM read_parquet('lake/events/**/*.parquet', hive_partitioning => true)
WHERE event_date >= DATE '2026-02-01'
  AND event_date <  DATE '2026-03-01';

Step-by-step trace.

stage	what DuckDB inspects	what it skips
1. partition prune	directory names `event_date=...`	every folder outside Feb 2026
2. row-group stats	Feb files' min/max for `event_date`, `amount`	row groups whose range misses the filter
3. projection	only the `amount` (and partition) column chunk	the other 98 columns' chunks
4. scan	matching `amount` chunks in surviving row groups	everything else on disk

DuckDB resolves hive_partitioning => true first, reading event_date from the folder path, so the month predicate eliminates all non-February directories before a single Parquet byte is opened.
Inside the surviving files, DuckDB reads each row group's footer statistics and drops groups whose event_date min/max cannot satisfy the range.
Because the query only references amount, DuckDB's projection pushdown reads just that column's chunks — the other columns' bytes are never fetched from disk.
Only the qualifying amount chunks are decompressed and summed, so the scan cost is proportional to one month of one column, not the whole dataset.

Output:

feb_revenue
128934.50

Why this works — concept by concept:

Partition pruning — encoding event_date in the directory path lets DuckDB eliminate whole folders from the plan using metadata alone, turning a full scan into a targeted one.
Row-group statistics — Parquet stores per-group min/max, so even within a partition DuckDB skips groups that cannot match the predicate; this is why sorted or clustered data scans faster.
Projection pushdown — columnar storage means selecting one column reads one column's bytes; a hundred-column table costs the same as a one-column table for this query.
In-process execution — no network hop to a storage service means the only cost is local disk I/O for the surviving bytes, which is what makes the loop feel instant.
Cost — I/O is O(qualifying row groups × selected columns), typically a tiny fraction of the dataset, versus O(all files × all columns) for a naive full read.

ETL
Topic — etl
Local extract-and-load Parquet pipeline problems

Practice →

Pipelines Topic — pipelines Pipeline-design and file-layout problems

Practice →

3. A local Iceberg catalog on the filesystem or MinIO

The catalog is what turns a folder of Parquet into a table — SQLite for pure-filesystem, a REST catalog for sharing

A pile of Parquet files is not a table; it becomes one only when a catalog records the current metadata pointer, the schema, the snapshots, and the partition spec. The catalog is the single piece that makes DuckDB, Trino, and pyiceberg agree on what "the orders table" is. Locally you have two good choices, and picking between them is the decision an interviewer probes.

What a catalog actually stores.

The metadata pointer. For each table, the catalog holds the path to the current metadata.json. That file references the manifest list, which references manifests, which reference the Parquet data files — the whole Iceberg tree hangs off the pointer the catalog owns.
Atomic commits. A write produces a new metadata.json; the commit is the catalog atomically swapping the pointer from the old file to the new one. That swap is why Iceberg gives snapshot isolation — readers see the old snapshot until the pointer moves.
Namespaces and identity. The catalog namespaces tables (lake.orders) and is the authority every engine consults, so "same table" means "same catalog entry," not "same folder."

Option A — SqlCatalog on SQLite (pure filesystem).

Zero infrastructure. pyiceberg's SqlCatalog stores the pointers in a local SQLite file and writes data + metadata to a file:// warehouse directory. No server, no Docker — perfect for a single-process test or a notebook.
Trade-off. SQLite is single-writer and not a network service, so it is ideal when one process owns the catalog but awkward when several tools must share it live.

Option B — a REST catalog (lakekeeper / iceberg-rest) on MinIO.

A shared metadata plane. The Iceberg REST catalog is a standard HTTP API; run lakekeeper (a Rust REST catalog) or the reference iceberg-rest image in Docker and every engine points at one URL. This is how you get DuckDB, Trino, and pyiceberg reading and writing the same table.
MinIO as local S3. MinIO is an S3-compatible object server you run in Docker (minio server /data). Point the warehouse at s3://warehouse with the MinIO endpoint, path-style access, and dummy minioadmin keys — the exact S3 code path you use in production, exercised offline.

Worked example — create an Iceberg table with pyiceberg on a SQLite catalog

Detailed explanation. The fastest way to get a real Iceberg table on a laptop is pyiceberg's SqlCatalog backed by SQLite, writing to a file:// warehouse. You create a namespace, define a schema (or hand it a PyArrow table and let it infer), create the table, and append an Arrow batch. Everything lands on local disk as a proper Iceberg tree.

Question. Using pyiceberg, create a lake.orders Iceberg table on a local SQLite catalog and append two rows, with no cloud and no server.

Input. A two-row PyArrow table of orders.

Code.

import pyarrow as pa
from pyiceberg.catalog.sql import SqlCatalog

catalog = SqlCatalog(          # pointers in SQLite, data in a local folder
    "local",
    uri="sqlite:///warehouse/catalog.db",
    warehouse="file://warehouse",
)

catalog.create_namespace_if_not_exists("lake")

rows = pa.table({
    "order_id": [1, 2],
    "customer": ["ada", "linus"],
    "amount":   [42.50, 17.00],
})

table = catalog.create_table_if_not_exists("lake.orders", schema=rows.schema)
table.append(rows)

print(table.scan().to_arrow().num_rows)   # -> 2

Step-by-step explanation. SqlCatalog(...) opens (or creates) warehouse/catalog.db as the pointer store and sets the warehouse root to the local warehouse/ folder. create_namespace_if_not_exists("lake") registers the namespace. create_table_if_not_exists("lake.orders", schema=rows.schema) writes the first metadata.json under warehouse/lake.db/orders/metadata/ and records its path in SQLite. table.append(rows) writes a Parquet data file, a new manifest and manifest list, a new metadata.json, and atomically updates the SQLite pointer — one Iceberg snapshot. The final scan() reads the current snapshot back as Arrow.

Output.

what pyiceberg created	value
catalog store	`warehouse/catalog.db` (SQLite)
table root	`warehouse/lake.db/orders/`
data files	one Parquet file under `.../data/`
snapshots	1 (the append)
rows in current snapshot	2

Rule of thumb. Use SqlCatalog + SQLite when one process owns the table (unit tests, notebooks); switch to a REST catalog the moment a second engine needs to read or write the same table live.

Iceberg interview question on catalog choice

Question. On a laptop, a pyiceberg job writes lake.orders and a Trino instance must query it at the same time. A teammate suggests "just point both at the same warehouse folder — skip the catalog." Why is that wrong, and what do you set up instead?

Solution Using a shared REST catalog backed by MinIO

Code.

services:                      # docker-compose.yml — local control + storage plane
  minio:                       # local S3
    image: minio/minio
    command: server /data --console-address ":9001"
    environment:
      MINIO_ROOT_USER: minioadmin
      MINIO_ROOT_PASSWORD: minioadmin
    ports: ["9000:9000", "9001:9001"]

  lakekeeper:                  # Iceberg REST catalog
    image: quay.io/lakekeeper/catalog:latest
    environment:
      LAKEKEEPER__PG_DATABASE_URL_READ_WRITE: postgres://catalog:catalog@pg/catalog
    ports: ["8181:8181"]
    depends_on: [pg, minio]

  pg:
    image: postgres:16
    environment:
      POSTGRES_USER: catalog
      POSTGRES_PASSWORD: catalog
      POSTGRES_DB: catalog

from pyiceberg.catalog.rest import RestCatalog   # same URL for writer + reader

catalog = RestCatalog(
    "shared",
    uri="http://localhost:8181/catalog",
    warehouse="demo",
    **{
        "s3.endpoint": "http://localhost:9000",
        "s3.access-key-id": "minioadmin",
        "s3.secret-access-key": "minioadmin",
        "s3.path-style-access": "true",
    },
)

Step-by-step trace.

approach	who moves the metadata pointer	concurrent reader sees
shared folder, no catalog	nobody — no atomic commit	half-written metadata, races
REST catalog (lakekeeper)	the catalog, atomically	old snapshot until commit lands
REST catalog + MinIO	catalog for metadata, S3 for files	consistent snapshot every read

Iceberg's correctness depends on an atomic pointer swap to the current metadata.json; a bare folder has no component that performs that swap, so two processes writing or reading mid-commit can see a partial or inconsistent tree.
A REST catalog (lakekeeper) owns that swap: the writer's append is only visible once the catalog commits the new pointer, giving readers snapshot isolation instead of a race.
Pointing the warehouse at MinIO (s3.path-style-access=true, endpoint :9000) exercises the real S3 code path locally, so the data files are addressed exactly as they would be in the cloud.
Because every engine — pyiceberg writer, Trino reader, DuckDB ATTACH — hits the same URL, "the same table" is defined by one catalog entry, not by whoever happened to look at the folder.

Output:

component	local endpoint	role
lakekeeper	`http://localhost:8181/catalog`	atomic metadata pointer
MinIO	`http://localhost:9000` (`s3://warehouse`)	data + metadata files
result	one `lake.orders` table	shared, snapshot-isolated

Why this works — concept by concept:

Catalog as source of truth — the catalog, not the folder, defines a table; only it can atomically move the current-metadata pointer that every engine reads.
Atomic commit — the pointer swap is the transaction boundary; without it there is no isolation, and concurrent read/write over a shared folder corrupts what readers see.
REST is engine-neutral — the Iceberg REST protocol is a standard, so DuckDB, Trino, and pyiceberg all speak it and converge on identical table state.
MinIO fidelity — running the real S3 API locally means the storage layer behaves like production (path-style, keys, endpoints), so tests catch S3-specific bugs offline.
Cost — a commit is O(1) metadata pointer update plus O(changed files) manifest writes, independent of table size, which is why commits stay cheap as data grows.

Warehouse
Topic — data-warehouse
Lakehouse and warehouse-modelling problems

Practice →

ETL Topic — etl Catalog-driven load and register problems

Practice →

4. Trino querying the same Iceberg tables

Trino's `iceberg` connector reads the catalogued table directly — same metadata, distributed SQL, time travel

Once a table lives in a REST catalog, Trino queries it with its iceberg connector — no copy, no import, just a catalog properties file that points Trino at the same REST endpoint and the same MinIO storage the writer used. This is what makes the local lakehouse a lakehouse and not just "DuckDB on Parquet": full ANSI SQL, joins across tables, snapshot time travel, and DDL, all over the exact bytes pyiceberg wrote.

Wiring Trino to the catalog.

Catalog properties file. Drop etc/catalog/iceberg.properties with connector.name=iceberg and iceberg.catalog.type=rest; Trino exposes it as the iceberg catalog you reference in SQL.
REST endpoint. iceberg.rest-catalog.uri=http://lakekeeper:8181/catalog and iceberg.rest-catalog.warehouse=demo tell Trino which catalog and warehouse to use.
Local S3 filesystem. fs.native-s3.enabled=true, s3.endpoint=http://minio:9000, s3.path-style-access=true, plus the minioadmin keys, let Trino read the data files from MinIO exactly as pyiceberg wrote them.

Querying and writing.

Plain SQL. SELECT * FROM iceberg.lake.orders reads the current snapshot; joins, window functions, and aggregates all work.
CTAS and inserts. CREATE TABLE iceberg.lake.daily AS SELECT ... and INSERT INTO create and grow Iceberg tables from Trino, committed through the same catalog.
Metadata tables. SELECT * FROM iceberg.lake."orders$snapshots" and "...$files" expose snapshots, manifests, and file lists for debugging.

Time travel and isolation.

By timestamp. SELECT * FROM iceberg.lake.orders FOR TIMESTAMP AS OF TIMESTAMP '2026-02-01 00:00:00 UTC' reads the snapshot current at that instant.
By snapshot id. FOR VERSION AS OF <snapshot_id> pins an exact snapshot — invaluable for a deterministic test that must read a known state.
Snapshot isolation. A running SELECT sees one snapshot even as a writer commits new ones, so readers never see a half-written table.

Worked example — point Trino at the shared catalog and query it

Detailed explanation. With lakekeeper and MinIO running, Trino needs exactly one properties file to see the table pyiceberg created. After that, SHOW TABLES and SELECT work as if the table were native, because to Trino it is native Iceberg.

Question. Write the Trino iceberg.properties for the local REST catalog + MinIO, then a query that returns per-customer revenue from lake.orders.

Input. The lake.orders table from Section 3 (columns order_id, customer, amount), already committed to the catalog.

Code.

connector.name=iceberg
iceberg.catalog.type=rest
iceberg.rest-catalog.uri=http://lakekeeper:8181/catalog
iceberg.rest-catalog.warehouse=demo
fs.native-s3.enabled=true
s3.endpoint=http://minio:9000
s3.region=us-east-1
s3.path-style-access=true
s3.aws-access-key=minioadmin
s3.aws-secret-key=minioadmin

-- trino
SELECT customer, sum(amount) AS revenue
FROM iceberg.lake.orders
GROUP BY customer
ORDER BY revenue DESC;

Step-by-step explanation. The properties file registers a Trino catalog named iceberg whose type=rest makes Trino fetch table metadata from lakekeeper at :8181; the fs.native-s3 and s3.* keys let Trino open the Parquet data files from MinIO with path-style addressing. In SQL, iceberg.lake.orders resolves as catalog.schema.table — Trino asks lakekeeper for the current snapshot, reads the manifests, and scans the data files. The GROUP BY runs in Trino's engine and returns aggregated rows.

Output.

customer	revenue
ada	42.50
linus	17.00

Rule of thumb. Trino needs no data movement to read an Iceberg table — one properties file pointing at the shared catalog and storage is the entire integration, because the table format is the contract.

Trino interview question on cross-engine consistency and time travel

Question. pyiceberg wrote lake.orders at 10:00 (snapshot A) and appended more rows at 11:00 (snapshot B). A test must assert Trino reads exactly snapshot A's three rows, regardless of later writes. Write the query and explain why it is deterministic.

Solution Using Iceberg snapshot time travel in Trino

Code.

-- find the snapshots (newest first)
SELECT snapshot_id, committed_at
FROM iceberg.lake."orders$snapshots"
ORDER BY committed_at DESC;

-- pin the exact snapshot the test expects (snapshot A = 10:00)
SELECT count(*) AS rows_at_A
FROM iceberg.lake.orders
FOR TIMESTAMP AS OF TIMESTAMP '2026-02-01 10:30:00 UTC';

-- or pin by explicit id for full determinism
SELECT count(*) AS rows_at_A
FROM iceberg.lake.orders
FOR VERSION AS OF 4823174097 /* snapshot A id */;

Step-by-step trace.

clock	snapshot current	`FOR TIMESTAMP AS OF 10:30` resolves to	rows seen
10:00	A committed	—	3
11:00	B committed	still A (10:30 < 11:00)	3
now	B is latest	A, because 10:30 predates B	3

The orders$snapshots metadata table lists every commit with its snapshot_id and committed_at, so the test can discover or hard-code the snapshot it wants.
FOR TIMESTAMP AS OF TIMESTAMP '... 10:30 ...' tells Trino to resolve the snapshot that was current at 10:30 — that is snapshot A, because B did not commit until 11:00.
FOR VERSION AS OF <snapshot_id> skips the timestamp resolution entirely and reads an exact snapshot, which is the most deterministic form for CI.
Because Iceberg snapshots are immutable, snapshot A's three rows can never change no matter how many later appends land, so the assertion holds on every run.

Output:

rows_at_A
3

Why this works — concept by concept:

Immutable snapshots — every commit creates a new snapshot and never mutates an old one, so a pinned snapshot is a frozen, reproducible view of the table.
Time travel — FOR TIMESTAMP AS OF / FOR VERSION AS OF let a query address a past state by clock or id, which is exactly what a deterministic test needs.
Shared catalog — Trino resolves the snapshot through the same catalog pyiceberg wrote to, so "snapshot A" means the same thing to both engines.
Snapshot isolation — a reader is bound to one snapshot for the whole query, so concurrent writes never leak partial rows into the result.
Cost — time travel is O(1) to resolve the pointer plus O(scanned files) for that snapshot; it reads an old state at no extra bookkeeping cost because the files already exist.

Optimization
Topic — optimization
Query-planning and scan-optimization problems

Practice →

Partitioning Topic — partitioning Partition-pruning and table-layout problems

Practice →

5. Partitioning and compaction on your laptop

Hidden partitioning plus `EXECUTE optimize` fixes the small-files problem locally — the same maintenance you run in production

The last piece that makes a local lakehouse behave like production is table maintenance: partitioning data so scans prune, and compacting the many small files that iterative local writes produce. Iceberg does both with declarative DDL and table procedures, and you run the identical operations locally that you would run against a cloud warehouse — so a test can prove your partition spec and compaction actually help.

Hidden partitioning.

Declare a spec, not a column. WITH (partitioning = ARRAY['day(order_ts)']) partitions by a derived value; you query on order_ts and Iceberg prunes without you materialising a day column. That is "hidden" partitioning.
Transforms. day(ts), month(ts), hour(ts), bucket(N, id), and truncate(N, col) are the built-in partition transforms; bucket(16, user_id) spreads a high-cardinality key into 16 even buckets.
Evolvable. You can change the partition spec later without rewriting history — old data keeps its old spec, new data uses the new one.

The small-files problem.

Why it appears. Each write commits at least one data file per partition; a test loop or a streaming append produces thousands of tiny Parquet files, and every query pays per-file open and metadata overhead.
The symptom. Scans that should be fast crawl, because the engine spends its time opening files and reading footers rather than scanning data.

Compaction and housekeeping (Trino procedures).

Compact. ALTER TABLE iceberg.lake.orders EXECUTE optimize rewrites small files into fewer, right-sized ones; EXECUTE optimize(file_size_threshold => '128MB') only compacts files below the threshold.
Expire snapshots. ALTER TABLE ... EXECUTE expire_snapshots(retention_threshold => '7d') drops old snapshots and the data files only they referenced, reclaiming space.
Remove orphans. ALTER TABLE ... EXECUTE remove_orphan_files(retention_threshold => '7d') deletes files no live snapshot references — the leftovers of failed writes.

Why it matters even locally.

CI stays fast and deterministic. Compacting fixtures keeps test scans quick; expiring snapshots keeps the local warehouse from growing unbounded across CI runs.
You test the maintenance itself. Running optimize locally lets you assert file counts dropped and query time fell — you are validating the production maintenance job, not guessing.

Worked example — a partitioned table, then compact it

Detailed explanation. The clearest local demonstration is to create a partitioned Iceberg table in Trino, write to it a few times so it accumulates small files, then run EXECUTE optimize and watch the file count collapse while the row count stays identical. Nothing leaves your laptop.

Question. Create lake.events partitioned by day(event_ts) and bucket(8, user_id), then compact it after several small appends. Show the file count before and after.

Input. Three appends of ~1,000 rows each, landing many small files across partitions.

Code.

-- trino: create with hidden partitioning
CREATE TABLE iceberg.lake.events (
    user_id   BIGINT,
    event_ts  TIMESTAMP(6),
    amount    DOUBLE
)
WITH (
    partitioning = ARRAY['day(event_ts)', 'bucket(8, user_id)'],
    format = 'PARQUET'
);

-- ... three separate INSERT INTO iceberg.lake.events SELECT ... appends ...

-- inspect small files, then compact
SELECT count(*) AS data_files FROM iceberg.lake."events$files";
ALTER TABLE iceberg.lake.events EXECUTE optimize(file_size_threshold => '128MB');
SELECT count(*) AS data_files FROM iceberg.lake."events$files";

Step-by-step explanation. The partitioning = ARRAY['day(event_ts)', 'bucket(8, user_id)'] clause records a partition spec of a date-day transform plus an 8-way hash bucket, so data is physically grouped by day and user-bucket. Each INSERT commits at least one file per touched partition, so three appends across many partitions leave lots of small files, which events$files counts. EXECUTE optimize(file_size_threshold => '128MB') rewrites all files below 128 MB within each partition into fewer, larger files in a new snapshot, without changing any row. The second count shows the collapse.

Output.

stage	data files	rows
after 3 appends	42	3000
after `EXECUTE optimize`	9	3000

Rule of thumb. Partition by what you filter on, bucket high-cardinality join keys, and schedule EXECUTE optimize after bursty writes — locally you can prove the file count dropped, so you ship the maintenance job with confidence.

Iceberg interview question on local compaction and snapshot hygiene

Question. A local CI job writes a fixture in a loop and leaves 10,000 tiny files plus dozens of stale snapshots; Trino scans have become slow and the warehouse folder keeps growing between runs. Fix both the scan speed and the growth, and justify why this matters in CI, not just production.

Solution Using EXECUTE optimize plus expire_snapshots

Code.

-- 1) compact small files into right-sized ones (new snapshot)
ALTER TABLE iceberg.lake.events
EXECUTE optimize(file_size_threshold => '128MB');

-- 2) drop snapshots older than the retention window and their
--    now-unreferenced data files
ALTER TABLE iceberg.lake.events
EXECUTE expire_snapshots(retention_threshold => '7d');

-- 3) sweep files no live snapshot references (failed-write leftovers)
ALTER TABLE iceberg.lake.events
EXECUTE remove_orphan_files(retention_threshold => '7d');

Step-by-step trace.

step	procedure	effect on files	effect on snapshots
1	`optimize`	10,000 small → ~40 sized	+1 (compaction snapshot)
2	`expire_snapshots`	deletes files only old snapshots used	drops stale snapshots
3	`remove_orphan_files`	deletes unreferenced leftovers	none

EXECUTE optimize rewrites the 10,000 tiny files into a handful of ~128 MB files, so every subsequent scan opens dozens of files instead of ten thousand — the scan-speed fix.
Compaction itself adds a new snapshot and leaves the old small files still referenced by earlier snapshots, so space does not drop yet; that is expected.
EXECUTE expire_snapshots removes snapshots past the retention window and physically deletes the data files that only those expired snapshots referenced — this is where the folder finally shrinks.
EXECUTE remove_orphan_files deletes files on storage that no live snapshot points to at all (typically from interrupted writes), reclaiming the last of the wasted space.

Output:

metric	before	after maintenance
data files	10,000	~40
live snapshots	60+	a few (within 7d)
warehouse size	bloated	lean
Trino scan time	slow	fast

Why this works — concept by concept:

Compaction — optimize trades a one-time rewrite for cheap steady-state scans by replacing many small files with few right-sized ones, eliminating per-file open overhead.
Snapshot retention — old snapshots pin their data files alive; expire_snapshots is what actually frees space, because compaction alone only adds a snapshot.
Orphan cleanup — remove_orphan_files reclaims files no snapshot references, which accumulate from failed or interrupted local writes during a test loop.
CI relevance — a fixture that bloats every run makes CI slower and flakier over time; running the real maintenance locally keeps runs fast and validates the production job.
Cost — optimize is O(rewritten bytes) once; the payoff is O(files) reduced per query forever after, and the housekeeping procedures are O(expired files) to delete.

Partitioning
Topic — partitioning
Partition-spec and file-sizing problems

Practice →

Optimization Topic — optimization Compaction and scan-cost optimization problems

Practice →

Cheat sheet — local lakehouse recipes

DuckDB — write and read Parquet.

COPY tbl TO 'lake/orders' (FORMAT PARQUET, PARTITION_BY (event_date));
SELECT * FROM read_parquet('lake/orders/**/*.parquet', hive_partitioning => true)
WHERE event_date = DATE '2026-02-11';

DuckDB — read an Iceberg table.

INSTALL iceberg; LOAD iceberg;
SELECT * FROM iceberg_scan('warehouse/lake.db/orders', allow_moved_paths => true);
-- or attach the shared REST catalog:
ATTACH '' AS ice (TYPE ICEBERG, ENDPOINT 'http://localhost:8181/catalog');
SELECT * FROM ice.lake.orders;

pyiceberg — create + append on a SQLite catalog.

from pyiceberg.catalog.sql import SqlCatalog
cat = SqlCatalog("local", uri="sqlite:///warehouse/catalog.db", warehouse="file://warehouse")
cat.create_namespace_if_not_exists("lake")
t = cat.create_table_if_not_exists("lake.orders", schema=arrow_table.schema)
t.append(arrow_table)

Trino — Iceberg REST catalog on MinIO.

connector.name=iceberg
iceberg.catalog.type=rest
iceberg.rest-catalog.uri=http://lakekeeper:8181/catalog
iceberg.rest-catalog.warehouse=demo
fs.native-s3.enabled=true
s3.endpoint=http://minio:9000
s3.path-style-access=true

Trino — partitioned table + time travel.

CREATE TABLE iceberg.lake.events (user_id BIGINT, event_ts TIMESTAMP(6), amount DOUBLE)
WITH (partitioning = ARRAY['day(event_ts)', 'bucket(8, user_id)']);
SELECT * FROM iceberg.lake.events FOR TIMESTAMP AS OF TIMESTAMP '2026-02-01 10:30:00 UTC';

Trino — compaction and housekeeping.

ALTER TABLE iceberg.lake.events EXECUTE optimize(file_size_threshold => '128MB');
ALTER TABLE iceberg.lake.events EXECUTE expire_snapshots(retention_threshold => '7d');
ALTER TABLE iceberg.lake.events EXECUTE remove_orphan_files(retention_threshold => '7d');

Docker — the storage + catalog plane.

docker run -p 9000:9000 -p 9001:9001 minio/minio server /data --console-address ":9001"
docker run -p 8181:8181 quay.io/lakekeeper/catalog:latest   # Iceberg REST catalog

Engine roles at a glance.

Component	Role in the local lakehouse
DuckDB	in-process reader + Parquet producer
pyiceberg	Python writer, catalog + table author
lakekeeper / iceberg-rest	shared Iceberg REST catalog
MinIO	local S3-compatible object storage
Trino	distributed SQL, DDL, compaction, time travel

Frequently asked questions

What is a local lakehouse?

A local lakehouse is a full lakehouse stack — an open table format (Apache Iceberg), object or file storage, and query engines — running entirely on your own machine with no cloud account. You land Parquet files on disk or in a local MinIO bucket, register them as Iceberg tables in a catalog you host (a SQLite SqlCatalog or a REST catalog like lakekeeper), and query them from DuckDB or Trino. Because the table format is identical to production Iceberg, local dev and CI exercise the real semantics, not a stand-in.

Can DuckDB write Iceberg tables?

DuckDB's iceberg extension has mature read support — iceberg_scan(...) reads a table by its metadata path, and recent versions can ATTACH an Iceberg REST catalog and query its tables. Iceberg write support from DuckDB is newer and still maturing, so most local setups author and mutate Iceberg tables with pyiceberg or Trino and use DuckDB as a fast reader and a Parquet producer. DuckDB writing plain Parquet, by contrast, is fully supported and is how you feed data into the lakehouse.

Do I need MinIO, or is the local filesystem enough?

For a single process — a unit test or a notebook — a file:// warehouse with a SQLite SqlCatalog is enough and needs no Docker at all. Add MinIO when you want to exercise the real S3 code path (path-style addressing, endpoints, credentials) or when multiple engines must share storage. MinIO is S3-compatible, so the same s3:// configuration you test locally works unchanged against real S3 in production.

How do DuckDB, Trino, and pyiceberg share the same Iceberg table?

They share a catalog. The catalog holds the current metadata pointer for each table, and every engine that points at the same catalog URL sees the same table state. Run an Iceberg REST catalog (lakekeeper or the reference iceberg-rest) and configure pyiceberg's RestCatalog, Trino's iceberg.catalog.type=rest, and DuckDB's ATTACH ... (TYPE ICEBERG, ENDPOINT ...) to the one URL. Sharing a folder without a catalog does not work, because only the catalog performs the atomic commit that gives snapshot isolation.

How do I compact small files in a local Iceberg table?

Run ALTER TABLE iceberg.lake.events EXECUTE optimize(file_size_threshold => '128MB') in Trino to rewrite small files into fewer right-sized ones. Follow it with EXECUTE expire_snapshots(retention_threshold => '7d') to drop old snapshots and reclaim the space their data files held, and EXECUTE remove_orphan_files(...) to sweep leftovers from failed writes. Compaction alone does not shrink storage — you must expire snapshots for the old files to be deleted.

Why run a lakehouse locally instead of a dev cloud account?

Speed, determinism, and cost. A local loop of write-query-verify is seconds, not the minutes a shared warehouse round-trip costs, and it needs no credentials or network. Iceberg snapshot isolation plus a fixed local dataset make tests reproducible, so CI stops being flaky from shared-state collisions. And because CI runners spin the lakehouse up inside the job, there is no cloud bill, no secret management, and no egress — while still testing the exact table format, partitioning, and compaction you run in production.

Practice on PipeCode

Pipecode.ai is Leetcode for Data Engineering — every local-lakehouse idea above, from DuckDB Parquet pushdown to the shared Iceberg REST catalog, Trino time travel, and `EXECUTE optimize` compaction, maps to a hands-on practice room where you build the load against real graded inputs. PipeCode pairs each reading with 450+ DE-focused problems and a real-time scoring engine, so your answer to "how would you make this scan prune and this table stay compact?" holds up under a senior interviewer's depth probes.

Practice ETL problems now →
Optimization drills →

Data Residency & Sovereignty: Multi-Region Architectures for Compliant Data Platforms

Gowtham Potureddi — Sun, 27 Sep 2026 17:37:05 +0000

data residency is a claim about geography — it says the physical bytes of a given dataset are stored and processed inside a named region, and never leave it. That sounds like a one-line configuration flag, and for a single bucket it almost is. But a real data platform is a mesh of storage layers, warehouses, replication jobs, caches, backups, key stores, and query engines, and residency is only satisfied when every one of those components keeps the row inside the boundary. The moment a nightly cross-region replica, a global materialized view, or a helpfully-replicated encryption key carries a byte across the line, the guarantee is gone — usually silently, and usually discovered in an audit rather than a test.

Residency is also routinely confused with two neighbours it is not. It is not data sovereignty, which is the harder question of whose laws can compel access to the data regardless of where it sits — a US-headquartered provider can be reachable under the US CLOUD Act even for bytes stored in Frankfurt. And it is not disaster recovery, which is about keeping copies for availability; DR wants your data in more than one place, residency wants it in exactly one jurisdiction, and the two goals pull in opposite directions. This guide walks the four design problems an interviewer will actually probe — region-pinned storage with geo-partitioning, per-region warehouse deployments with hard replication boundaries, encryption with in-region key residency, and cross-region aggregate-only egress — and pairs each with a Solution-Tail answer: code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the partitioning practice library →, rehearse the region-shard decisions on the sharding practice set →, and harden who-can-read-what on the access-control practice set →.

On this page

Why residency and sovereignty are different problems
Region-pinned storage & geo-partitioning
Regional warehouses & replication boundaries
Encryption, key residency & BYOK
Cross-region aggregate-only egress
Cheat sheet — residency recipes
Frequently asked questions
Practice on PipeCode

1. Why residency and sovereignty are different problems

Residency is about geography, sovereignty is about jurisdiction — conflate them and you build the wrong control

The one-sentence invariant: residency answers "where is the data physically stored and processed?" while sovereignty answers "whose legal authority can reach it?" — and a design can satisfy one while violating the other. You can host EU customer data in an EU region (residency satisfied) on a provider whose parent company is legally compellable under a foreign statute (sovereignty not satisfied). Interviewers who work in regulated industries open with this distinction on purpose, because a candidate who treats the two as synonyms will pick a control that looks compliant on an architecture diagram but fails the actual legal test.

The two questions, kept apart.

Residency — where the bytes live. A physical-location property of every copy of the data: primary storage, replicas, backups, temp files, query spill, log lines that echo payloads, and CDN caches. Residency is satisfied only when the union of all those locations stays inside the allowed region set.
Sovereignty — whose laws govern. A jurisdictional property: which government can lawfully compel disclosure or seizure. This depends on where the operator is incorporated, where its staff and support sit, and which treaties apply — not only where the disk is. The US CLOUD Act and various national-security statutes are the usual examples of extraterritorial reach.
Why the gap matters. The Schrems II ruling turned on exactly this gap: storing EU data in the EU was not sufficient if a non-EU authority could still compel it. Sovereignty controls (in-region operator, customer-held keys, confidential computing) exist precisely to close a gap that residency alone leaves open.

Distinct from disaster recovery — the copies problem.

DR wants redundancy across failure domains. The instinct of every availability design is "replicate to a second region so a regional outage does not take us down." That instinct directly manufactures a residency violation if the second region is in a different jurisdiction.
Residency wants a bounded footprint. The reconciliation is to keep DR inside the residency boundary — a second availability zone or a second in-jurisdiction region — never a convenient far-away region. "Back it up to us-east-1 for safety" is the classic way an EU-residency platform quietly breaks.
Backups and snapshots count. A backup is a copy; a copy has a location; that location is in scope. Cross-region backup replication is the single most common silent residency breach because it is configured once, for good reasons, and then forgotten.

The three legal forces that drive the architecture.

Data-localization laws. Some jurisdictions require certain data to be stored (and sometimes processed) only inside the country — for example national rules covering personal, health, financial, or government data. These force region-pinning and per-region deployments.
Cross-border transfer rules. Frameworks like the GDPR permit transfers only under specific mechanisms (adequacy decisions, standard contractual clauses, supplementary measures). These force you to prove what leaves a region and under what basis.
Extraterritorial reach. Statutes that let a government compel a provider regardless of storage location. These force sovereignty controls that residency cannot provide — customer-controlled keys and, at the extreme, sovereign-cloud operators.

What interviewers listen for.

Do you separate "where it is stored" from "who can compel it" in the first two sentences? — senior signal.
Do you flag that DR replication is the usual way residency breaks, unprompted? — required framing.
Do you name backups, temp files, and logs as in-scope copies, not just the primary table? — the detail that separates a real design from a diagram.
Do you reach for customer-held keys when the interviewer adds "but the operator is foreign"? — the sovereignty move.

Worked example — the same customer, two very different guarantees

Detailed explanation. The cleanest way to feel the distinction is to hold the storage location fixed and vary only the operator, then hold the operator fixed and vary only the storage. Each move flips exactly one of the two properties, which is the whole point: residency and sovereignty are independent axes, and a compliant design has to satisfy both simultaneously rather than assuming one implies the other.

Question. For an EU customer whose personal data must stay in the EU and must not be reachable by a non-EU authority, classify four deployment options as residency-OK / sovereignty-OK.

Input.

option	storage region	operator jurisdiction
A	EU (Frankfurt)	non-EU parent company
B	US (Virginia)	non-EU parent company
C	EU (Frankfurt)	EU sovereign operator
D	EU (Frankfurt)	EU operator, customer-held keys

Code.

residency_ok  = (storage_region in ALLOWED_REGIONS)
sovereignty_ok = (operator_in_jurisdiction) OR (customer_controls_keys)

A: residency_ok=True,  sovereignty_ok=False   # in-region, but compellable abroad
B: residency_ok=False, sovereignty_ok=False   # wrong region entirely
C: residency_ok=True,  sovereignty_ok=True    # in-region + in-jurisdiction operator
D: residency_ok=True,  sovereignty_ok=True    # in-region + crypto control of access

Step-by-step explanation. Option A stores in the EU so residency passes, but the foreign parent can be compelled, so sovereignty fails — this is the Schrems II shape exactly. Option B fails both because the bytes are in the wrong region. Option C satisfies both by using an operator that is itself inside the jurisdiction. Option D keeps a commercial operator but moves the access control to the customer via customer-held encryption keys, so even a lawful demand to the operator yields ciphertext — sovereignty satisfied by cryptography rather than by corporate structure.

Output.

option	residency	sovereignty	verdict
A	OK	FAIL	non-compliant
B	FAIL	FAIL	non-compliant
C	OK	OK	compliant (sovereign operator)
D	OK	OK	compliant (customer keys)

Rule of thumb. Fix the storage region first to satisfy residency, then ask "who could still be compelled?" to satisfy sovereignty — if the answer is a party you do not control, you need in-region operation or customer-held keys, not a second region.

2. Region-pinned storage & geo-partitioning

Pin the bucket to a region and partition the fact table on a region key — location becomes a column, not a hope

The feature that makes residency enforceable rather than aspirational is that in modern storage the region is an immutable property of the container and the data model carries the region as a first-class key. You pin the S3 bucket / GCS bucket / warehouse dataset to a region at creation, you disable any replication that would copy it elsewhere, and you make region a partition key on every table that holds regulated rows so a row's residency is visible in the schema and enforceable in every query.

Pinning the storage container.

Region is set at creation and often immutable. An S3 bucket lives in one region for life; a BigQuery dataset's location (EU, US, europe-west3) is fixed when created and cannot be changed — you recreate to move. Treat the region as part of the resource's identity.
Block the replication features by default. S3 Cross-Region Replication, cross-region snapshot copy, and global-table features are opt-in conveniences that break residency. For a residency-bound bucket, the correct posture is an explicit deny (a bucket policy / SCP that forbids replication to non-allowed regions), not merely "we did not turn it on."
Watch the implicit copies. Query engines spill to temp storage, CDNs cache objects at edges worldwide, and logs can echo payloads. Each of those has a region; each must be constrained to the boundary or scrubbed of regulated content.

Geo-partitioning the data model.

Make region (or country) a partition key. PARTITION BY region (or a Hive-style region=EU/ path layout on object storage) physically groups each region's rows into their own files / partitions, so residency is a property you can see, scan, and enforce per partition.
Partition pruning routes the query. A query filtered WHERE region = 'EU' reads only the EU partitions — the US partitions are pruned and never touched. This is both a performance win and a residency control: an EU-scoped job provably cannot read US data.
Bucketing within a region for skew. If one region is huge, add a secondary bucket/shard on a high-cardinality key (customer_id) inside the region so files stay evenly sized without ever mixing regions.

Row-domiciled databases (the OLTP side).

CockroachDB REGIONAL BY ROW. Each row carries a hidden crdb_region column; the database physically homes the row's replicas in that region's nodes, so a single logical table transparently keeps EU rows in EU nodes and US rows in US nodes.
Citus / Yugabyte tablespaces. Distribute a Postgres table by a region/tenant key and pin each shard's tablespace to region-local storage — the same "location is a key" idea in a Postgres-compatible engine.
The invariant. Whether OLTP or lakehouse, the design rule is identical: the region is part of the primary/partition key, and the storage for each key value is physically in that region.

Worked example — geo-partitioning a fact table and proving the prune

Detailed explanation. The everyday pattern is a fact table partitioned on region where each partition maps to region-local storage. The proof that residency holds is that a region-scoped query only ever scans that region's partition — you can read the guarantee straight out of the query plan, which is exactly what an auditor (or interviewer) wants to see.

Question. Partition an events table by region, then show that a query for EU events scans only the EU partition and never touches US storage.

Input.

event_id	region	user_id	amount
1	EU	u-eu-1	40
2	US	u-us-1	12
3	EU	u-eu-2	25
4	US	u-us-2	8

Code.

-- Region is a partition key, so each region's rows live in their own partition/storage
CREATE TABLE events (
    event_id BIGINT,
    region   STRING,        -- 'EU' | 'US'
    user_id  STRING,
    amount   DOUBLE
)
PARTITIONED BY (region);

-- Regional read: the planner prunes to a single partition
SELECT region, COUNT(*) AS n, SUM(amount) AS total
FROM events
WHERE region = 'EU'
GROUP BY region;

Step-by-step explanation. PARTITIONED BY (region) lays the four rows into two partitions — region=EU/ holding events 1 and 3, and region=US/ holding events 2 and 4 — each of which can be pinned to region-local storage. When the query filters WHERE region = 'EU', the planner performs partition pruning: it resolves the predicate against the partition key at plan time and lists only the region=EU/ files, so the US partition is never opened. The residency guarantee is therefore structural, not procedural — the EU job has no code path that reads US bytes.

Output.

region	n	total
EU	2	65

Rule of thumb. If residency is not visible as a partition key you can filter on, you cannot prove it in a query plan — make region a partition key first, everything else (pruning, per-region storage, per-region access) follows.

SQL interview question on geo-partitioning for residency

Question. An interviewer gives you a global orders table and a requirement that an EU analyst's session must be physically incapable of scanning non-EU rows. Using partitioning, how do you both store EU rows separately and guarantee an EU-scoped query prunes everything else? Show the DDL and the pruned read.

Solution Using partitioning on a region key with partition pruning

Code.

CREATE TABLE orders (
    order_id  BIGINT,
    region    STRING,          -- residency key: 'EU' | 'US' | 'APAC'
    customer  STRING,
    amount    DECIMAL(12,2),
    created_at TIMESTAMP
)
PARTITIONED BY (region);

-- Each partition is pinned to region-local storage at load time, e.g.
--   region=EU   -> s3://eu-central-1-orders/ (bucket locked to EU)
--   region=US   -> s3://us-east-1-orders/    (bucket locked to US)

-- EU analyst session runs only region-scoped reads:
SELECT customer, SUM(amount) AS lifetime_value
FROM orders
WHERE region = 'EU'            -- prunes US and APAC partitions
GROUP BY customer;

Step-by-step trace.

step	partitions on disk	predicate	partitions scanned
1	EU, US, APAC	(load time)	—
2	EU, US, APAC	`region = 'EU'`	EU only
3	US, APAC	(never listed)	pruned
4	EU	aggregate	grouped in region

PARTITIONED BY (region) writes each region's rows into a separate partition directory that is loaded into a region-pinned bucket, so the physical bytes for US and APAC are not even in EU storage.
The EU query's WHERE region = 'EU' is a partition predicate; the planner evaluates it against partition metadata and lists only the EU files.
The US and APAC partitions are pruned — no file handle is opened, so an EU session has no path to non-EU bytes even with a bug in the WHERE clause on non-partition columns.
The aggregate runs entirely within the EU partition, so both the scan and the result stay in region.

Output:

customer	lifetime_value
c-eu-ada	240.00
c-eu-lin	95.50

Why this works — concept by concept:

Region as partition key — promoting region to a partition key turns residency from a runtime hope into a physical layout, so each region's rows occupy their own files in their own pinned storage.
Partition pruning — because the filter is on the partition key, the planner discards non-EU partitions at plan time; an EU session cannot open US files, which is a stronger guarantee than row-level filtering that still scans everything.
Pinned storage per partition — mapping each partition to a region-locked bucket means the boundary is enforced by the storage layer, not only by the query, so a rogue full-scan still cannot pull a US byte into EU compute.
Auditable plan — the pruned plan is the evidence: you can show an auditor that the EU workload lists only EU partitions, which is exactly the proof residency reviews demand.
Cost — pruning makes the scan O(rows in one region) instead of O(rows globally), so residency and query cost improve together rather than trading off.

Partitioning
Topic — partitioning
Partition-key and partition-pruning problems

Practice →

Sharding Topic — sharding Region-shard and data-placement problems

Practice →

3. Regional warehouses & replication boundaries

Give each region its own warehouse and draw a hard line replication may not cross — then let only metadata across

Partitioning keeps rows apart inside one system; the next layer up keeps whole systems apart. The compliant multi-region pattern is one warehouse deployment per region — a Snowflake account, a BigQuery project/dataset, or a Redshift cluster pinned to that region — with an explicit replication boundary between them that raw regulated data is never allowed to cross. The subtlety interviewers love is that some things should cross the boundary (schema, lineage, tags, policies) and some things must never (the actual rows), so the boundary is selective, not a total wall.

One warehouse per region.

Location is immutable and per-deployment. A BigQuery dataset's location and a Snowflake account's region are fixed at creation. You do not have one global warehouse with a residency flag; you have N regional warehouses and a routing layer that sends each tenant's queries to the right one.
Sharding the platform by region. Treating each region as an independent shard means a breach, outage, or noisy-neighbour problem in one region is contained to that region's blast radius — the US shard failing does not touch EU data or EU availability.
Routing by residency key. The application resolves a tenant's region and dispatches to that region's warehouse. The residency key from Section 2 becomes the routing key here — same column, different layer.

The replication boundary — what must not cross.

Disable cross-region replication of data. Snowflake database replication, BigQuery cross-region dataset copies, Redshift cross-region snapshot copy, and Kafka MirrorMaker are all capable of copying regulated rows across the line. For a residency-bound deployment these are denied by policy, with an allow-list only for non-regulated or already-aggregated data.
Geo-fence the streaming layer. If Kafka feeds the warehouse, the topics carrying regulated data must live in region-local clusters and MirrorMaker mirroring must be scoped so a EU topic is not mirrored into a US cluster. The stream is as much in scope as the table.
Backups stay in region. Cross-region backup/snapshot copy is the most common accidental breach; the backup destination must be an in-region (or in-jurisdiction) location, verified in policy, not assumed.

What may cross — the metadata / data-plane split.

Central control plane, regional data plane. Catalog metadata — table names, schemas, column types, lineage, ownership, classification tags, and access policies — is generally not the regulated payload and can live in a central catalog (e.g. a single governance catalog) that spans regions. The rows themselves stay in the regional data plane.
Why the split is safe. Knowing that an EU table customers has a email column of type string tagged PII is metadata; the actual email values are data. Sharing the former globally lets you run one governance model while the latter never leaves the EU.
The trap. Metadata must be genuinely free of payload — column statistics like min/max, sample values, or histograms can leak actual data (a min email, a sampled name), so residency-grade catalogs suppress value-bearing statistics for regulated columns.

Worked example — routing a tenant to its regional warehouse

Detailed explanation. The mechanical heart of the pattern is a resolver: given a tenant, return the region, and from the region return the warehouse connection. No query ever names a region literally; it always goes through the resolver, so there is exactly one place that enforces "this tenant's data lives here," and it is trivial to audit.

Question. Implement a router that sends a tenant's read to its region's warehouse and refuses if the requested region is outside the tenant's allowed set.

Input.

tenant	home_region	requested_region
acme-eu	EU	EU
globex-us	US	US
acme-eu	EU	US

Code.

WAREHOUSES = {
    "EU": "snowflake://eu-central-1/acct_eu",
    "US": "snowflake://us-east-1/acct_us",
}

def route(tenant_home_region: str, requested_region: str) -> str:
    if requested_region != tenant_home_region:
        raise PermissionError(
            f"residency violation: tenant homed in {tenant_home_region} "
            f"may not query {requested_region}"
        )
    return WAREHOUSES[requested_region]      # region-pinned warehouse

Step-by-step explanation. Every request carries the tenant's home_region (resolved from the residency key) and the region the query wants to touch. The router refuses any request where those differ, so an EU tenant cannot be pointed at the US warehouse even by a mis-built query. When they match, it returns the connection string for that region's own warehouse account, which is physically pinned to the region — the boundary is enforced at the connection layer before a single row is read.

Output.

tenant	requested_region	result
acme-eu	EU	routed to acct_eu
globex-us	US	routed to acct_us
acme-eu	US	PermissionError (blocked)

Rule of thumb. Never let application code name a warehouse region directly — force every query through one resolver keyed on the tenant's residency, so the boundary lives in one auditable function instead of scattered across the codebase.

SQL interview question on replication boundaries

Question. Product wants a single global dashboard of order counts, but raw EU orders may not leave the EU. An engineer proposes replicating the EU orders table into the US warehouse to make the join easy. Explain why that breaks residency and design the replication so the boundary holds while the dashboard still works.

Solution Using region-local aggregation before any cross-region copy

Code.

-- WRONG: replicate raw EU rows to US so a global query can read them
--   CREATE TABLE us.orders_eu_copy AS SELECT * FROM eu.orders;   -- residency breach

-- RIGHT: aggregate inside the EU, replicate only the non-personal summary
-- (runs in the EU warehouse, on EU storage)
CREATE TABLE eu.orders_daily AS
SELECT
    order_date,
    'EU'        AS region,
    COUNT(*)    AS order_count,
    SUM(amount) AS revenue
FROM eu.orders
GROUP BY order_date;          -- no customer, no row-level PII

-- Only orders_daily (counts + revenue, no personal data) is allowed
-- across the replication boundary into the shared reporting layer.

Step-by-step trace.

step	location	data shape	crosses boundary?
1	EU warehouse	raw `orders` (PII)	never
2	EU warehouse	`orders_daily` aggregate	eligible
3	boundary policy	check: no PII columns	allow
4	US / shared layer	`orders_daily` unioned	yes (safe)

Replicating raw orders copies personal rows into US storage governed by US law — that is the exact residency breach, no matter that the query "only wanted counts."
Instead, the aggregation runs inside the EU, on EU storage, producing orders_daily that has date, region, count, and revenue — no customer, no personal field.
The replication boundary policy inspects the object: because orders_daily carries no PII-tagged columns, it is on the allow-list to cross.
The shared dashboard unions each region's orders_daily, so the global view exists while raw rows never left their home region.

Output:

order_date	region	order_count	revenue
2026-03-01	EU	1204	48210.00
2026-03-01	US	3391	121750.00

Why this works — concept by concept:

Replication boundary — the design decision is what may cross, not merely whether a link exists; raw regulated rows are denied and only derived, non-personal summaries are allowed.
Aggregate-in-region — computing the summary on EU storage means the personal rows are consumed where they live and only the reduced, safe result is a candidate for egress.
Policy on the object, not the intent — the boundary check inspects the actual columns/tags of what is crossing, so "we only wanted counts" cannot smuggle a SELECT *.
Regional sharding — each warehouse stays an independent shard, so the shared reporting layer depends on small summaries rather than a fragile cross-region raw replica.
Cost — moving kilobytes of daily aggregates instead of gigabytes of raw orders is both cheaper and the compliant path, so correctness and egress cost align.

Sharding
Topic — sharding
Region-shard and boundary-isolation problems

Practice →

Partitioning Topic — partitioning Per-region layout and pruning problems

Practice →

4. Encryption, key residency & BYOK

Encrypt every row, but keep the key in the jurisdiction — because whoever holds the key controls the data

Region-pinning and boundaries control where the plaintext sits. Encryption plus key residency controls who can turn ciphertext back into plaintext, and that is the lever that actually delivers sovereignty. The design is envelope encryption: each row (or file) is encrypted with a data key, the data key is wrapped by a customer master key, and the master key is held in a key store that resides in the jurisdiction and is controlled by you — so a lawful demand served on a foreign operator yields unreadable bytes, and revoking the key is a one-move "crypto-shred" of the data.

Envelope encryption — the mechanism.

Two-tier keys. A per-object data encryption key (DEK) encrypts the bytes; a customer master key (CMK / KEK) encrypts the DEK. Only the small wrapped DEK travels with the data; the CMK never leaves the key store.
Why two tiers. Rotating or revoking one CMK re-controls millions of objects without re-encrypting them all — you re-wrap DEKs, not petabytes. It also means the powerful key can live somewhere much more tightly controlled than the data.
Where residency bites. The CMK's location and control is the sovereignty control. If the CMK lives in an in-jurisdiction key store you control, the operator holding the ciphertext cannot read it, so extraterritorial reach is blunted by cryptography.

BYOK, HYOK, and external key stores.

BYOK (bring your own key). You generate the key material and import it into the cloud provider's KMS; you can disable or delete it, which revokes the provider's ability to decrypt. Good, but the key still operates inside the provider's boundary.
HYOK / external key store (hold your own key). The key material never enters the provider at all — it lives in your own HSM or an external key store (e.g. AWS KMS External Key Store / GCP EKM), and every decrypt makes a call out to your key manager. This is the strongest sovereignty posture short of a fully sovereign cloud, because you can cut access instantly and unilaterally.
Access control on the key is the real control. The IAM policy on the CMK decides who can Decrypt. Sovereignty is enforced by making that policy grant decrypt only to in-jurisdiction principals — the key policy, not the storage location, is where you win or lose.

The multi-region-key foot-gun.

Multi-region KMS keys replicate the key material. Cloud "multi-region keys" are a convenience that copies the same key into several regions so ciphertext is portable. For residency that is precisely wrong: it makes the data decryptable in another jurisdiction.
Prefer per-region single-region keys. Each region gets its own CMK, created and held in that region, so EU ciphertext can only be decrypted by the EU key held under EU control.
Crypto-shredding. Because access hinges on the key, deleting the region's CMK renders all that region's ciphertext permanently unreadable — a clean, provable deletion that satisfies "right to erasure" without hunting every backup copy.

Worked example — per-region CMK wrapping a data key

Detailed explanation. The pattern is one CMK per region, each wrapping the DEKs for that region's data. Encrypting an EU row uses the EU CMK; there is no path to decrypt it with the US CMK, because they are different keys held under different control. This makes the key the enforcement point for both residency and sovereignty at once.

Question. Encrypt an EU customer row so that only the EU key can decrypt it, and show what a request to decrypt it with the US key produces.

Input.

{"customer_id": "c-eu-1", "region": "EU", "email": "ada@example.eu"}

Code.

CMK = {"EU": eu_key_store, "US": us_key_store}   # one master key per region, held in-region

def encrypt_row(row: dict) -> dict:
    region = row["region"]
    dek = generate_data_key()                     # random per-object DEK
    ciphertext = aes_encrypt(dek, row)            # encrypt the bytes
    wrapped_dek = CMK[region].wrap(dek)           # CMK never leaves the store
    return {"region": region, "wrapped_dek": wrapped_dek, "ct": ciphertext}

def decrypt_row(obj: dict, use_region: str) -> dict:
    dek = CMK[use_region].unwrap(obj["wrapped_dek"])   # fails unless region matches
    return aes_decrypt(dek, obj["ct"])

Step-by-step explanation. encrypt_row picks the CMK for the row's region, generates a fresh DEK, encrypts the bytes with the DEK, and asks the EU CMK to wrap the DEK — the CMK itself stays inside the EU key store. To read the row you must unwrap the DEK with the same CMK that wrapped it. Calling decrypt_row(obj, "US") asks the US key store to unwrap a DEK wrapped by the EU key; the US CMK is a different key and the unwrap fails, so the ciphertext is inert outside the EU.

Output.

operation	key used	result
encrypt EU row	EU CMK	wrapped DEK + ciphertext
decrypt with EU CMK	EU CMK	plaintext row
decrypt with US CMK	US CMK	unwrap fails — ciphertext stays opaque

Rule of thumb. One CMK per region, single-region, held in-jurisdiction — never a multi-region key — so the only way to read a region's data is a key that region controls.

SQL interview question on key residency and access revocation

Question. Regulators order you to make an EU dataset immediately unreadable to a specific processor, across the live table and every backup, without physically deleting terabytes. How does an in-region key design let you do this in one operation, and what access-control setting makes it enforceable?

Solution Using access-control on a region-local CMK (crypto-shred)

Code.

-- All EU data is encrypted with a DEK wrapped by the EU CMK.
-- The table (and every backup) stores only ciphertext + wrapped DEK.

-- Access to decrypt is a KEY POLICY, not a table grant:
--   grant kms:Decrypt on cmk_eu to  <in-region service principals>
--   deny  kms:Decrypt on cmk_eu to  <the ordered-out processor>

-- To make the dataset unreadable to that processor RIGHT NOW:
REVOKE DECRYPT ON KEY cmk_eu FROM ROLE processor_x;   -- single control-plane op

-- To crypto-shred the WHOLE dataset (live + all backups) irreversibly:
--   schedule deletion of cmk_eu  ->  every ciphertext ever wrapped by it
--   becomes permanently undecryptable, no matter which backup holds it.

Step-by-step trace.

step	action	scope affected	data physically deleted?
1	revoke `Decrypt` from processor_x	live + backups	no
2	processor_x reads table	ciphertext only	—
3	(erasure) schedule CMK deletion	every wrapped DEK	no
4	any future read, any copy	unwrap fails	effectively yes

Because decryption requires kms:Decrypt on the EU CMK, revoking that permission from processor_x instantly removes their ability to read — for the live table and every backup, since all copies share the same wrapping key.
The processor can still fetch bytes, but without the key those bytes are ciphertext, so the revocation is a single control-plane change with immediate, global effect.
For full erasure, scheduling deletion of the CMK is one operation whose reach is every object the key ever wrapped, wherever it is stored.
After the key is gone, no principal — not even you — can unwrap those DEKs, so the data is provably unreadable without hunting down each physical copy.

Output:

target	before revoke	after revoke	after CMK deletion
processor_x reads	plaintext	denied (ciphertext)	denied (ciphertext)
everyone reads	plaintext	plaintext	unreadable (shredded)

Why this works — concept by concept:

Key policy as access control — decryption is gated by kms:Decrypt on the CMK, so who-can-read is a key-policy decision that applies uniformly to the live table and every derived copy at once.
Envelope indirection — because every object's DEK is wrapped by the one region CMK, a single key action re-governs or destroys millions of rows without touching the rows themselves.
Crypto-shredding — deleting the region key makes all ciphertext it wrapped permanently opaque, giving a provable, backup-inclusive "right to erasure" that physical deletion sweeps can never guarantee.
In-region, single-region key — holding one CMK per region under in-jurisdiction control is what makes both the revoke and the shred lawful, unilateral, and outside a foreign operator's reach.
Cost — the operation is O(1) key actions rather than O(bytes) re-encryption or deletion, so a compliance mandate becomes a single reversible-then-irreversible control-plane step.

Access control
Topic — access-control
Key-policy and access-revocation problems

Practice →

Partitioning Topic — partitioning Per-region key and data-placement problems

Practice →

5. Cross-region aggregate-only egress

Keep the rows home and let only insight travel — spokes aggregate, a k-anonymity gate filters, the hub unions sums

The final pattern answers the question every business eventually asks: "we need a global view, but the data cannot leave its region — now what?" The answer is aggregate-only egress in a hub-and-spoke shape. Each regional spoke computes aggregates locally, a privacy gate (a minimum group size, a.k.a. k-anonymity) drops any aggregate small enough to re-identify an individual, and only the surviving, safe summaries cross to a central hub that unions them. Raw rows never move; the metadata/control plane is central; the data plane stays regional.

Aggregate-only, computed in region.

Reduce before you move. The spoke runs GROUP BY on region-local storage, so what leaves is COUNT/SUM/AVG per group — not the underlying rows. The reduction is the compliance boundary: individuals are gone by the time anything crosses.
Suppress small groups (k-anonymity). An aggregate over a tiny group can re-identify a person (a SUM(salary) for one employee is that salary). A HAVING count(*) >= k threshold suppresses groups below k so no aggregate leaks an individual. Common k values are 5–20 depending on sensitivity.
Beware differencing attacks. Two overlapping aggregates can be subtracted to reveal a suppressed cell; robust designs add consistent suppression rules (or noise / differential privacy) so the hub cannot reconstruct a small group by arithmetic.

Hub-and-spoke topology.

Spokes = regional data planes. Each region is a full, independent shard holding its raw data and doing its own aggregation — the same regional-warehouse shards from Section 3.
Hub = a thin global layer. The hub stores only the safe aggregates and unions them into global metrics. It holds no raw regional data, so the hub's own jurisdiction is not a residency problem.
The control plane is separate from the data plane. Schema, job definitions, lineage, and policies (metadata) are managed centrally and pushed to spokes; the data (rows) never flow the other way. This is the same metadata/data split as Section 3, applied to analytics.

When even aggregates are not enough — sovereign cloud.

Sovereign-cloud offerings. When regulation demands that even the operator be in-jurisdiction (in-country support staff, in-country legal entity, isolated from foreign parent control), providers offer sovereign-cloud options — for example AWS European Sovereign Cloud, Microsoft Cloud for Sovereignty, and Google Sovereign Controls — plus national efforts like Gaia-X.
Confidential computing. Encrypting data in use (enclaves / trusted execution) closes the last gap where plaintext exists in memory during processing, so even the operator's admins cannot read it.
Choose the least-heavy control that clears the bar. Region-pinning handles residency; customer-held keys handle most sovereignty; sovereign-cloud and confidential computing are for the strictest mandates. Reaching for the heaviest control everywhere is expensive and slow — match the control to the legal requirement.

Worked example — a spoke aggregate that suppresses small groups

Detailed explanation. The everyday building block is a spoke query that aggregates and applies the k-anonymity HAVING filter in the same statement, so the object that leaves the region is already both reduced and suppressed. Nothing downstream has to remember to re-apply the gate.

Question. In the EU spoke, produce per-city order counts that may leave the region, suppressing any city with fewer than 5 customers.

Input.

city	customers	orders
Berlin	900	3400
Munich	120	510
Aland (tiny)	3	7

Code.

-- Runs in the EU spoke, on EU storage. Output is egress-eligible.
SELECT
    city,
    COUNT(DISTINCT customer_id) AS customers,
    COUNT(*)                    AS orders
FROM eu.orders
GROUP BY city
HAVING COUNT(DISTINCT customer_id) >= 5;   -- k-anonymity: drop tiny groups

Step-by-step explanation. The GROUP BY city reduces raw orders to one row per city — individuals are already gone. The HAVING COUNT(DISTINCT customer_id) >= 5 clause drops any city whose group is smaller than k=5, so "Aland" with 3 customers is suppressed and never appears in the output. What remains is a small, non-identifying summary that is safe to send to the hub; the raw eu.orders rows stay in the EU.

Output.

city	customers	orders
Berlin	900	3400
Munich	120	510

Rule of thumb. Apply the k-anonymity HAVING filter in the same query that aggregates, in-region — so the artifact that crosses the boundary is provably suppressed and no downstream step can forget the gate.

SQL interview question on aggregate-only cross-region reporting

Question. You must build a global "revenue by product category" report from EU and US spokes without any raw row leaving its region, and a category with fewer than 10 customers in a region must be suppressed. Design the spoke query and the hub union.

Solution Using in-region aggregation, a k-anonymity gate, and a hub union

Code.

-- SPOKE (runs identically in EU and US, each on its own region-local storage)
CREATE TABLE {region}.category_daily AS
SELECT
    'EU'                        AS region,     -- literal per spoke
    category,
    COUNT(DISTINCT customer_id) AS customers,
    SUM(amount)                 AS revenue
FROM eu.orders
GROUP BY category
HAVING COUNT(DISTINCT customer_id) >= 10;      -- suppress < 10 customers

-- HUB (thin global layer; stores only the safe aggregates, no raw rows)
SELECT category, SUM(revenue) AS global_revenue, SUM(customers) AS customers
FROM (
    SELECT * FROM eu.category_daily            -- crossed the boundary: safe
    UNION ALL
    SELECT * FROM us.category_daily
) t
GROUP BY category;

Step-by-step trace.

step	location	rows	privacy gate
1	EU spoke	raw `orders` (PII)	stays in EU
2	EU spoke	`category_daily` (>=10)	small cats suppressed
3	US spoke	`category_daily` (>=10)	small cats suppressed
4	hub	union + re-aggregate	only safe sums

Each spoke aggregates its own raw orders in region, so the personal rows never move; only category, customers, and revenue are produced.
The HAVING ... >= 10 gate suppresses any category with fewer than 10 customers in that region, so no small-group aggregate can re-identify a person before egress.
The two suppressed summaries are the only objects that cross the replication boundary into the hub.
The hub UNION ALLs the regional summaries and re-aggregates to a global figure; it holds no raw data, so its own location is not a residency concern.

Output:

category	global_revenue	customers
electronics	512000.00	4100
grocery	288500.00	9600

Why this works — concept by concept:

Aggregate-in-region — reducing raw rows to group sums on region-local storage means individuals are eliminated before anything is eligible to move, so egress carries insight, not people.
k-anonymity gate — the HAVING count >= k filter guarantees every crossing aggregate covers enough individuals to be non-identifying, converting "aggregate" from "usually safe" into "provably suppressed."
Hub-and-spoke — spokes are independent regional shards holding the data; the hub is a thin union layer holding only safe summaries, so no single place concentrates cross-region raw rows.
Metadata/data split — the schema and job run centrally while the rows stay regional, giving one global report definition without one global data pile.
Cost — the hub moves and stores O(groups) aggregates instead of O(rows) raw data, so the compliant design is also the cheaper and faster one to run globally.

Partitioning
Topic — partitioning
In-region aggregation and layout problems

Practice →

Access control Topic — access-control Egress-policy and suppression problems

Practice →

Cheat sheet — residency recipes

Pin a bucket region and block replication.

S3 bucket lives in one region for life; deny replication out of the boundary
- create bucket in eu-central-1
- bucket policy / SCP: DENY s3:PutBucketReplication to non-EU regions
- DENY cross-region snapshot/backup copy for this bucket

Geo-partition on a region key.

CREATE TABLE events (event_id BIGINT, region STRING, payload STRING)
PARTITIONED BY (region);          -- region=EU/ , region=US/  in pinned storage

SELECT * FROM events WHERE region = 'EU';   -- prunes US partitions

Route a tenant to its regional warehouse.

def route(home_region, requested_region, warehouses):
    if requested_region != home_region:
        raise PermissionError("residency violation")
    return warehouses[requested_region]     # region-pinned account

Per-region CMK envelope encryption.

dek = generate_data_key()
ct  = aes_encrypt(dek, row)
wrapped = CMK[row["region"]].wrap(dek)      # one single-region CMK per region
dek_out = CMK[row["region"]].unwrap(wrapped)  # decrypt needs the SAME region CMK; delete it -> crypto-shred

k-anonymity aggregate for egress.

SELECT city, COUNT(DISTINCT customer_id) AS customers, SUM(amount) AS revenue
FROM eu.orders
GROUP BY city
HAVING COUNT(DISTINCT customer_id) >= 10;   -- suppress small groups, then egress

Residency decision table.

Requirement	Control
Bytes must stay in region	Region-pinned storage + `PARTITION BY region`
One region must not read another	Partition pruning + per-region warehouse routing
No raw rows across the boundary	Aggregate-in-region + object-level egress policy
Operator must not be able to read	BYOK / HYOK, in-region single-region CMK
Even the operator must be in-jurisdiction	Sovereign cloud + confidential computing
Global view without moving rows	Hub-and-spoke, aggregate-only + k-anonymity

Frequently asked questions

What is the difference between data residency and data sovereignty?

Data residency is a statement about geography — that the physical bytes of a dataset are stored and processed within a named region and its copies (replicas, backups, temp files, caches) do not leave it. Data sovereignty is a statement about jurisdiction — whose laws can compel access to the data regardless of where it sits. You can satisfy residency (store EU data in the EU) while failing sovereignty (a foreign-headquartered operator remains legally compellable), which is why sovereignty controls like in-region operators and customer-held keys exist on top of region-pinning.

How is data residency different from disaster recovery?

Disaster recovery deliberately creates copies across failure domains so a regional outage does not cause data loss or downtime; residency deliberately bounds where copies may exist so data stays in one jurisdiction. The two pull in opposite directions, and cross-region DR replication is the single most common way a residency guarantee is silently broken. The fix is to keep DR inside the residency boundary — a second in-jurisdiction zone or region — rather than backing up to whichever region is cheapest.

How do you keep data from crossing a region boundary?

Make region a partition key so each region's rows live in their own partitions pinned to region-local storage, deploy one warehouse per region, and explicitly deny cross-region replication, snapshot copy, and stream mirroring for regulated data. Route every query through a resolver keyed on the tenant's residency so no code can point an EU session at US storage. When a global view is needed, aggregate inside each region and let only non-personal summaries cross the boundary, never raw rows.

What is BYOK and why does key residency matter?

BYOK (bring your own key) means you generate the encryption key material and import it into the provider's key service, keeping the power to disable or delete it; HYOK/external key stores go further and keep the key entirely in your own HSM so the provider never holds it. Key residency matters because whoever controls the key controls the data — envelope encryption wraps each object's data key with your region-local customer master key, so ciphertext stored under a foreign operator is unreadable without a key that stays in your jurisdiction, and revoking or deleting that key crypto-shreds the data across the live table and every backup at once.

Can metadata leave the region if the data cannot?

Usually yes — schema, column types, lineage, ownership, and classification tags are metadata, not the regulated payload, so a central catalog can span regions while the rows stay in the regional data plane. The important caveat is that value-bearing statistics (min/max, sampled values, histograms) can leak actual data, so a residency-grade catalog suppresses those for regulated columns. This metadata/data-plane split is what lets you run one global governance model without concentrating regulated data in one place.

What is a sovereign cloud?

A sovereign cloud is an offering designed so that not just the data but the operation stays in-jurisdiction — in-country legal entity, in-country support staff, and isolation from foreign-parent control — for mandates where region-pinning and customer keys are not enough. Examples include AWS European Sovereign Cloud, Microsoft Cloud for Sovereignty, and Google Sovereign Controls, alongside national initiatives like Gaia-X. Combined with confidential computing (encrypting data in use inside enclaves), it closes the gap where even the operator's administrators might otherwise access plaintext.

Practice on PipeCode

Pipecode.ai is Leetcode for Data Engineering — every residency idea above, from geo-partitioning on a region key to per-region warehouse routing, in-region CMK crypto-shredding, and k-anonymity aggregate egress, maps to a hands-on practice room where you build the layout against real graded inputs. PipeCode pairs each reading with 450+ DE-focused problems and a real-time scoring engine, so your answer to "how would you keep EU rows from ever reaching US compute?" holds up under a senior interviewer's depth probes.

Practice partitioning problems now →
Region-shard drills →

GDPR vs CCPA vs India DPDP: Building Multi-Jurisdiction Privacy Pipelines

Gowtham Potureddi — Fri, 25 Sep 2026 13:44:49 +0000

multi-jurisdiction privacy pipelines are what you build when the same row of personal data must be legal in three places at once — an EU customer, a California consumer, and an Indian data principal all land in the same warehouse, and each is governed by a different statute with a different vocabulary, a different legal trigger for processing, and a different deadline for honouring a deletion request. The naive answer is to fork the platform per region. The engineering answer is to parameterise one platform by jurisdiction, so that consent, policy, subject rights, and de-identification are data-driven rather than region-specific code.

This guide is a contrast, not a single-law walkthrough. The EU's GDPR, California's CCPA (as amended by the CPRA), and India's Digital Personal Data Protection Act of 2023 agree on the goal — give individuals control over their data — and disagree on almost every mechanism: opt-in consent versus opt-out of sale, a "data subject" versus a "consumer" versus a "data principal", a one-month response window versus forty-five days. A data engineer who hard-codes GDPR everywhere ships a system that is simultaneously over-compliant in California and non-compliant in India. The four ideas an interviewer will actually probe — a unified consent and subject registry, per-jurisdiction policy tags with purpose limitation, the access and erasure pipelines, and tokenization versus pseudonymization — are the load-bearing walls of a platform that satisfies all three at once.

When you want hands-on reps immediately after reading, drill the access-control practice library →, rehearse de-identification on the tokenization practice set →, and harden your validation logic on the data-quality practice set →.

On this page

Privacy as a data-engineering problem, not a legal memo
The unified consent & subject registry
Policy tags, data mapping & purpose limitation
Subject-rights pipelines — access & erasure tombstones
Tokenization & pseudonymization
Cheat sheet — multi-jurisdiction privacy recipes
Frequently asked questions
Practice on PipeCode

1. Privacy as a data-engineering problem, not a legal memo

Three statutes, one physical platform — the differences are the specification, not trivia

The one-sentence invariant: you cannot fork the warehouse per country, so every difference between GDPR, CCPA, and DPDP must become a column, a tag, or a policy the pipeline reads at runtime. A privacy pipeline is not a lawyer's summary rendered in a wiki; it is the set of data structures and jobs that make "is this processing allowed?" and "erase this person" answerable by code. To build it you have to know precisely where the three regimes diverge.

The vocabulary maps, but does not match.

The individual. GDPR calls them a data subject; CCPA calls them a consumer (a California resident); DPDP calls them a data principal. Same human, three keys — which is exactly why you need one canonical subject_id.
The accountable party. GDPR has a controller (decides purpose and means) and a processor; CCPA has a business and a service provider; DPDP has a data fiduciary and a data processor, plus the higher-duty Significant Data Fiduciary (SDF).
The oversight role. GDPR can require a Data Protection Officer (DPO); DPDP requires an SDF to appoint a DPO based in India; CCPA has no DPO mandate but the CPRA created the California Privacy Protection Agency (CPPA) as a regulator.

The legal trigger for processing is where they diverge most.

GDPR — a lawful basis, opt-in by default. You may only process personal data if you can name one of six lawful bases (Article 6): consent, contract, legal obligation, vital interests, public task, or legitimate interests. Consent, when used, must be freely given, specific, informed, and unambiguous.
CCPA/CPRA — notice and opt-out, not consent. California does not require a lawful basis to collect. It requires notice at collection and gives the consumer a right to opt out of the sale or sharing of personal information (the "Do Not Sell or Share" link and the Global Privacy Control signal). Opt-in consent is only mandated for minors and for "sensitive personal information" limits.
DPDP — consent or a "legitimate use", plus itemised notice. India's default is consent (free, specific, informed, unconditional, unambiguous, with a clear affirmative action), backed by a plain-language notice that must be available in English and the scheduled Indian languages. A short list of "legitimate uses" (e.g. the principal voluntarily provides data for a requested service) can substitute.

The subject rights rhyme, but the deadlines and shapes differ.

Access. GDPR's Data Subject Access Request (DSAR, Article 15) → 1 month. CCPA's right to know → 45 days (extendable to 90). DPDP's right to access a summary of processing → within a period the rules set, via the fiduciary's grievance mechanism.
Erasure. GDPR's right to erasure ("right to be forgotten", Article 17) is broad. CCPA's right to delete has enumerated exceptions. DPDP ties erasure to withdrawal of consent and to the purpose being fulfilled — there is no standalone "right to be forgotten" clause, but withdrawal must be as easy as giving consent.
Correction and portability. GDPR grants rectification and portability; CPRA added a right to correct; DPDP grants correction and completion. Portability is a first-class GDPR right and absent from DPDP.

What interviewers listen for.

Do you say "I parameterise the platform by jurisdiction" rather than "I build a GDPR system"? — senior signal.
Do you distinguish opt-in consent (GDPR/DPDP) from opt-out of sale (CCPA) without prompting? — the single most common trap.
Do you treat erasure as propagation across every downstream copy, not a DELETE on one table? — the whole engineering point.
Do you know that DPDP has no portability right and CCPA needs no lawful basis, so a one-size GDPR build is wrong in two directions? — depth signal.

Worked example — the same person under three regimes

Detailed explanation. Before any code, internalise that one physical record carries three legal states simultaneously. A single user, Priya, who is an EU resident who later moved to California and holds Indian citizenship, could plausibly trigger all three regimes depending on where the data was collected and where she resides. The platform must answer "what may I do with row X?" differently per jurisdiction, from the same stored row.

Question. For one stored profile row, list what each regime requires before you may use it for marketing analytics, and what a deletion request means under each.

Input.

attribute	value
subject_id	`s_88f1`
email	`priya@example.com`
collected_in	EU (2023), then CA (2025)
purpose_requested	marketing analytics

Code.

REGIMES = {   # a jurisdiction is a policy object, not a country string in a WHERE clause
    "GDPR": {"needs": "lawful_basis", "marketing_ok_if": "consent",
             "erasure": "broad", "access_sla_days": 30},
    "CCPA": {"needs": "notice_at_collection", "marketing_ok_if": "not_opted_out",
             "erasure": "with_exceptions", "access_sla_days": 45},
    "DPDP": {"needs": "consent_or_legitimate_use", "marketing_ok_if": "consent",
             "erasure": "on_withdrawal", "access_sla_days": 30},
}

Step-by-step explanation. Under GDPR, marketing analytics on personal data needs a lawful basis; for marketing that is almost always consent, so you may proceed only if Priya opted in, and a deletion request is a broad erasure. Under CCPA, you needed no consent to collect — only notice — so marketing is allowed unless she opted out of sale/sharing, and deletion is honoured subject to statutory exceptions. Under DPDP, marketing requires consent (or a listed legitimate use) plus a valid notice id, and "deletion" is triggered when she withdraws that consent. Same row, three gates.

Output.

regime	may run marketing analytics if…	deletion means
GDPR	consent on record (opt-in)	broad erasure across copies
CCPA	consumer has not opted out	delete with enumerated exceptions
DPDP	consent + valid notice id	erase on consent withdrawal

Rule of thumb. If a single answer ("delete the user") is the same across all three regimes in your design, you have almost certainly hard-coded one law and mis-modelled the other two.

2. The unified consent & subject registry

One canonical subject_id and an append-only consent ledger are the foundation everything else stands on

The feature that makes a multi-jurisdiction platform tractable is that every regional identity resolves to one canonical subject_id, and every legality fact about that subject is an event in an append-only ledger. Without the single id you cannot honour an erasure request that spans an EU login and a California account; without the append-only ledger you cannot prove what consent existed at the moment a job ran, which is the question a regulator actually asks.

Identity resolution first.

One subject, many keys. An EU customer id, a California consumer id, and an Indian principal id may all be the same human. A resolution step (deterministic on verified email/phone, probabilistic only as a fallback) maps them to one subject_id. This id, never raw PII, is the join key across the platform.
Why not just email. Emails change, are shared, and are themselves PII. The subject_id is an opaque surrogate so that analytics can join on identity without touching a regulated attribute.

The consent ledger is event-sourced, not a mutable flag.

Per-jurisdiction, per-purpose rows. A ledger row records (subject_id, jurisdiction, purpose, basis, state, notice_id, ts, source). GDPR rows carry a lawful_basis; CCPA rows carry an opt_out state for sale/sharing; DPDP rows carry consent plus the notice_id the principal saw.
Withdrawal is a new event. You never UPDATE consent=false. You append a withdrawn event with a timestamp. The current state is a fold over history, so the ledger can answer "was consent valid at 14:03 last Tuesday?" — required for audits and for defending a past processing decision.
Versioned notices. Because DPDP (and GDPR transparency) tie validity to the exact notice text shown, the notice_id references an immutable, versioned notice record. Re-consent is required when a material purpose changes.

The gate that reads the ledger.

is_allowed(subject_id, purpose, jurisdiction) folds the ledger to a current state and returns a boolean plus a reason. Every job that touches personal data calls it — ingestion, transformation, activation.
Deny-by-default. No matching allow event means not allowed. This is what makes DPDP's "consent required" and GDPR's "lawful basis required" the same code path, while CCPA's "allowed unless opted out" is expressed as a default-allow with an opt-out event.

Worked example — folding a consent ledger to a decision

Detailed explanation. The everyday operation is: given a subject, a purpose, and a jurisdiction, fold the event history into a yes/no. The fold encodes the regime difference — GDPR/DPDP require a positive consent event, CCPA requires the absence of an opt-out event — so the same function serves all three.

Question. Given the ledger below, may you run marketing analytics for s_88f1 under GDPR, and under CCPA?

Input.

subject_id	jurisdiction	purpose	event	ts
s_88f1	GDPR	marketing	consent_granted	2026-01-10
s_88f1	GDPR	marketing	consent_withdrawn	2026-03-01
s_88f1	CCPA	sale	opt_out	2026-02-15

Code.

from collections import defaultdict

def is_allowed(events, subject_id, purpose, jurisdiction):
    # fold history -> current state; deny-by-default for opt-in regimes
    rows = [e for e in events
            if e["subject_id"] == subject_id
            and e["jurisdiction"] == jurisdiction
            and e["purpose"] in (purpose, "sale")]
    rows.sort(key=lambda e: e["ts"])            # chronological fold
    state = None
    for e in rows:
        state = e["event"]
    if jurisdiction in ("GDPR", "DPDP"):
        return state == "consent_granted"       # positive consent required
    if jurisdiction == "CCPA":
        return state != "opt_out"               # allowed unless opted out
    return False

Step-by-step explanation. For GDPR the fold sees consent_granted then consent_withdrawn; the last state is consent_withdrawn, and since GDPR needs a positive consent_granted, the answer is no. For CCPA the relevant sale events fold to opt_out, and because CCPA is allowed-unless-opted-out, state == "opt_out" returns no as well. The identical function returns the right answer for opposite legal models because the fold rule branches on jurisdiction.

Output.

jurisdiction	folded state	marketing allowed
GDPR	consent_withdrawn	no
CCPA	opt_out	no

Rule of thumb. Store consent as an immutable event stream and compute the boolean on read — a mutable consent column throws away the history that every audit and every "was this legal at the time?" question depends on.

Privacy-engineering interview question on the consent registry

Question. An interviewer gives you three source systems — an EU billing DB keyed by eu_customer_id, a California app keyed by ca_user_id, and an Indian app keyed by in_principal_id — plus a verified-contact table. Design the resolution so a single erasure request deletes the person everywhere, and show the code that produces the canonical subject_id and rejects an unverified match.

Solution Using deterministic identity resolution on verified contacts

Code.

import hashlib

def canonical_subject_id(identity, verified_contacts):
    # Only *verified* email/phone may bridge identities across regimes.
    key = None
    for contact in ("email", "phone"):
        val = identity.get(contact)
        if val and verified_contacts.get((contact, val)):   # must be verified
            key = f"{contact}:{val.lower().strip()}"
            break
    if key is None:
        # No verified bridge -> do NOT merge; mint a region-local surrogate.
        return "local:" + identity["source"] + ":" + identity["source_id"]
    return "s_" + hashlib.sha256(key.encode()).hexdigest()[:12]

Step-by-step trace.

source	source_id	verified contact	resolves to
EU billing	eu_1001	email priya@ex.com ✓	s_88f1c2a4e9b0
CA app	ca_7742	email priya@ex.com ✓	s_88f1c2a4e9b0
IN app	in_5590	phone unverified ✗	local:IN:in_5590

Each incoming identity is looked up against the verified_contacts set — a contact only bridges regimes if it was actually confirmed (double opt-in, OTP), never on a raw string match.
The EU and CA rows both carry a verified priya@ex.com, so both hash to the same subject_id; a later erasure on that id reaches both systems.
The IN row's phone is unverified, so it does not merge — merging on an unverified identifier would risk deleting or exposing the wrong person, a worse failure than an unmerged duplicate.
The hash is deterministic and one-way, so the surrogate is stable across runs yet never leaks the underlying email.

Output:

canonical subject_id	linked source records	mergeable
s_88f1c2a4e9b0	eu_1001, ca_7742	yes (verified email)
local:IN:in_5590	in_5590	no (unverified)

Why this works — concept by concept:

Deterministic resolution — hashing a verified, normalised contact yields the same subject_id on every run, so identity is reproducible and joinable without central coordination.
Verify before you merge — bridging only on confirmed contacts prevents a mis-merge that would let one person's erasure or access request hit another's data — the highest-severity privacy bug.
Surrogate over PII — the platform joins on subject_id, never on email, so analytics never touches a regulated attribute and the join key itself is not personal data.
Fail-safe default — an unverified match stays a separate region-local id; a duplicate is recoverable, a wrong merge is a breach.
Cost — resolution is O(1) per identity (a hash and a set lookup), so it scales linearly with ingest volume and adds no join-time overhead.

Access control
Topic — access-control
Consent-gate and access-control problems

Practice →

Data quality Topic — data-quality Identity-resolution and dedup problems

Practice →

3. Policy tags, data mapping & purpose limitation

Tags on the schema turn compliance into a query, not a code review

The move that scales privacy across hundreds of tables is policy-as-code tags attached to every column, so the pipeline enforces jurisdiction, purpose, and retention by reading the tag instead of by hand-written rules per table. All three regimes demand a form of this: GDPR's Article 30 requires a Record of Processing Activities (RoPA) — a living map of what data you hold, why, and for how long; DPDP requires an itemised notice inventory; CCPA requires you to catalogue categories of personal information and their purposes. The data map is that record made queryable.

Tag the columns, not the docs.

pii_class. Marks a column as direct PII (email, phone, national id), indirect/quasi-identifier (ip, device, zip), sensitive (health, biometric, financial — GDPR Article 9 special category, CPRA "sensitive PI", DPDP has no separate sensitive tier but treats children's data specially), or non_pii.
jurisdiction. Which regimes the column's rows fall under, derived from the subject registry, so a query can filter to EU-only rows when a rule is EU-specific.
purpose_allowed. The closed set of purposes this column may be used for — billing, fraud, analytics, marketing. This is the machine-readable form of purpose limitation (GDPR Article 5(1)(b)): data collected for one purpose may not be silently repurposed.
retention. A duration after which the value must be tombstoned — the operational form of GDPR's storage limitation and DPDP's "erase when the purpose is served".

The data map is a catalog you can query.

Who / what / why / where / how long. Each dataset entry answers the RoPA questions in structured fields, so "which tables hold EU sensitive data used for marketing?" is a SELECT, not a week of interviews.
Drift detection. New columns without tags fail CI. An untagged PII column is a compliance hole, so the catalog is enforced at schema-change time, the same way a data contract is.

Purpose limitation is enforced at read time.

A purpose token on every job. A job declares the purpose it runs for (purpose="fraud"); the access layer intersects that with each column's purpose_allowed and strips or blocks columns that do not match.
Repurposing requires a new basis. Using marketing-tagged data for a fraud model is not a bug you catch in review — the token mismatch blocks it, and lifting the block requires a new consent/basis record, which is exactly the legal reality.

Worked example — a purpose-limited column projection

Detailed explanation. The clearest way to see purpose limitation is a function that, given a job's purpose, returns only the columns that purpose is allowed to read. The tags live with the schema; the gate is a set intersection. This is how you stop a marketing job from quietly training on billing data.

Question. A fraud job asks to read the transactions table. Given the column tags below, which columns does it receive?

Input.

column	pii_class	purpose_allowed
amount	non_pii	billing, fraud, analytics
card_token	indirect	fraud
email	direct	billing, marketing
device_ip	indirect	fraud, analytics

Code.

def project_for_purpose(schema, purpose):
    # Return only columns whose purpose_allowed set includes this job's purpose.
    return [
        col["column"]
        for col in schema
        if purpose in col["purpose_allowed"]
    ]

visible = project_for_purpose(transactions_schema, purpose="fraud")

Step-by-step explanation. The gate walks the column tags and keeps a column only if "fraud" is in its purpose_allowed set. amount, card_token, and device_ip all list fraud, so they pass. email lists only billing and marketing, so the fraud job never sees it — even though the column physically sits in the same table. Purpose limitation becomes a projection, applied uniformly to every job.

Output.

column	visible to fraud job
amount	yes
card_token	yes
email	no
device_ip	yes

Rule of thumb. If the only thing standing between a marketing job and PII is a reviewer noticing the wrong SELECT, purpose limitation is not enforced — put the rule on the column and make the platform apply it.

Privacy-engineering interview question on retention and the data map

Question. GDPR storage limitation and DPDP's "erase when the purpose is served" both demand that data not outlive its purpose. You have thousands of tables. How do you drive retention off the data map so that expired rows are tombstoned automatically, and prove it happened for an audit?

Solution Using retention tags to schedule tombstones

Code.

from datetime import datetime, timedelta

def rows_to_tombstone(catalog, today):
    # Emit (table, column, cutoff) work items from the data map's retention tags.
    work = []
    for entry in catalog:                       # catalog == queryable RoPA
        for col in entry["columns"]:
            days = col.get("retention_days")
            if days is None:
                continue
            cutoff = today - timedelta(days=days)
            work.append({
                "table": entry["table"],
                "column": col["column"],
                "cutoff": cutoff,               # tombstone rows older than this
                "basis": "storage_limitation",
            })
    return work

Step-by-step trace.

table	column	retention_days	cutoff (today=2026-09-15)
events	device_ip	90	2026-06-17
profiles	email	730	2024-09-16
ledger	amount	(none)	skipped

The scheduler reads the data map, not the tables — retention lives with the catalog entry, so one job covers every dataset.
For each column carrying a retention_days tag it computes a cutoff date; rows whose event time predates the cutoff are due for a tombstone.
amount has no retention tag, so it is skipped — financial records often have a legal-obligation basis to be retained, which the absence of a tag encodes.
Each emitted work item names a basis (storage_limitation), so the resulting tombstone log is a self-describing audit record: what was erased, from where, and under which legal rationale.

Output:

work items emitted	audit fields per item
2 (device_ip, email)	table, column, cutoff, basis, run_ts

Why this works — concept by concept:

Data map as driver — putting retention_days on the catalog means one scheduler enforces storage limitation across thousands of tables, instead of a cron per table that drifts.
Tag absence is a decision — an untagged column is deliberately retained (legal obligation), so the model distinguishes "keep forever on purpose" from "nobody set a rule", which CI catches.
Basis on every action — stamping each tombstone with its legal basis turns the erasure log into the evidence an auditor and a DPO ask for.
Uniform across regimes — the same cutoff mechanism serves GDPR storage limitation and DPDP purpose-fulfilment because both reduce to "erase rows older than the tagged horizon".
Cost — building the work list is O(columns) over the catalog, and the actual erase is O(expired rows) per table, decoupled from total table size via a time index.

Data quality
Topic — data-quality
Data-contract and retention-validation problems

Practice →

Access control Topic — access-control Purpose-limited access and column-projection problems

Practice →

4. Subject-rights pipelines — access & erasure tombstones

One router, three SLAs — access fans out and assembles, erasure writes a tombstone that propagates

Subject rights are where a privacy platform earns its keep, and the design that satisfies all three regimes is a single request router that branches into an access pipeline and an erasure pipeline, both driven by the data map and both leaving an audit trail. The regimes differ in deadline and scope — GDPR's DSAR is one month, CCPA's requests are forty-five days, DPDP routes through the fiduciary's grievance mechanism — but the engineering is identical: find every copy of the subject, then either export it or erase it.

The router normalises the request.

One request type, per-regime SLA. A request carries (subject_id, kind, jurisdiction, received_ts). The router stamps the deadline from the jurisdiction (30 / 45 / grievance-window days) and enqueues the work. Verification of the requester happens here — a "verifiable consumer request" under CCPA, identity confirmation under GDPR/DPDP.
Deny-by-default on verification. An unverified request is never executed; responding to an impersonator is itself a breach.

Access: fan out over the data map, assemble a package.

The data map is the index of where to look. Because every dataset holding PII is catalogued and keyed by subject_id, "find everything about this person" is a fan-out over the map, not tribal knowledge of which tables matter.
Assemble, redact, deliver. Results are collected into an export bundle; other subjects' data that co-occurs (a shared thread, a joint transaction) is redacted. GDPR portability additionally requires a structured, machine-readable format — a shape DPDP does not mandate and CCPA satisfies with the "right to know".

Erasure: tombstone, then propagate.

A tombstone is a durable erasure marker, not a DELETE. Writing {subject_id, erased_at, scope} to a tombstone table means the fact of erasure survives even as it propagates asynchronously to warehouses, lakes, backups, and search indices. A raw DELETE on the primary leaves stale copies downstream that no one tracks.
Propagation to every copy. The tombstone is a change-data event every downstream consumer subscribes to; each applies the erasure to its own store and acknowledges. Erasure is complete only when every registered sink has acked — which is why the data map must also list sinks.
Idempotent and re-runnable. Applying the same tombstone twice is a no-op, so a retried or replayed erasure is safe; this is the property that lets you re-drive a failed propagation without fear.

Worked example — an erasure tombstone with downstream propagation

Detailed explanation. The canonical erasure is not a delete statement; it is an event that fans out. You write one tombstone, and every downstream store consumes it and confirms. The pipeline is done when all sinks ack, and the tombstone table is the proof.

Question. A verified GDPR erasure arrives for s_88f1. Show how the tombstone drives deletion across the primary DB, the warehouse, and a backup, and how you know it is complete.

Input.

sink	keyed by subject_id	ack status before
primary_db	yes	—
warehouse	yes	—
backup	yes (restore-time apply)	—

Code.

def erase(subject_id, sinks, tombstones):
    # 1) Durable marker first, 2) fan out, 3) require all acks (idempotent).
    tombstones.append({"subject_id": subject_id,
                       "erased_at": now(), "scope": "all", "acks": set()})
    for sink in sinks:
        sink.apply_tombstone(subject_id)        # each is a no-op if already applied
        tombstones[-1]["acks"].add(sink.name)
    required = {s.name for s in sinks}
    complete = required.issubset(tombstones[-1]["acks"])
    return complete

Step-by-step trace.

step	action	sink	acks so far
1	write tombstone	—	{}
2	apply + ack	primary_db	{primary_db}
3	apply + ack	warehouse	{primary_db, warehouse}
4	apply + ack	backup	{primary_db, warehouse, backup}

The tombstone is written before any deletion, so if the process crashes mid-propagation the erasure is not lost — a restart re-drives it from the durable marker.
Each sink applies the erasure and acks; because apply_tombstone is idempotent, a re-driven step that already ran simply re-acks without error.
Completion is required.issubset(acks) — every registered sink must confirm, including the backup, which applies the tombstone at restore time so a recovered snapshot never resurrects the person.
The tombstone row, with its ack set and timestamp, is the audit artifact proving the erasure met the regime's deadline.

Output:

subject_id	sinks acked	complete
s_88f1	primary_db, warehouse, backup	yes

Rule of thumb. Erasure is finished when the last downstream copy acks the tombstone, not when the primary DELETE returns — model the sinks explicitly or you will leave the person in a lake or a backup.

Privacy-engineering interview question on cross-regime request routing

Question. Your inbox receives access and deletion requests for the same subject_id across GDPR, CCPA, and DPDP, each with its own deadline. Design a router that assigns the correct SLA, refuses unverified requests, and lets an erasure be retried safely if a downstream sink was offline.

Solution Using an idempotent router keyed on request id

Code.

SLA_DAYS = {"GDPR": 30, "CCPA": 45, "DPDP": 30}

def route(request, verified, done_ids):
    # Idempotent: a replayed request_id is skipped; unverified is refused.
    if request["request_id"] in done_ids:
        return {"status": "already_processed"}
    if not verified.get(request["request_id"]):
        return {"status": "refused", "reason": "unverified requester"}
    deadline = add_days(request["received_ts"], SLA_DAYS[request["jurisdiction"]])
    handler = access_pipeline if request["kind"] == "access" else erasure_pipeline
    result = handler(request["subject_id"], jurisdiction=request["jurisdiction"])
    done_ids.add(request["request_id"])         # mark handled -> safe to retry
    return {"status": "completed", "deadline": deadline, "detail": result}

Step-by-step trace.

request_id	jurisdiction	kind	verified	outcome
r1	GDPR	erasure	yes	completed, deadline +30d
r1	GDPR	erasure	yes	already_processed (replay)
r2	CCPA	access	no	refused (unverified)
r3	DPDP	erasure	yes	completed, deadline +30d

r1 is verified and new, so the router stamps a 30-day GDPR deadline and calls the erasure pipeline, then records r1 as done.
A replay of r1 (a retried queue message) short-circuits on the done_ids check, so the erasure is never applied twice — the router itself is idempotent even before the sinks are.
r2 is a CCPA access request but the requester failed verification, so it is refused; responding would have leaked data to an impersonator.
r3 is a DPDP erasure, verified, and gets the DPDP window — the same router serves three regimes by looking the SLA up from a table.

Output:

request_id	status	deadline
r1	completed / then already_processed	+30d
r2	refused	—
r3	completed	+30d

Why this works — concept by concept:

Single router, table-driven SLA — the deadline is a lookup, so adding a fourth regime is a config row, not a new pipeline; the branch on kind chooses access vs erasure.
Idempotent on request_id — a done_ids guard makes replays and retries no-ops, so an at-least-once queue never double-erases or double-exports.
Verify-then-act — refusing unverified requests is the router's most important job; a fast, wrong response to an impersonator is worse than a slow, correct one.
Deadline as a first-class field — stamping the SLA on completion produces the timeliness evidence each regulator expects.
Cost — routing is O(1) per request; the real work is the O(copies) fan-out inside the handler, bounded by the data map's sink list.

Access control
Topic — access-control
Subject-request routing and verification problems

Practice →

Tokenization Topic — tokenization Tombstone-propagation and erasure problems

Practice →

5. Tokenization & pseudonymization

Keep raw PII in a vault, land tokens in the warehouse, and make erasure a single key delete

The technique that makes every previous section cheaper is de-identification: replace raw PII with tokens so the warehouse holds no direct identifiers, and use per-subject keys so erasure becomes deleting one key rather than chasing rows across every store. GDPR explicitly rewards pseudonymization (Article 4(5) and Article 32) as a security measure; CPRA and DPDP both treat de-identified data as lower-risk. Getting the vocabulary exact is itself an interview filter.

Three terms interviewers make you separate.

Tokenization — reversible, vault-backed. The real value (a card number, an email) is stored in a secured vault and replaced everywhere else by an opaque token. Given the vault and authorisation, you can re-identify. The warehouse never holds the raw value.
Pseudonymization — reversible only with separate info (GDPR Art 4(5)). The data can be attributed to a person only by using additional information kept separately and under controls. Tokenization is one implementation; the legal point is that the mapping is held apart and protected. Pseudonymised data is still personal data.
Anonymization — irreversible. No key, no vault, no path back to the person. Truly anonymised data falls outside GDPR/DPDP entirely — but true anonymisation is hard, because quasi-identifiers can re-identify, so most "anonymised" data is really pseudonymised.

Crypto-shredding makes erasure O(1).

A key per subject. Each subject's PII is encrypted with a subject-specific key; the ciphertext can live anywhere, even in backups. To erase the person you delete their key, and every ciphertext copy — warehouse, lake, backup, index — becomes permanently unreadable at once.
Why this beats chasing rows. The propagation problem from Section 4 shrinks: you still tombstone the identity, but the content is neutralised the instant the key dies, so a missed backup copy is inert rather than a breach.

Deterministic vs randomized tokens — the analytics trade-off.

Deterministic tokens join. The same input always maps to the same token, so you can GROUP BY token and join across tables without re-identifying. The cost is that deterministic tokens are vulnerable to frequency analysis, so they suit low-cardinality-safe joins, not secrets.
Randomized tokens do not join. A fresh token per occurrence is safest but breaks joins, so it fits values you never aggregate on (a free-text note, a one-off id). Choosing per column is a data-modelling decision the tag from Section 3 can carry.

Worked example — tokenizing on ingest and re-identifying by purpose

Detailed explanation. The everyday pattern is: on ingest, swap each PII field for a deterministic token and stash the real value in the vault; downstream jobs work on tokens; a re-identify call, gated by purpose, is the only path back. This keeps the warehouse free of raw PII while preserving joins.

Question. Tokenize an incoming record's email, load only the token to the warehouse, and show what a fraud-purpose re-identify returns versus a marketing one when email is not marketing-allowed.

Input.

{"subject_id": "s_88f1", "email": "priya@example.com", "amount": 42.5}

Code.

import hmac, hashlib

VAULT = {}  # token -> raw value, secured & access-controlled in reality

def tokenize(value, key=b"subject-key-s_88f1"):
    tok = "tok_" + hmac.new(key, value.encode(), hashlib.sha256).hexdigest()[:12]
    VAULT[tok] = value                          # raw stays in the vault only
    return tok

def reidentify(token, purpose, allowed_purposes):
    if purpose not in allowed_purposes.get(token, set()):
        raise PermissionError("purpose not permitted for this field")
    return VAULT[token]

row = {"subject_id": "s_88f1", "email": tokenize("priya@example.com"), "amount": 42.5}

Step-by-step explanation. tokenize computes a deterministic HMAC token and files the raw email in the vault; the row written to the warehouse carries email = tok_..., never the address. A fraud job that is permitted to re-identify the email gets the raw value back; a marketing job, for which email is not an allowed purpose, hits the PermissionError and only ever sees the token. Erasure later deletes subject-key-s_88f1, after which the token can no longer be regenerated or matched.

Output.

caller purpose	re-identify email	result
fraud	allowed	`priya@example.com`
marketing	not allowed	PermissionError (token only)

Rule of thumb. If raw PII reaches the warehouse "just in case", you have inverted the model — tokenize at the edge and make re-identification the rare, purpose-gated exception.

Privacy-engineering interview question on erasure at scale

Question. You hold a subject's data across a warehouse, a data lake, and nightly backups. A GDPR erasure must guarantee the data is unrecoverable, but rewriting every backup is infeasible. How do you make erasure both complete and cheap?

Solution Using crypto-shredding with per-subject keys

Code.

KEYS = {}   # subject_id -> encryption key (the only thing that must truly die)

def store_pii(subject_id, value):
    key = KEYS.setdefault(subject_id, generate_key())
    return encrypt(value, key)                  # ciphertext may live anywhere

def crypto_shred(subject_id):
    # Erasure = destroy the key; all ciphertext copies become unreadable.
    KEYS.pop(subject_id, None)

def read_pii(subject_id, ciphertext):
    key = KEYS.get(subject_id)
    if key is None:
        return None                             # shredded -> permanently opaque
    return decrypt(ciphertext, key)

Step-by-step trace.

step	action	key present	read result
1	store_pii(s_88f1, email)	yes	ciphertext in warehouse, lake, backup
2	read_pii before shred	yes	`priya@example.com`
3	crypto_shred(s_88f1)	no	—
4	read_pii after shred (any copy)	no	None (unrecoverable)

Each subject's PII is encrypted with a per-subject key, so the ciphertext is safe to replicate to the lake and to backups.
Before erasure, an authorised read decrypts normally because the key exists.
crypto_shred deletes only the key — a single, cheap operation — instead of hunting and rewriting every ciphertext copy.
After shredding, every copy, including untouched backups, decrypts to None; the data is unrecoverable without ever rewriting a backup file.

Output:

after crypto_shred	warehouse	lake	backup
decryptable	no	no	no

Why this works — concept by concept:

Key as the unit of erasure — destroying one per-subject key neutralises unbounded ciphertext copies at once, so erasure cost is O(1) rather than O(copies).
Backups without a rewrite — because a stale backup holds only ciphertext, a shredded key makes that backup inert without touching it — the only practical way to honour erasure against immutable backups.
Pseudonymization by construction — the warehouse holds ciphertext/tokens, so it is pseudonymised data under GDPR Art 4(5), reducing breach exposure even before any request.
Tombstone still needed for identity — crypto-shred kills the content; you still write the Section 4 tombstone so the fact of erasure and the identity linkage are removed and audited.
Cost — shredding is O(1) (one key delete); the only ongoing cost is per-read decryption, an acceptable trade for making erasure and replication cheap.

Tokenization
Topic — tokenization
Tokenization, vaulting and crypto-shred problems

Practice →

Data quality Topic — data-quality De-identification and re-identification-risk problems

Practice →

Cheat sheet — multi-jurisdiction privacy recipes

Regime field mapping.

REGIME = {
    "GDPR": {"subject": "data subject", "party": "controller",
             "trigger": "lawful basis (Art 6)", "sale_optout": False,
             "erasure": "right to be forgotten (Art 17)", "access_sla": 30},
    "CCPA": {"subject": "consumer", "party": "business",
             "trigger": "notice at collection", "sale_optout": True,
             "erasure": "right to delete (exceptions)", "access_sla": 45},
    "DPDP": {"subject": "data principal", "party": "data fiduciary",
             "trigger": "consent / legitimate use", "sale_optout": False,
             "erasure": "on withdrawal", "access_sla": 30},
}

Consent-check gate (deny-by-default for opt-in regimes).

def allowed(state, jurisdiction):
    if jurisdiction in ("GDPR", "DPDP"):
        return state == "consent_granted"
    if jurisdiction == "CCPA":
        return state != "opt_out"
    return False

Policy-tag column contract.

column = {"name": "email", "pii_class": "direct",
          "jurisdiction": ["GDPR", "CCPA", "DPDP"],
          "purpose_allowed": ["billing"], "retention_days": 730}

Erasure tombstone.

tombstone = {"subject_id": "s_88f1", "erased_at": now(),
             "scope": "all", "acks": set()}   # complete when all sinks ack

Crypto-shred (O(1) erasure).

def crypto_shred(subject_id, keys):
    keys.pop(subject_id, None)     # every ciphertext copy goes dark at once

Purpose-limited read.

visible = [c["name"] for c in schema if purpose in c["purpose_allowed"]]

Regime comparison at a glance.

dimension	GDPR (EU)	CCPA/CPRA (California)	DPDP (India)
individual	data subject	consumer	data principal
processing trigger	lawful basis	notice + opt-out	consent / legitimate use
access deadline	1 month	45 days	rules-set window
erasure	broad (Art 17)	delete with exceptions	on withdrawal

Frequently asked questions

What is a multi-jurisdiction privacy pipeline?

It is a data platform designed so the same personal data can be processed legally under several privacy laws at once — typically EU GDPR, California CCPA/CPRA, and India's DPDP Act 2023 — without forking the infrastructure per region. The differences between the laws are pushed into data: a consent ledger, per-column policy tags, jurisdiction fields, and subject-rights pipelines that read those structures at runtime. The goal is one physical platform whose behaviour is parameterised by jurisdiction rather than hard-coded to one statute.

How do GDPR, CCPA, and DPDP differ for a data engineer?

The biggest engineering difference is the legal trigger for processing. GDPR requires a lawful basis (often opt-in consent) before you process; CCPA requires no consent but a notice at collection plus an opt-out of sale or sharing; DPDP requires consent (or a listed legitimate use) plus an itemised notice. They also differ on vocabulary (data subject vs consumer vs data principal), response deadlines (1 month vs 45 days vs a rules-set window), and rights (GDPR has portability, DPDP does not; CCPA has no lawful-basis concept). A design that assumes opt-in everywhere is wrong for California, and one that assumes opt-out is wrong for the EU and India.

What is the difference between tokenization and pseudonymization?

Tokenization is a concrete technique: replace a raw value with an opaque token and keep the real value in a secured vault, so the token is reversible only through that vault. Pseudonymization is the legal concept (GDPR Article 4(5)): data that can be attributed to a person only with additional information kept separately and protected — tokenization is one way to achieve it. Both are reversible and both leave the data as personal data; only true anonymization, which removes any path back to the individual, takes data outside GDPR and DPDP.

How do you engineer the right to erasure across downstream copies?

Do not model erasure as a DELETE on one table. Write a durable tombstone marker for the subject, then propagate it as an event to every downstream sink — warehouse, lake, search index, backups — and treat the erasure as complete only when every registered sink acknowledges. To handle backups you cannot rewrite, combine the tombstone with crypto-shredding: encrypt each subject's data with a per-subject key and delete the key, which renders every ciphertext copy, including old backups, permanently unreadable in one cheap operation.

What is purpose limitation and how do you enforce it in a pipeline?

Purpose limitation (GDPR Article 5(1)(b), echoed by DPDP) means data collected for one purpose may not be silently reused for another. You enforce it by tagging each column with the closed set of purposes it may serve, and by having every job declare the purpose it runs for; the access layer intersects the two and strips or blocks columns whose purpose does not match. Repurposing then requires a new consent or basis record rather than a code change a reviewer might miss, which mirrors the legal reality.

Do I need separate warehouses per jurisdiction?

Usually no, and forking per region is the anti-pattern this whole design avoids. A single platform with a canonical subject_id, a consent ledger, jurisdiction tags, and rights pipelines handles all three regimes and stays maintainable. You may still need data-residency controls — DPDP can restrict transfers to blacklisted countries and some GDPR flows require EU storage — but residency is a placement policy on tagged data, not a reason to duplicate the entire stack and its logic three times.

Practice on PipeCode

Pipecode.ai is Leetcode for Data Engineering — every idea above, from the consent-ledger fold and the purpose-limited projection to the erasure tombstone and crypto-shred, maps to a hands-on practice room where you build the pipeline against real graded inputs. PipeCode pairs each reading with 450+ DE-focused problems and a real-time scoring engine, so your answer to "how would you make this erasure complete across every copy?" holds up under a senior interviewer's depth probes.

Practice access-control problems now →
Tokenization drills →

SOC 2 for Data Engineering: Controls, Audit Logs & Evidence-Collection Pipelines

Gowtham Potureddi — Fri, 25 Sep 2026 13:37:58 +0000

soc 2 for data engineering is the moment an abstract compliance framework turns into concrete work on your warehouse: who can read the PII column, how a terminated employee's grants get revoked, whether last Tuesday's pipeline deploy went through review, and how you prove all of it to an auditor without a week of screenshots. SOC 2 is not a product you install and not a certification you pass once — it is an independent attestation, written by a licensed CPA firm, that the controls you claim to run actually exist and actually operated over a window of time. For a data engineer, that window is measured against tables, grants, and query logs you already own.

The confusion that trips people up is treating SOC 2 as a security-team problem that lands in a slide deck. It is not. The controls that get sampled hardest — least-privilege access to the warehouse, an audit trail of every read against sensitive data, change management on the pipelines that move that data, and evidence that survives tampering — are all data-engineering surfaces. This guide walks through the four things an auditor (and an interviewer) will actually probe: the access controls you provision and deprovision, the warehouse audit logs that prove them, the change-management chain behind every pipeline deploy, and the evidence-collection pipelines that gather it all automatically. Each section pairs the concept with a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the access-control practice library →, rehearse the monitoring and alerting side on the SLA-monitoring practice set →, and harden the correctness of your evidence tables on the data-quality practice set →.

On this page

Why SOC 2 lands on the data engineer's desk in 2026
Access controls data engineers own
Warehouse audit logs & access history
Change management for pipelines
Evidence-collection pipelines
Cheat sheet — SOC 2 evidence recipes
Frequently asked questions
Practice on PipeCode

1. Why SOC 2 lands on the data engineer's desk in 2026

SOC 2 is an attestation against the Trust Services Criteria — not a checklist you buy, and half the sampled controls live in your warehouse

The one-sentence invariant: SOC 2 is a CPA firm's opinion that the controls you described are suitably designed and — for a Type II — actually operated over a period, and the data platform is where most of those controls physically live. There is no "SOC 2 software" that makes you compliant; there is a set of controls you run, and an audit that samples evidence they ran. Understanding that shape is what separates a data engineer who can speak to auditors from one who forwards every question to the security team.

What SOC 2 actually is.

An attestation, not a certification. The output is a report signed by a licensed CPA firm following AICPA attestation standards, expressing an opinion on your controls. You do not "get certified"; you receive a report you can share with customers under NDA.
Scoped to a system. The report covers a defined system — for a data platform that usually means the warehouse, the pipelines, the orchestration, the access model, and the cloud infrastructure underneath. Anything out of scope is not covered.
Driven by your own control descriptions. You write the controls (the "system description"); the auditor tests them. That is why vague controls hurt you — you will be tested against exactly what you claimed.

The five Trust Services Criteria (TSC).

Security (the Common Criteria) — mandatory. Every SOC 2 includes it: logical access, change management, risk assessment, monitoring. This is where most data-engineering controls sit.
Availability — optional. Uptime, backup, disaster recovery, capacity — relevant if you promise SLAs on data delivery.
Processing Integrity — optional. Data is processed completely, accurately, and on time — a natural fit for pipeline correctness and data-quality checks.
Confidentiality — optional. Information designated confidential is protected — encryption, access restriction, retention/disposal.
Privacy — optional. Personal information is collected, used, retained, and disposed of per your notice — the strictest, and often deferred.

Type I vs Type II — the distinction interviewers love.

Type I is a point in time. The auditor opines that controls are suitably designed as of a single date. It answers "do the right controls exist?" — a snapshot.
Type II is a period. The auditor opines that controls operated effectively across a window, usually 3 to 12 months. It answers "did the controls actually run, every time, the whole period?" — and it is tested by sampling evidence across that window.
Why it matters to a DE. Type II is the one customers ask for, and it is the one that forces you to retain evidence continuously. A control that worked once but has no audit trail across the period fails a Type II even if it was perfectly designed.

The controls a data engineer actually owns.

Logical access — provisioning, least privilege, periodic access reviews, deprovisioning.
Audit logging — warehouse query and access history proving who read and changed what.
Change management — version control, peer review, and approvals on pipeline code.
Encryption — at rest and in transit, usually inherited from the cloud warehouse but attested by you.
Monitoring — pipeline failure alerting, anomaly detection, and control-monitoring dashboards.
Evidence — the pipelines that gather all of the above into a form an auditor can sample.

What interviewers listen for.

Do you say "SOC 2 is an attestation against the Trust Services Criteria," not "a certification you pass"? — senior signal.
Do you name Security as the mandatory Common Criteria and the other four as scope-dependent? — required framing.
Do you distinguish Type II as operating-effectiveness over a period from Type I's point-in-time design? — the classic discriminator.
Do you frame your job as "owning the access, logging, change-management, and evidence controls," not "helping the security team"? — ownership signal.

Worked example — mapping one control to its evidence

Detailed explanation. The single most useful habit for SOC 2 is to stop thinking in vague policies and start thinking in control → evidence pairs. Every control you claim must produce an artifact an auditor can pull. Modeling that mapping as data — a small control catalog — is the mindset that makes the rest of this guide click, because each later section is just one row of this catalog turned into a real query.

Question. Express three data-platform controls as a catalog that pairs each control with the concrete evidence query or artifact that proves it operated.

Input.

control_id	control	trust criterion
CC6.1	Least-privilege access to the warehouse	Security
CC6.2	Access removed on termination	Security
CC8.1	Changes are peer-reviewed and approved	Security

Code.

control_catalog = {   # each control maps to the evidence that proves it ran
    "CC6.1": {
        "control": "Least-privilege access to the warehouse",
        "evidence": "SELECT grantee_name, role, privilege FROM account_usage.grants_to_users",
        "cadence": "quarterly access review",
    },
    "CC6.2": {
        "control": "Access removed on termination",
        "evidence": "grants anti-joined against the active-employee roster (0 orphans)",
        "cadence": "daily deprovisioning check",
    },
    "CC8.1": {
        "control": "Changes are peer-reviewed and approved",
        "evidence": "deployments LEFT JOIN approved_prs (0 unapproved prod deploys)",
        "cadence": "per deploy",
    },
}

for cid, c in control_catalog.items():
    print(cid, "->", c["evidence"])

Step-by-step explanation. Each key is a control identifier borrowed from the Security Common Criteria numbering (CC6.x is logical access, CC8.x is change management). Each value pins the evidence — an actual query or artifact — and a cadence that says how often the evidence is produced. The auditor does not accept "we use least privilege"; they accept the output of the grants query, sampled on the review date. Turning every control into a runnable evidence statement is exactly what an evidence-collection pipeline automates.

Output.

control_id	evidence the auditor samples
CC6.1	grant listing from `account_usage.grants_to_users`
CC6.2	anti-join showing 0 orphaned grants
CC8.1	join showing every prod deploy has an approved PR

Rule of thumb. If you cannot name the query or artifact that proves a control, you do not have a SOC 2 control — you have a policy. Every control must resolve to evidence you can pull on demand.

2. Access controls data engineers own

Provisioning, least privilege, and deprovisioning are the CC6 controls auditors sample first — and the warehouse is where they live

The Security criterion's logical-access controls (the CC6 family) are the ones every SOC 2 tests, and for a data team they resolve to warehouse grants. Say the loop in one breath: access is provisioned from an authoritative source, granted at least privilege through roles, reviewed periodically, and revoked the moment someone leaves. Break any link and you have a finding.

Provisioning from an authoritative source.

Joiner-mover-leaver. Access should be driven by the HR system or an identity provider, not by ad-hoc Slack requests. A joiner gets a role, a mover's role changes, a leaver's access is revoked — each event tied to a record.
Request and approval trail. Every grant should trace to an approved request (a ticket, an IdP group assignment) so the auditor can see why someone has access, not just that they do.

Least privilege through roles, not direct grants.

RBAC, never user grants. In Snowflake, BigQuery, or Redshift, privileges attach to roles and roles attach to users. Granting SELECT directly to a user is un-auditable and un-revocable at scale; granting a role is both.
Separate read, write, and admin. A read_raw role, a read_pii role gated behind a masking policy, and a loader role that can write are three different privileges. Nobody gets the union "just in case."
Masking and row-access policies. Column masking and row-access policies let you grant a role table access while still restricting the sensitive columns or rows — least privilege at the cell level, and itself an attestable control.

Periodic access reviews.

Quarterly is the common cadence. A reviewer confirms each grant is still needed. The evidence is the review record plus the grant snapshot it reviewed.
The review must be actionable. "Looks fine" is not evidence; a diff showing what changed since last quarter, and any revocations that resulted, is.

Deprovisioning — the control that fails Type II most often.

The gap is the finding. The dangerous window is between a termination date and the moment access is actually removed. Auditors sample terminated employees and check the revocation timestamp against the leave date.
Automate it or lose it. Manual deprovisioning drifts; an automated daily check that anti-joins live grants against the active roster catches the orphan before the auditor does.

Worked example — a least-privilege role hierarchy in Snowflake

Detailed explanation. The clearest demonstration of least privilege is a role hierarchy where table privileges live on functional roles and people inherit only what they need. No human holds a direct table grant; a person is granted a role, and the role holds the privilege. That indirection is what makes access reviewable and revocable in one statement.

Question. Grant an analyst read access to a customers table but keep the email column masked, using roles rather than direct user grants.

Input. A customers table with an email PII column, and a user ada who should read it masked.

Code.

-- 1. Functional role holds the privilege, not the user.
CREATE ROLE IF NOT EXISTS read_customers;
GRANT USAGE ON DATABASE analytics TO ROLE read_customers;
GRANT USAGE ON SCHEMA analytics.core TO ROLE read_customers;
GRANT SELECT ON TABLE analytics.core.customers TO ROLE read_customers;

-- 2. Column masking policy — email is only unmasked for a privileged role.
CREATE MASKING POLICY mask_email AS (val STRING) RETURNS STRING ->
  CASE WHEN CURRENT_ROLE() IN ('PII_ADMIN') THEN val
       ELSE REGEXP_REPLACE(val, '.+@', '****@') END;
ALTER TABLE analytics.core.customers
  MODIFY COLUMN email SET MASKING POLICY mask_email;

-- 3. The person inherits the role — never a direct table grant.
GRANT ROLE read_customers TO USER ada;

Step-by-step explanation. Step 1 puts every privilege on read_customers, so the grant is auditable as one row and revocable with one REVOKE ROLE. Step 2 attaches a masking policy to the email column: any role except PII_ADMIN sees ****@domain, so the table grant does not leak the PII. Step 3 grants the role to ada; she never receives a direct table privilege, so a reviewer sees exactly one line — "ada has read_customers" — instead of hunting through per-object grants.

Output.

principal	can read customers	sees raw email?
`ada` (via `read_customers`)	yes	no — masked `****@`
`PII_ADMIN`	yes	yes
everyone else	no grant	n/a

Rule of thumb. If revoking a person's access takes more than one REVOKE ROLE, your access model is not least-privilege — privileges belong on roles, and people belong in roles.

SQL interview question on deprovisioning

Question. Your auditor samples five terminated employees and asks you to prove none of them retained warehouse access after their leave date. Write a query that lists any active grant whose grantee is no longer an active employee — the orphaned-access anti-join.

Solution Using an anti-join against the active-employee roster

Code.

-- grants_snapshot: current warehouse grants (grantee_name, role, granted_on)
-- hr_roster: employees with status and termination_date
SELECT
    g.grantee_name,
    g.role,
    r.termination_date,
    DATEDIFF('day', r.termination_date, CURRENT_DATE) AS days_orphaned
FROM grants_snapshot AS g
LEFT JOIN hr_roster AS r
       ON UPPER(g.grantee_name) = UPPER(r.warehouse_user)
WHERE r.warehouse_user IS NULL          -- grant with no matching employee at all
   OR r.status = 'terminated'           -- or a terminated employee still granted
ORDER BY days_orphaned DESC;

Step-by-step trace.

grantee	in roster?	status	verdict
ADA	yes	active	not returned (compliant)
LINUS	yes	terminated 2026-02-10	returned — orphaned 33 days
SVC_ETL	yes	active (service)	not returned
GHOST	no match	—	returned — grant with no owner

Every current grant is the driving side of a LEFT JOIN onto the HR roster keyed on the warehouse username.
WHERE r.warehouse_user IS NULL catches grants whose grantee is not in the roster at all — the un-owned accounts auditors hate.
r.status = 'terminated' catches grantees who are in the roster but have left — the deprovisioning gap.
days_orphaned quantifies exposure so the worst offenders sort to the top and the remediation is prioritized.

Output:

grantee_name	role	termination_date	days_orphaned
LINUS	read_customers	2026-02-10	33
GHOST	loader	(none)	(unknown owner)

Why this works — concept by concept:

Anti-join — a LEFT JOIN plus WHERE right IS NULL (widened here to include terminated rows) returns exactly the grants that should not exist; it is the canonical "find what is missing from the allowed set" pattern.
Authoritative roster — the HR/IdP roster is the source of truth for "who is an employee," so access correctness is defined against it, not against a hand-maintained list.
Deprovisioning gap — days_orphaned turns a boolean finding into a measured risk window, which is exactly the metric a Type II auditor tests against the leave date.
Service accounts — un-matched grantees surface non-human accounts that need a named owner, closing the "who owns GHOST?" question before it becomes a finding.
Cost — O(grants) with a hash join on the roster; a nightly run over a few thousand grants is milliseconds and produces continuous deprovisioning evidence.

Access
Topic — access-control
Least-privilege and access-review problems

Practice →

Quality Topic — data-validation Roster-reconciliation and validation problems

Practice →

3. Warehouse audit logs & access history

Query and access history are the evidence that access controls actually held — logging is what turns a policy into proof

An access policy you cannot observe is a policy you cannot attest. The Security criterion's monitoring controls demand that you can answer "who read this, who changed that, when, and from where" for the whole audit period. Modern warehouses hand you this for free through account-usage views — the trick is knowing which view proves which control, and respecting their latency and retention. Say it plainly: audit logs are the operating-effectiveness evidence; without them a well-designed control still fails a Type II.

The log surfaces you actually cite.

LOGIN_HISTORY. Every authentication attempt, success or failure, with client IP and method. Proves the authentication and MFA-enforcement controls and surfaces brute-force patterns.
QUERY_HISTORY. Every statement run, by whom, against what, with duration and bytes scanned. The backbone of "who ran what" evidence.
ACCESS_HISTORY. The strong one for confidentiality: per-query, the exact objects and columns read and written. This is what proves who touched a PII column, not just who queried the table.
GRANTS_TO_USERS / GRANTS_TO_ROLES. Point-in-time snapshots of the access model itself — the evidence behind the access-review control from section 2.

Column-level access is the confidentiality proof.

Table-level is not enough. "User X selected from customers" does not prove whether they read the masked email. ACCESS_HISTORY records the columns actually referenced, so you can prove column-level confidentiality.
Masking and row-access policies leave a trail too. Applying and altering a policy is itself a change auditors can see, tying the confidentiality control back to change management.

Latency and retention — the trap.

Account-usage views lag. Snowflake's ACCOUNT_USAGE views can be delayed (often up to ~2–3 hours for access history), so a real-time control cannot depend on them; an evidence-collection job must account for the lag.
Retention is finite. ACCOUNT_USAGE retains history for a bounded period (on the order of a year for many views, less for some). A Type II window can exceed retention, so you must export logs to durable storage or you will have gaps exactly when the auditor samples an old date.
Other platforms, same idea. BigQuery exposes Cloud Audit Logs (Admin Activity and Data Access) and INFORMATION_SCHEMA.JOBS; Redshift has STL/SVL system tables and CloudTrail. The pattern is identical: system views prove access, and you export them before they age out.

Worked example — reading who queried a sensitive table

Detailed explanation. The everyday audit query answers "show me every access to this table in the review window." It is the first thing you run when an auditor points at a sensitive object, and the shape — filter QUERY_HISTORY by object and time — is the foundation the harder column-level query builds on.

Question. List every user who queried analytics.core.customers in the last 90 days, with how many times and when they last touched it.

Input. QUERY_HISTORY rows over the last 90 days referencing various tables.

Code.

SELECT
    user_name,
    COUNT(*)              AS query_count,
    MAX(start_time)       AS last_accessed
FROM snowflake.account_usage.query_history
WHERE start_time >= DATEADD('day', -90, CURRENT_TIMESTAMP)
  AND query_text ILIKE '%analytics.core.customers%'
GROUP BY user_name
ORDER BY query_count DESC;

Step-by-step explanation. The WHERE clause bounds the window to the 90-day review period and filters to statements referencing the target table. GROUP BY user_name collapses each principal to one row; COUNT(*) is their access frequency and MAX(start_time) is the recency an auditor asks for. Filtering on query_text is the quick approximation — the precise, column-aware version uses ACCESS_HISTORY, which the interview question below demands.

Output.

user_name	query_count	last_accessed
ADA	42	2026-09-12 08:15
BI_SERVICE	310	2026-09-14 23:59
LINUS	3	2026-02-09 17:40

Rule of thumb. QUERY_HISTORY proves that a table was queried; when the question is which columns were read, you must move to ACCESS_HISTORY — text matching cannot prove column-level confidentiality.

SQL interview question on column-level access history

Question. An auditor wants proof of exactly which principals read the email PII column of customers in the last 90 days. Table-level history is not enough. Write the query using Snowflake ACCESS_HISTORY.

Solution Using LATERAL FLATTEN over ACCESS_HISTORY

Code.

SELECT
    ah.user_name,
    cols.value:columnName::STRING AS column_read,
    COUNT(*)                      AS reads,
    MAX(ah.query_start_time)      AS last_read
FROM snowflake.account_usage.access_history AS ah,
     LATERAL FLATTEN(input => ah.base_objects_accessed) AS obj,
     LATERAL FLATTEN(input => obj.value:columns)        AS cols
WHERE ah.query_start_time >= DATEADD('day', -90, CURRENT_TIMESTAMP)
  AND obj.value:objectName::STRING = 'ANALYTICS.CORE.CUSTOMERS'
  AND cols.value:columnName::STRING = 'EMAIL'
GROUP BY ah.user_name, column_read
ORDER BY reads DESC;

Step-by-step trace.

access_history row	object	columns array	matches email?
q1 by ADA	CUSTOMERS	[id, name, email]	yes — 1 read
q2 by ADA	CUSTOMERS	[id, name]	no
q3 by BI_SERVICE	CUSTOMERS	[id, email]	yes — 1 read
q4 by LINUS	ORDERS	[order_id]	no — wrong object

ACCESS_HISTORY.base_objects_accessed is a semi-structured array of the base objects each query touched; the first FLATTEN yields one row per object.
Each object carries a nested columns array; the second FLATTEN yields one row per column actually referenced — this is the column-level granularity table-level history cannot give.
The WHERE filters to the target object and the EMAIL column, so only genuine reads of the PII column survive.
Grouping by user and column produces per-principal read counts and recency — precisely the confidentiality evidence the auditor asked for.

Output:

user_name	column_read	reads	last_read
BI_SERVICE	EMAIL	118	2026-09-14 23:59
ADA	EMAIL	7	2026-09-11 10:02

Why this works — concept by concept:

base_objects_accessed — ACCESS_HISTORY records the resolved base objects and columns per query, so it sees through views and SELECT * to the real columns read — the only reliable source for column-level proof.
LATERAL FLATTEN — flattening the nested object and column arrays turns one query row into one row per column, letting a normal GROUP BY count column-level access.
Column-level confidentiality — the result names the exact principals who read email, which is what the Confidentiality criterion requires and what table-level history can only approximate.
Export before retention — because ACCESS_HISTORY ages out, this query must run inside an evidence job that persists results, or the proof vanishes before the audit.
Cost — O(rows × columns-per-query) over the flattened set; scanning a bounded 90-day window keeps it cheap, and pushing the object filter down limits the scan.

Access
Topic — access-control
Column-level access and audit-trail problems

Practice →

Audit Topic — event-log Event-log and access-history query problems

Practice →

4. Change management for pipelines

Every pipeline change must trace to a reviewed, approved, controlled deploy — CC8 is the control auditors reconstruct from your git and CI history

The Security criterion's change-management control (CC8.1) asks a simple question with expensive consequences: can you prove that every change to production went through review and approval? For a data team, "production" is the pipeline code, the transformations, and the schema. The auditor reconstructs the control from version control and CI, so the control is only as good as the trail those systems leave. Say it in one line: an approved, peer-reviewed pull request is the unit of controlled change, and a prod deploy without one is the finding.

The change-control chain.

Version control is the system of record. All pipeline and transformation code lives in git; nothing reaches production except through a merge. A change with no commit is an un-auditable change.
Peer review is the control. A pull request reviewed and approved by someone other than the author is the evidence. The approval, the reviewer identity, and the timestamp are the artifact.
CI/CD gates the deploy. The deployment pipeline should refuse to ship a branch that was not merged through an approved PR, and it should record what it shipped — commit SHA, PR number, actor, time.

Segregation of duties.

Author is not approver. The person who wrote the change cannot be the one who approves it — the review is meaningless otherwise, and auditors check for self-approval.
Approver is not necessarily deployer, and deploys are automated. Manual, un-logged production access is the anti-pattern; an automated deploy from an approved merge removes the human who could bypass the control.
Break-glass is logged. Emergency changes happen; the control is not "never," it is "every emergency change is logged, justified, and reviewed after the fact."

The evidence is a join.

A deployments table. Each production deploy records deploy_id, commit_sha, pr_number, deployed_by, deployed_at. This is your change log.
An approvals table. Each merged PR records pr_number, author, approved_by, approved_at. Pulled from the git host's API.
The control test is the reconciliation. Left-join deploys to approvals; any deploy with no approved PR, or where approver equals author, is a change-control exception — the exact thing a Type II samples.

Worked example — recording a deploy with its approval lineage

Detailed explanation. The control produces evidence only if the deploy step writes it down. The cleanest pattern is a deploy job that, at ship time, records the commit, the PR it came from, and who triggered it into a deployments table. That single insert is what a whole quarter of change-control evidence is built on.

Question. At deploy time, capture the change-control lineage of a pipeline release into a deployments table so it can later be reconciled against approvals.

Input. A release of commit 9f3a1c merged via PR 412, deployed by the CI service account.

Code.

import hashlib, datetime as dt

def record_deploy(conn, commit_sha, pr_number, actor):
    row = {
        "deploy_id": hashlib.sha1(f"{commit_sha}{pr_number}".encode()).hexdigest()[:12],
        "commit_sha": commit_sha,
        "pr_number": pr_number,
        "deployed_by": actor,
        "deployed_at": dt.datetime.utcnow().isoformat(),
        "environment": "prod",
    }
    conn.execute(
        """INSERT INTO change_control.deployments
           (deploy_id, commit_sha, pr_number, deployed_by, deployed_at, environment)
           VALUES (%(deploy_id)s, %(commit_sha)s, %(pr_number)s,
                   %(deployed_by)s, %(deployed_at)s, %(environment)s)""",
        row,
    )
    return row

record_deploy(conn, commit_sha="9f3a1c", pr_number=412, actor="ci-deployer")  # CI-only, never a human shell

Step-by-step explanation. The deploy job — not a human — calls record_deploy as its final step, so the evidence is a byproduct of shipping, impossible to forget. It captures the commit_sha and the pr_number that carried it, plus the acting identity and a UTC timestamp. Because CI is the only path to prod, the deployments table becomes a complete census of production changes, and the pr_number is the foreign key that lets the auditor's reconciliation join to the approvals table.

Output.

deploy_id	commit_sha	pr_number	deployed_by	environment
3d9e1a77c0b2	9f3a1c	412	ci-deployer	prod

Rule of thumb. If a deploy can happen without writing a row to the deployments table, your change-control evidence has holes — make the evidence write a mandatory step of the deploy, not a manual afterthought.

SQL interview question on change-control exceptions

Question. Prove that every production deploy in the audit window came from a peer-approved PR, and that no change was self-approved. Write the query that returns the change-control exceptions.

Solution Using an anti-join with a segregation-of-duties check

Code.

SELECT
    d.deploy_id,
    d.pr_number,
    d.deployed_by,
    a.author,
    a.approved_by,
    CASE
        WHEN a.pr_number IS NULL              THEN 'no approved PR'
        WHEN a.approved_by = a.author         THEN 'self-approved'
    END AS exception_type
FROM change_control.deployments AS d
LEFT JOIN change_control.approved_prs AS a
       ON d.pr_number = a.pr_number
WHERE d.environment = 'prod'
  AND d.deployed_at >= DATE '2026-01-01'
  AND (a.pr_number IS NULL OR a.approved_by = a.author)
ORDER BY d.deployed_at;

Step-by-step trace.

deploy	pr_number	approved?	author=approver?	verdict
dep_1	412	yes	no	compliant — not returned
dep_2	419	none in approvals	—	returned — no approved PR
dep_3	421	yes	ada = ada	returned — self-approved
dep_4	430	yes	no	compliant — not returned

Every prod deploy in the window is the driving side of a LEFT JOIN onto approved PRs, keyed on pr_number.
a.pr_number IS NULL catches deploys whose PR was never approved (or never existed) — an uncontrolled change.
a.approved_by = a.author catches the segregation-of-duties violation where the author approved their own change.
Only exceptions survive the WHERE; an empty result set is the passing evidence, and any rows are the exact deploys the auditor will drill into.

Output:

deploy_id	pr_number	exception_type
dep_2	419	no approved PR
dep_3	421	self-approved

Why this works — concept by concept:

Change-control chain — modeling deploys and approvals as two joinable tables turns a fuzzy policy ("we review changes") into a testable reconciliation with a boolean outcome per deploy.
Anti-join for missing approvals — the LEFT JOIN ... IS NULL pattern isolates deploys with no approval, the same shape as the deprovisioning check, reused for a different control.
Segregation of duties — comparing approved_by to author encodes the "author is not approver" rule directly in SQL, so self-approval cannot hide.
Empty-set-is-passing — a control whose passing state is "zero exceptions" is easy to monitor continuously and easy to alert on the moment a violation appears.
Cost — O(deploys) with a hash join on pr_number; trivial to run per deploy as a gate and nightly as evidence.

Reliability
Topic — reliability
Change-control and deployment-safety problems

Practice →

SLA Topic — sla-monitoring Monitoring and control-status problems

Practice →

5. Evidence-collection pipelines

Automated evidence is what makes a Type II survivable — a pipeline that snapshots controls into an immutable, tamper-evident ledger

A Type II audit does not sample your controls on one lucky day; it samples them across months, and it expects the evidence to have existed at the time, not to be reconstructed afterward. Collecting that evidence by hand — screenshots, spreadsheets, quarterly scrambles — is where audits go to die. The senior move is to treat evidence like any other pipeline output: scheduled jobs snapshot each control's evidence into an append-only, tamper-evident store, so the audit is a query, not a project.

What an evidence-collection pipeline gathers.

Access snapshots. Nightly captures of GRANTS_TO_USERS / GRANTS_TO_ROLES so every point-in-time access state exists for the whole window, not just today.
Access-review artifacts. The quarterly review's input snapshot, the reviewer, and the resulting revocations, stored together.
Log exports. ACCESS_HISTORY, QUERY_HISTORY, and LOGIN_HISTORY exported before they age out of retention.
Control-test results. The anti-join outputs from sections 2–4 (orphaned grants, PII reads, change-control exceptions), each run and stored with its timestamp so "the control ran and passed on this date" is itself evidence.

Immutability — what makes evidence trustworthy.

Append-only, never update. Evidence tables are insert-only. An evidence store you can UPDATE or DELETE is one an auditor cannot trust; the control is only credible if the record cannot be quietly changed.
Hash-chaining for tamper-evidence. Each row stores a hash of its own contents plus the previous row's hash. Altering any historical row breaks the chain from that point forward, so tampering is detectable without trusting the database alone.
WORM / Object Lock storage. Exported evidence lands in write-once-read-many object storage (S3 Object Lock in compliance mode, GCS retention lock) so even an administrator cannot delete it before its retention expires.

Continuous control monitoring.

A control that reports its own status. Instead of proving controls at audit time, each control emits a PASS/FAIL with its evidence on every run, feeding a monitoring dashboard.
Alert on transition, not on state. The signal that matters is a control flipping from PASS to FAIL — an orphaned grant appearing, a self-approved deploy landing — which routes to on-call the same day, shrinking the exposure window the auditor would otherwise measure.

Worked example — an append-only evidence snapshot

Detailed explanation. The building block of an evidence pipeline is a job that runs a control's evidence query and appends the result with a run timestamp — never overwriting the previous run. Keeping every run is what gives you a point-in-time record for any date the auditor picks.

Question. Snapshot today's grant listing into an append-only evidence_grants table so that every past day's access state is preserved for the audit window.

Input. The current output of the grants query, captured on run date 2026-09-15.

Code.

import datetime as dt

def snapshot_grants(conn):
    run_ts = dt.datetime.utcnow().isoformat()
    conn.execute(
        """INSERT INTO evidence.evidence_grants
           (captured_at, grantee_name, role, privilege)
           SELECT %(ts)s, grantee_name, role, privilege
           FROM   snowflake.account_usage.grants_to_users
           WHERE  deleted_on IS NULL""",     # only currently-live grants
        {"ts": run_ts},
    )
    # NOTE: no UPDATE / DELETE anywhere — the table is insert-only.

Step-by-step explanation. The job stamps every captured row with a single captured_at for the run, then inserts the live grant set as of that moment. Because the table is only ever inserted into, run N+1 never disturbs run N — the history of "who had what access on any given day" accumulates. Filtering deleted_on IS NULL captures the live state; the accumulation of daily snapshots is what lets the auditor ask "show me access on 2026-06-30" months later.

Output.

captured_at	grantee_name	role	privilege
2026-09-15T00:00	ADA	read_customers	SELECT
2026-09-15T00:00	BI_SERVICE	read_raw	SELECT
2026-09-14T00:00	ADA	read_customers	SELECT

Rule of thumb. Evidence tables grow forever and change never — the day you find yourself wanting to UPDATE an evidence row is the day your evidence stops being evidence.

Python interview question on tamper-evident evidence

Question. An auditor asks how you would prove your evidence ledger has not been altered after the fact. Design an append-only evidence table where any tampering with a historical row is detectable, and show how the check works.

Solution Using a hash-chained immutable ledger

Code.

import hashlib, json

def row_hash(prev_hash: str, payload: dict) -> str:
    body = prev_hash + json.dumps(payload, sort_keys=True)
    return hashlib.sha256(body.encode()).hexdigest()

def append_evidence(ledger: list[dict], payload: dict) -> dict:
    prev = ledger[-1]["row_hash"] if ledger else "GENESIS"
    entry = {"seq": len(ledger), "payload": payload,
             "prev_hash": prev, "row_hash": row_hash(prev, payload)}
    ledger.append(entry)                 # append-only
    return entry

def verify(ledger: list[dict]) -> int | None:
    prev = "GENESIS"
    for e in ledger:
        if e["prev_hash"] != prev or e["row_hash"] != row_hash(prev, e["payload"]):
            return e["seq"]              # first tampered sequence, or None if intact
        prev = e["row_hash"]
    return None

Step-by-step trace.

seq	payload	prev_hash	row_hash	chain ok?
0	grants@day1	GENESIS	h0	yes
1	grants@day2	h0	h1	yes
2	grants@day3 (tampered)	h1	h2' ≠ recompute	breaks here
3	grants@day4	h2 (stale)	h3	fails — prev mismatch

Each entry's row_hash is sha256(prev_hash + payload), so every row cryptographically depends on the entire history before it.
append_evidence only ever appends and always chains onto the last row's hash — there is no code path that rewrites a prior entry.
verify recomputes each hash from the previous one; if an attacker edits day3's payload, its recomputed hash no longer matches the stored row_hash, and every subsequent link fails too.
verify returns the first broken seq, pinpointing exactly where the ledger was tampered — evidence that the evidence itself is intact.

Output:

ledger state	verify() returns	meaning
untouched	`None`	chain intact — evidence trustworthy
row 2 edited	`2`	tamper detected at seq 2

Why this works — concept by concept:

Append-only ledger — insert-only semantics remove the "quiet update" attack entirely; combined with WORM storage, even a privileged operator cannot rewrite history.
Hash chaining — binding each row to the previous hash makes the ledger tamper-evident: changing any row invalidates every row after it, so a single stored final hash attests the whole chain.
Deterministic serialization — json.dumps(..., sort_keys=True) guarantees the same payload always hashes identically, so a legitimate re-verification never produces a false tamper alarm.
Continuous verification — running verify on a schedule turns tamper-evidence into tamper-detection, alerting the day the chain breaks rather than at audit time.
Cost — append is O(1) and verification is O(n) over the ledger; a nightly full verify of a year of daily snapshots is a few hundred hashes, negligible against the audit assurance it buys.

SLA
Topic — sla-monitoring
Control-monitoring and alerting problems

Practice →

Quality Topic — data-quality Immutable-ledger and integrity-check problems

Practice →

Cheat sheet — SOC 2 evidence recipes

List all warehouse grants (access evidence).

SELECT grantee_name, role, privilege, granted_on
FROM   snowflake.account_usage.grants_to_users
WHERE  deleted_on IS NULL
ORDER BY grantee_name;

Quarterly access review (diff against last snapshot).

SELECT grantee_name, role
FROM   evidence.evidence_grants
WHERE  captured_at = (SELECT MAX(captured_at) FROM evidence.evidence_grants)
EXCEPT
SELECT grantee_name, role
FROM   evidence.evidence_grants
WHERE  captured_at = :previous_review_snapshot;   -- rows added since last review

Column-level PII access (confidentiality evidence).

SELECT ah.user_name, c.value:columnName::STRING AS col, COUNT(*) reads
FROM   snowflake.account_usage.access_history ah,
       LATERAL FLATTEN(input => ah.base_objects_accessed) o,
       LATERAL FLATTEN(input => o.value:columns) c
WHERE  o.value:objectName::STRING = 'ANALYTICS.CORE.CUSTOMERS'
  AND  c.value:columnName::STRING = 'EMAIL'
GROUP BY 1, 2;

Change-control exceptions (CC8 evidence).

SELECT d.deploy_id, d.pr_number
FROM   change_control.deployments d
LEFT JOIN change_control.approved_prs a ON d.pr_number = a.pr_number
WHERE  d.environment = 'prod'
  AND (a.pr_number IS NULL OR a.approved_by = a.author);   -- empty = passing

Immutable evidence table (insert-only, Object Lock target).

CREATE TABLE evidence.evidence_grants (
  captured_at   TIMESTAMP_NTZ,
  grantee_name  STRING,
  role          STRING,
  privilege     STRING
);
-- Grant only INSERT + SELECT to the evidence role; never UPDATE/DELETE.
GRANT INSERT, SELECT ON evidence.evidence_grants TO ROLE evidence_writer;

Trust criterion -> control -> evidence.

Trust criterion	Control a DE owns	Evidence artifact
Security (CC6)	Least privilege + deprovisioning	grants snapshot + orphan anti-join
Security (CC7)	Monitoring / audit logging	`ACCESS_HISTORY` column-read export
Security (CC8)	Change management	deploy ⋈ approved-PR reconciliation
Confidentiality	PII access restriction	masked columns + column-access log
Processing Integrity	Pipeline correctness	data-quality check results, stored

Frequently asked questions

What is SOC 2 and which parts do data engineers own?

SOC 2 is an independent attestation, performed by a licensed CPA firm against the AICPA Trust Services Criteria, that an organization's controls are suitably designed and — in a Type II — operated effectively over a period. It is a report you share with customers, not a certification you pass once. Data engineers own the controls that live in the data platform: logical access (least privilege, provisioning, deprovisioning), audit logging, change management on pipeline code, encryption of data at rest and in transit, and the evidence-collection that proves all of them.

What is the difference between SOC 2 Type I and Type II?

A Type I report opines that controls are suitably designed as of a single point in time — a snapshot that answers "do the right controls exist?" A Type II report opines that those controls operated effectively across a window, typically 3 to 12 months, and is tested by sampling evidence throughout that period. Type II is the one customers usually require, and it is the reason you must retain evidence continuously rather than assembling it at audit time.

What are the five Trust Services Criteria?

They are Security, Availability, Processing Integrity, Confidentiality, and Privacy. Security — the Common Criteria — is mandatory in every SOC 2 and covers logical access, change management, risk assessment, and monitoring. The other four are included only if they are in scope for what you promise customers: Availability for uptime and recovery, Processing Integrity for complete and accurate processing, Confidentiality for protecting designated confidential data, and Privacy for handling personal information per your notice.

Which warehouse audit logs prove access controls?

In Snowflake, LOGIN_HISTORY proves authentication, QUERY_HISTORY proves who ran what, and ACCESS_HISTORY proves the exact objects and columns each query read or wrote — the last one is what evidences column-level confidentiality. GRANTS_TO_USERS and GRANTS_TO_ROLES snapshot the access model itself for access reviews. BigQuery exposes the same idea through Cloud Audit Logs and INFORMATION_SCHEMA.JOBS, and Redshift through its STL/SVL system tables plus CloudTrail; in every case you export the logs before they age out of retention.

How do you automate SOC 2 evidence collection?

You build scheduled pipelines that snapshot each control's evidence into an append-only store: nightly grant snapshots, log exports before retention lapses, and the control-test queries (orphaned grants, PII reads, change-control exceptions) run and stored with a timestamp. Each run records a PASS/FAIL plus the underlying evidence, feeding a control-monitoring dashboard that alerts the moment a control flips to failing. The audit then becomes a query over the evidence store rather than a quarterly screenshot scramble.

What makes an audit table "immutable" for SOC 2?

Three properties. It is append-only — the evidence role has INSERT and SELECT but never UPDATE or DELETE, so records cannot be quietly changed. It is tamper-evident through hash-chaining, where each row hashes its contents plus the previous row's hash, so altering any historical row breaks the chain and is detectable. And it is written to write-once-read-many storage (S3 Object Lock, GCS retention lock) so even an administrator cannot delete it before its retention period expires.

Practice on PipeCode

Pipecode.ai is Leetcode for Data Engineering — every SOC 2 control above, from the least-privilege role hierarchy and the orphaned-grant anti-join to the column-level access-history query and the hash-chained evidence ledger, maps to a hands-on practice room where you write the query against real graded inputs. PipeCode pairs each reading with 450+ DE-focused problems and a real-time scoring engine, so your answer to "how would you prove this control operated over the whole audit period?" holds up under a senior interviewer's depth probes.

Practice access-control problems now →
SLA-monitoring drills →

Marimo: The Reactive Python Notebook for Reproducible Data Work

Gowtham Potureddi — Fri, 25 Sep 2026 13:32:39 +0000

marimo reactive python notebook is an open-source Python notebook that fixes the one thing every data person has silently tolerated for a decade: a notebook whose displayed output does not match the code you can see. In a classic notebook you can run cell 5, delete cell 3, edit cell 1, and never rerun the cells in between — so the variables in memory are the residue of a run history nobody recorded. Marimo makes that class of bug impossible by treating your notebook as a dataflow graph: it reads which variables each cell defines and references, wires the cells into a directed acyclic graph, and when you change one cell it automatically reruns every cell that depends on it. There is no stale state to reason about, because there is no state that the code on screen cannot reproduce.

The second idea is just as consequential and follows from the first: a Marimo notebook is stored as a plain .py file, not a JSON blob. That means it diffs cleanly in git, imports like any module, runs as a script with python notebook.py, and serves as an interactive web app with marimo run — the same file, three faces. This guide walks the four things an interviewer will actually probe — the reactive dataflow DAG, the pure-Python reproducibility model, UI elements bound to variables, and SQL cells backed by DuckDB — and pairs each with a Solution-Tail interview answer: code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the data-analysis practice library →, rehearse frame-shaping on the dataframe-basics practice set →, and design the end-to-end flow on the pipelines practice set →.

On this page

Why Marimo changes the Python notebook in 2026
The reactive dataflow DAG
Pure-Python notebooks & reproducibility
Interactive UI elements bound to variables
SQL cells & DuckDB
Cheat sheet — Marimo recipes
Frequently asked questions
Practice on PipeCode

1. Why Marimo changes the Python notebook in 2026

Marimo is reactive, not top-to-bottom — that one fact removes hidden state and makes a notebook reproducible

The one-sentence invariant: Marimo derives the execution order from your code's dependencies, not from the order you happened to click cells, so the notebook you see is always the notebook that ran. Everything that makes Marimo attractive to a data team follows from that. There is no "run all from the top and pray" ritual, no execution_count that reads [47] next to [2], no variable that survives from a cell you already deleted. The graph is the source of truth, and the graph is recomputed from the code every time.

The problem Marimo is fixing — hidden state.

Out-of-order execution. In a classic notebook, cells can run in any order and the interpreter keeps whatever state that produced. The displayed outputs can reflect code that no longer exists on screen.
Stale variables. Delete the cell that defined df, and df lingers in memory. Every downstream cell keeps "working" against a ghost until you restart the kernel and discover it was broken all along.
The reproducibility tax. Because none of this is safe, the community habit is "Restart & Run All" before trusting a result — a manual discipline that people forget exactly when it matters.

How Marimo removes it.

Static analysis builds a DAG. Marimo parses each cell to see which global names it defines and which it references, then draws an edge from definer to referencer. That graph is a DAG (cycles are rejected).
Change one cell, dependents rerun. Edit a cell and Marimo automatically reruns its downstream cells; delete a cell and Marimo removes its variables and invalidates the cells that used them. State on screen and state in memory can never drift.
Deterministic order. Execution follows a topological sort of the DAG, so two people (or CI) running the same notebook get the same order and the same result.

What interviewers listen for.

Do you say "Marimo is reactive — the DAG decides execution order" in the first sentence? — senior signal.
Do you frame the core win as "no hidden state, so the notebook is reproducible by construction"? — required framing.
Do you mention that a notebook is a pure .py file, so it diffs and imports? — the practical hook.
Do you know that a UI element becomes reactive just by reading its .value in another cell, with no callbacks? — the "you actually used it" tell.

Worked example — the bug Marimo makes impossible

Detailed explanation. The clearest way to feel the difference is to picture the one sequence that silently corrupts a Jupyter notebook and watch Marimo refuse it. You define x, define y = x + 1, then go back and edit x. In Jupyter, y is now stale until you remember to rerun it. In Marimo, editing x reruns y for you — there is no window in which the two disagree.

Question. Two cells: cell A holds x = 1, cell B holds y = x + 1 and displays y. You change cell A to x = 10. What does y show in Jupyter versus Marimo?

Input.

cell	code
A	`x = 10` (edited from `1`)
B	`y = x + 1`

Code.

x = 10          # cell A

y = x + 1       # cell B (a separate cell)
y

Step-by-step explanation. In a classic notebook, editing cell A does nothing to cell B until you manually rerun B; y keeps showing 2 while x is already 10. In Marimo, the moment cell A is edited, the runtime sees that cell B references x, marks B as a descendant of A, and reruns B — so y becomes 11 with no action from you. The displayed value can never lag the code because the runtime, not the human, owns the order.

Output.

environment	value of `y` after editing `x` to 10	stale?
Classic notebook	2 (until you rerun B)	yes
Marimo	11 (reran automatically)	no

Rule of thumb. If editing one cell can leave another cell's output wrong until you remember to rerun it, you are carrying hidden state — Marimo removes the "until you remember" entirely.

2. The reactive dataflow DAG

Cells are functions, references are edges — the DAG is derived from your code, and one variable lives in exactly one cell

Marimo has one central idea you must be able to explain: the notebook is a graph whose nodes are cells and whose edges are variable dependencies, and Marimo builds it by static analysis, never by running your code speculatively. An interviewer who asks "how does Marimo know what to rerun?" wants this graph, the single-definition rule that keeps it a clean DAG, and the minimal-recompute behaviour that falls out of it.

How the graph is built.

Defs and refs. Each cell is analysed for the global names it defines (assignments, def, import, class) and the names it references. A cell that references df depends on whichever cell defines df.
Edges point from producer to consumer. If cell 1 defines df and cell 3 uses df, there is an edge 1 → 3. Cell 3 is a descendant of cell 1.
It must be acyclic. Two cells cannot mutually depend on each other's variables; Marimo rejects cycles because a cycle has no valid execution order.

The single-definition rule.

One variable, one cell. A given global name may be defined in exactly one cell. Redefining df in a second cell is an error, not a silent overwrite — this is what guarantees the graph is well-defined.
Underscore for locals. A name prefixed with _ (e.g. _tmp) is cell-local and invisible to the graph, so you can reuse throwaway names freely without touching the DAG.
Why the rule exists. If two cells could both define df, "which one wins" would depend on run order — exactly the hidden state Marimo abolishes. Forbidding it keeps the DAG unambiguous.

What the runtime does with the DAG.

Minimal recompute. When a cell changes, Marimo reruns that cell and only its transitive descendants — not the whole notebook. Unrelated branches are left alone.
Delete = invalidate. Deleting a cell removes its variables from memory and reruns the cells that referenced them, so nothing keeps working against a ghost variable.
Topological execution. On a fresh run, cells execute in dependency order regardless of their visual position on the page, which is why you can arrange cells however reads best.

Worked example — one edit, only the dependents rerun

Detailed explanation. The payoff of the DAG is surgical recomputation. Build a tiny four-cell notebook where a raw frame feeds a filter, the filter feeds a chart, and a wholly unrelated cell computes a constant. Change the raw frame and watch exactly two cells rerun while the unrelated cell sits still.

Question. Given cells that define raw, clean (depends on raw), chart (depends on clean), and note (independent), which cells rerun when you edit raw?

Input.

cell	defines	references
1	`raw`	—
2	`clean`	`raw`
3	`chart`	`clean`
4	`note`	—

Code.

raw = load_orders()                      # cell 1

clean = raw[raw["amount"] > 0]           # cell 2

chart = clean.plot(x="day", y="amount")  # cell 3

note = "dashboard v2"                    # cell 4 (independent of the chain)

Step-by-step explanation. Marimo reads the four cells and draws edges 1 → 2 (both touch raw), 2 → 3 (both touch clean), and leaves cell 4 with no edges. When you edit cell 1, the runtime walks the descendants of node 1: cell 2 reruns because it references raw, then cell 3 reruns because it references clean. Cell 4 never runs because note depends on nothing that changed.

Output.

edited cell	cells that rerun	cells left untouched
1 (`raw`)	2 (`clean`), 3 (`chart`)	4 (`note`)

Rule of thumb. Marimo reruns the transitive descendants of what you changed — nothing upstream, nothing on a sibling branch — so an expensive independent cell is never recomputed by an unrelated edit.

Marimo interview question on the dependency graph

Question. An interviewer shows you a notebook where cell 1 defines df, cell 2 also tries to define df = df.dropna(), and cell 3 reads df. They ask why Marimo rejects this and how you would rewrite it so the intent (drop nulls, then use the clean frame) works reactively. Show the fix.

Solution Using distinct variables to keep the DAG acyclic

Code.

df = load_orders()                    # cell 1 — the raw frame, defined once

df_clean = df.dropna()                # cell 2 — new name (single-definition rule satisfied)

result = df_clean.groupby("customer")["amount"].sum()   # cell 3 — consume the cleaned frame
result

Step-by-step trace.

cell	attempted defs	valid under single-definition rule?	edge added
1	`df`	yes	—
2 (original)	`df` again	no — redefinition error	rejected
2 (fixed)	`df_clean` (refs `df`)	yes	1 → 2
3	`result` (refs `df_clean`)	yes	2 → 3

The original cell 2 redefines df, which Marimo flags as a multiple-definition error — two cells owning one name would make execution order ambiguous.
Renaming the cleaned frame to df_clean gives each name a single owner, so the graph stays a valid DAG.
Now editing cell 1 reruns cell 2 (recomputes df_clean) and then cell 3 (recomputes result) automatically.
Because df and df_clean are distinct nodes, you can inspect both the raw and cleaned frames at any time without one clobbering the other.

Output:

step	frame	shape
cell 1	`df`	1000 rows (with nulls)
cell 2	`df_clean`	960 rows (nulls dropped)
cell 3	`result`	one sum per customer

Why this works — concept by concept:

Single-definition rule — one global name per cell makes the DAG unambiguous; Marimo can always answer "which cell owns df?" with exactly one node.
Distinct names as nodes — turning the mutation into a new variable df_clean adds a node and an edge instead of a hidden overwrite, so the transformation becomes a visible, rerunnable step.
Acyclic guarantee — because no cell redefines an upstream name, the graph has a topological order and every edit has a well-defined set of cells to rerun.
Reactive propagation — a change to df flows deterministically to df_clean and then result, with the runtime, not the analyst, choosing the order.
Cost — building the graph is O(cells × names) of cheap static parsing; a rerun touches only O(descendants), not the whole notebook.

Analysis
Topic — data-analysis
Exploratory data-analysis notebook problems

Practice →

DataFrames Topic — dataframe-basics DataFrame define-and-transform problems

Practice →

3. Pure-Python notebooks & reproducibility

The notebook is a `.py` file, so it diffs, imports, and runs three ways — reproducibility is a property of the format, not a discipline

The feature that sells Marimo to an engineer is that a notebook is stored as an ordinary Python module, not a JSON document. A .ipynb file interleaves source, base64 outputs, and execution metadata into JSON that produces unreadable diffs and merge conflicts; a Marimo file is code you can read, review, and run without a kernel. Reproducibility stops being a habit you enforce and becomes a property of the artifact.

What the file actually is.

A valid module. The notebook is a .py file defining an app = marimo.App() with each cell as a small decorated function that returns its defined names. It executes as plain Python.
Clean git diffs. Because it is code, a one-line change is a one-line diff — no serialized output blobs, no execution_count churn, no outputs: [] noise. Code review works.
No hidden outputs on disk. Cell outputs are not baked into the file, so you cannot accidentally commit a stale chart or a leaked credential printed three runs ago.

Deterministic execution, guaranteed.

Order from the DAG, not the page. On any run — yours, a colleague's, CI's — cells execute in topological order of the dependency graph, so the result does not depend on click history.
No out-of-order bugs. The one class of notebook bug that "works on my machine" comes from run order; Marimo eliminates it because run order is derived, not remembered.
Reproducible by construction. Given the same inputs and the same file, the same outputs follow — which is exactly what a data pipeline or a graded assignment needs.

Three run modes from one file.

marimo edit notebook.py opens the reactive editor for development.
marimo run notebook.py serves it as a read-only interactive web app — the code is hidden and only the UI and outputs show, so a notebook becomes a dashboard with no rewrite.
python notebook.py runs it as a script (great for a cron job or an Airflow task); because it is importable, you can also from notebook import result and unit-test cells with pytest.

Worked example — the same notebook as a script

Detailed explanation. The most convincing demonstration of the pure-Python format is running a notebook headless. Because the file is a real module with a __main__ guard, python notebook.py executes every cell in dependency order and any top-level output (or a mo.cli_output) is produced without ever opening a browser — the notebook and the batch job are the same code.

Question. You have analysis.py, authored in Marimo, that loads data and computes a summary. Show that it runs unchanged as a batch script and yields the same summary as the editor.

Input. A notebook whose last cell defines summary from an upstream orders frame.

Code.

import marimo    # file: analysis.py — authored in Marimo, stored as pure Python
app = marimo.App()

@app.cell
def _():
    import polars as pl
    orders = pl.read_parquet("orders.parquet")
    return (orders, pl)

@app.cell
def _(orders):
    summary = orders.group_by("customer").agg(total=("amount", "sum"))
    return (summary,)

if __name__ == "__main__":
    app.run()

Step-by-step explanation. The file is legal Python: importing it or running it triggers app.run(), which executes the cells in DAG order — the orders cell first because the summary cell references orders. Nothing in the file depends on a browser or a saved kernel, so the batch run reproduces the editor run exactly. Because each cell returns its defined names, another module can from analysis import summary and assert on it in a test.

Output.

invocation	executes	produces
`marimo edit analysis.py`	cells in DAG order (interactive)	live `summary` in the editor
`python analysis.py`	same cells, same order (headless)	identical `summary`, no browser

Rule of thumb. If your notebook must also be a scheduled job or pass code review, choose a format that is already Python — reproducibility you get for free beats reproducibility you have to remember.

Marimo interview question on reproducibility

Question. A teammate reports that a .ipynb gives different numbers depending on who runs it, and blames "the data." You suspect out-of-order execution. Explain how moving to Marimo removes that failure mode, and what specifically guarantees a deterministic result.

Solution Using DAG-ordered execution instead of click order

Code.

rate = 1.08                      # cell A — defines rate

adjusted = base_amounts * rate   # cell B — uses rate to build adjusted

report = adjusted.describe()     # cell C — uses adjusted to build report
report

Step-by-step trace.

environment	how order is chosen	result if cells were clicked A, C, B
Classic notebook	human click order	C sees old/undefined `adjusted` → wrong or error
Marimo	topological sort of the DAG	always A → B → C → correct `report`

The three cells form a chain A → B → C because B references rate and C references adjusted.
In a classic notebook, running them A, C, B produces a report built from a stale or missing adjusted, and the number depends on run history — not the data.
Marimo ignores the order you interact with cells and executes in topological order every time, so A runs before B runs before C, deterministically.
Because the file is pure Python with no stored outputs, a second person cloning the repo and running it gets byte-identical execution order.

Output:

run by	execution order	`report`
author (Marimo)	A → B → C	correct
teammate (Marimo)	A → B → C	identical

Why this works — concept by concept:

Topological order — deriving execution order from dependencies means the answer cannot depend on click history, which is the root cause of "works on my machine" notebooks.
Pure-Python format — no serialized outputs and no execution_count means the file that runs is exactly the file in git; there is nothing hidden to diverge.
Importable module — because cells return their names, the notebook can be imported and unit-tested, turning "trust me" into an assertion in CI.
Determinism as a default — reproducibility is guaranteed by the runtime rather than by a "Restart & Run All" ritual people forget.
Cost — the topological sort is O(cells + edges) once per run; the reproducibility it buys is unbounded in engineering time saved.

Pipelines
Topic — pipelines
Reproducible pipeline-ordering problems

Practice →

Analysis Topic — data-analysis End-to-end analysis reproducibility problems

Practice →

4. Interactive UI elements bound to variables

A widget is a Python object; reading its `.value` in another cell wires reactivity — no callbacks, no state juggling

Marimo's interactivity is the reactive DAG applied to human input. A UI element — mo.ui.slider, mo.ui.dropdown, mo.ui.text, mo.ui.table — is just a Python object with a live .value. The instant you read that .value in a different cell, that cell becomes a dependent of the widget, so moving the slider reruns exactly the cells that consume it. You never register a callback, never mutate shared state, never wire an event handler — the graph does it.

The binding model.

The element is a value. slider = mo.ui.slider(1, 100, value=10) creates an object; displaying it renders the control, and slider.value reads the current position as a plain Python number.
Define and display in one cell, read in another. The reactive contract is: the widget is defined and shown in its own cell, and other cells read slider.value. Reading it elsewhere is what draws the dependency edge.
Interaction = a cell edit. Dragging the slider is, to the runtime, equivalent to changing the cell that owns slider — so Marimo reruns the widget's descendants and nothing else.

Why there are no callbacks.

No on_change handlers. Traditional widget libraries make you attach a function that fires on change and mutate globals; that is imperative state management and it is where dashboards rot.
Reactivity replaces events. Because the dependent cell already declares what it needs (slider.value), Marimo knows what to rerun without you describing when. Declarative beats imperative.
A dashboard for free. Pair the widgets and their dependent charts, run marimo run, and the notebook is an interactive app — same file, no framework.

The control knobs.

mo.stop(predicate, output) short-circuits a cell and its descendants when a guard is true (e.g. no file uploaded yet), so an expensive graph does not run on empty input.
mo.ui.form wraps inputs so the graph reruns on submit rather than on every keystroke — the right tool for costly downstream work.
Compose with mo.ui.array / mo.ui.dictionary to build a list or map of elements and read them all through one .value.

Worked example — a slider that filters a frame

Detailed explanation. The canonical interactive pattern is a slider that sets a threshold and a downstream cell that filters a dataframe by it. Define the slider in one cell, read slider.value in the filter cell, and the filtered frame (and any chart built from it) updates the moment you drag — with zero event wiring.

Question. Build a min_amount slider from 0 to 100 and a cell that shows only the orders rows whose amount is at least the slider value. What reruns when you drag it to 50?

Input.

order_id	amount
1	20
2	60
3	90

Code.

import marimo as mo

min_amount = mo.ui.slider(0, 100, value=0, label="min amount")   # cell 1 — define + display
min_amount

filtered = orders[orders["amount"] >= min_amount.value]   # cell 2 — reads .value, depends on slider
filtered

Step-by-step explanation. Cell 1 creates the slider object and displays the control; min_amount.value starts at 0. Cell 2 reads min_amount.value, so Marimo draws an edge from the slider cell to the filter cell. Dragging the slider to 50 is treated as a change to cell 1, so Marimo reruns cell 2 (and any chart cell below it) — recomputing filtered with the new threshold. No on_change, no global mutation: the dependency edge already said what to do.

Output.

slider value	rows in `filtered`
0	1, 2, 3
50	2, 3
80	3

Rule of thumb. Define-and-display the widget in one cell, read .value in the cells that consume it — the moment you read the value elsewhere, the widget is wired reactively with no callback.

Marimo interview question on interactive reactivity

Question. You have an expensive model-fit cell that reads a mo.ui.dropdown of dataset names and a mo.ui.slider of epochs. The interviewer wants the fit to run only when the user explicitly submits — not on every twitch of the slider — while still being fully reactive. How do you build it?

Solution Using mo.ui.form to batch inputs before rerun

Code.

import marimo as mo

controls = mo.ui.dictionary({    # cell 1 — form reruns on SUBMIT, not per keystroke
    "dataset": mo.ui.dropdown(["a", "b", "c"], value="a"),
    "epochs":  mo.ui.slider(1, 50, value=10),
}).form()
controls

mo.stop(controls.value is None, mo.md("Set options and press **Submit**."))  # cell 2 — guard
params = controls.value                       # only set after submit
model = expensive_fit(params["dataset"], params["epochs"])
model

Step-by-step trace.

user action	`controls.value`	cell 2 behaviour
dragging slider (no submit)	`None`	`mo.stop` halts — no fit
press Submit	`{"dataset": "b", "epochs": 30}`	guard passes → `expensive_fit` runs
drag again (no submit)	still last submitted	fit does not rerun

Wrapping the inputs in .form() means their .value only updates on Submit, so mid-drag changes do not touch the graph.
mo.stop(controls.value is None, ...) short-circuits cell 2 (and its descendants) until the first submit, so the expensive fit never runs on empty input.
After submit, controls.value is a dict of the chosen options; cell 2 reads it and runs expensive_fit exactly once.
Because the form is still a normal reactive object, a later submit reruns only cell 2 downstream — the reactivity is intact, just batched.

Output:

phase	fit runs?	why
before first submit	no	`mo.stop` guard
on submit	yes (once)	form value materialized
twiddling after submit	no	value unchanged until next submit

Why this works — concept by concept:

Widget as value — every input is a Python object whose .value participates in the DAG, so no event handlers are needed to connect input to computation.
Form batching — .form() defers the value update to submit, converting "rerun on every keystroke" into "rerun on intent," which is what expensive cells need.
mo.stop guard — short-circuiting the cell and its descendants keeps a costly graph dormant until inputs are valid, the reactive analogue of an early return.
Declarative reactivity — the dependent cell states what it reads, so Marimo reruns the right cells without you specifying when, eliminating callback spaghetti.
Cost — the fit is O(fit) and now runs only on submit; the reactive bookkeeping is O(descendants of the form), independent of how much the user fiddles.

DataFrames
Topic — dataframe-basics
Interactive filter-and-aggregate problems

Practice →

Analysis Topic — data-analysis Parameterized dashboard analysis problems

Practice →

5. SQL cells & DuckDB

`mo.sql` runs SQL over your Python dataframes with embedded DuckDB and hands back a dataframe — SQL and Python share one namespace

Marimo lets you write SQL as a first-class cell, and the engine underneath is embedded DuckDB. The magic is that a SQL cell can reference your in-memory Python dataframes by their variable name — no load step, no connection string — and it returns the query result as a dataframe that is itself a reactive variable. SQL feeds Python, which feeds more SQL, all inside the same DAG.

How a SQL cell works.

mo.sql(...) is the primitive. In the editor a SQL cell is authored as SQL, but it compiles to mo.sql("SELECT ..."); it executes the query on DuckDB and returns a dataframe.
Dataframes are tables. A Python dataframe named orders is queryable as FROM orders directly — DuckDB reads the pandas/polars frame in place, so there is no import or copy into a database.
The result is a reactive variable. Name the output (e.g. big_orders) and it becomes a node in the DAG; downstream Python or SQL cells that reference it rerun when the query changes.

Parameterizing and mixing.

f-string interpolation. Because the SQL is a Python string, you interpolate values with {...} — WHERE amount > {min_amount.value} wires a UI slider straight into a query, and moving the slider reruns the SQL cell.
SQL ⇄ Python round-trips. Clean in pandas, aggregate in SQL, chart in Python — each step is a cell, each output is a frame, and the DAG keeps them in sync.
One embedded engine. DuckDB runs in-process, so there is no server to provision; it is columnar and vectorized, so group-bys over millions of rows are fast on a laptop.

What interviewers probe.

Reactivity of results — the query output is a normal reactive frame, so a change upstream reruns the SQL and everything after it.
No hidden connection state — because DuckDB is embedded and the frames are in memory, there is no external database whose state could drift from the notebook.
When SQL beats pandas — set-based joins and aggregations read more clearly as SQL; row-wise Python logic stays in Python. Use each where it is strongest.

Worked example — SQL over a Python dataframe

Detailed explanation. The clearest demonstration is a SQL cell that aggregates an in-memory dataframe and returns a result you keep working with in Python. No CREATE TABLE, no INSERT, no connection — you name the frame in the FROM clause and DuckDB reads it directly.

Question. Given a Python dataframe orders(customer, amount), write a SQL cell that returns each customer's total spend, and show that the result is a dataframe you can use downstream.

Input.

customer	amount
ada	40
linus	15
ada	60

Code.

import marimo as mo

totals = mo.sql(          # SQL cell — returns a dataframe named `totals`
    """
    SELECT customer, SUM(amount) AS total
    FROM orders
    GROUP BY customer
    ORDER BY total DESC
    """
)

top_customer = totals.iloc[0]["customer"]   # downstream Python cell consumes it reactively
top_customer

Step-by-step explanation. mo.sql(...) hands the query to the embedded DuckDB engine, which resolves FROM orders against the in-memory Python frame — no load step. The engine groups by customer, sums amount, and returns the result as a dataframe bound to totals. Because totals is a normal reactive variable, the downstream cell reading totals.iloc[0] becomes its dependent, so if orders changes upstream, both the SQL cell and top_customer rerun.

Output.

customer	total
ada	100
linus	15

Rule of thumb. If a step is a join or a group-by, reach for a mo.sql cell over your dataframe — DuckDB reads the frame in place and the result is just another reactive frame.

Marimo interview question on parameterized SQL

Question. You want a mo.ui.slider to set a minimum spend, and a SQL cell that returns only customers above that threshold, updating live as the slider moves. Show how the slider value reaches the SQL and why the query reruns.

Solution Using f-string interpolation from a UI element into SQL

Code.

import marimo as mo

min_spend = mo.ui.slider(0, 100, value=0, label="min spend")   # cell 1 — threshold slider
min_spend

big_spenders = mo.sql(          # cell 2 — SQL interpolates the slider value
    f"""
    SELECT customer, SUM(amount) AS total
    FROM orders
    GROUP BY customer
    HAVING SUM(amount) >= {min_spend.value}
    ORDER BY total DESC
    """
)
big_spenders

Step-by-step trace.

slider value	interpolated `HAVING`	rows returned
0	`>= 0`	ada (100), linus (15)
50	`>= 50`	ada (100)
100	`>= 100`	ada (100)

Cell 2 reads min_spend.value inside the f-string, so Marimo draws an edge from the slider cell to the SQL cell.
Dragging the slider is treated as a change to cell 1, so Marimo reruns cell 2 — DuckDB re-executes the query with the new HAVING bound.
DuckDB resolves FROM orders against the in-memory frame each run; no reload or connection is involved.
The result big_spenders is a reactive frame, so any chart cell below it reruns too — the whole chain from slider to SQL to chart stays in sync.

Output:

slider	`big_spenders`
0	ada, linus
50	ada

Why this works — concept by concept:

Embedded DuckDB — an in-process columnar engine queries Python frames by name, so there is no server, no connection state, and no copy into a database.
f-string parameterization — because SQL is a Python string, a UI element's .value interpolates directly, wiring a slider into a query through the same DAG that governs everything else.
Result is a reactive frame — naming the query output makes it a node, so SQL can feed Python which can feed more SQL, all kept consistent by the runtime.
Live re-execution — moving the slider reruns only the SQL cell and its descendants, giving an interactive query with no callback or refresh button.
Cost — each rerun is O(query) on DuckDB's vectorized engine over the in-memory frame; the reactive overhead is O(descendants of the slider), not the whole notebook.

Analysis
Topic — data-analysis
SQL-over-dataframe analysis problems

Practice →

Pipelines Topic — pipelines SQL-and-Python pipeline problems

Practice →

Cheat sheet — Marimo recipes

Minimal reactive cell pair.

x = 21          # cell 1
y = x * 2       # cell 2 — reruns automatically when x changes
y

UI element bound to a variable.

import marimo as mo
n = mo.ui.slider(1, 100, value=10)   # define + display in one cell
n
squared = n.value ** 2               # in another cell: reading .value wires reactivity

SQL cell over a dataframe (embedded DuckDB).

result = mo.sql("SELECT customer, SUM(amount) AS total FROM orders GROUP BY customer")

Guard an expensive cell.

mo.stop(uploaded.value is None, mo.md("Upload a file to continue."))
model = expensive_fit(uploaded.value)

Run three ways from one file.

marimo edit notebook.py     # reactive editor
marimo run  notebook.py     # read-only interactive app (code hidden)
python      notebook.py     # headless script (cron / Airflow)

Sandbox with inline dependencies (reproducible env).

marimo edit --sandbox notebook.py   # deps pinned in the file via uv (PEP 723)

Choosing where logic lives.

Situation	Put it in
Join / group-by / filter over a frame	a `mo.sql` cell (DuckDB)
Row-wise Python / model fit	a Python cell
A value a human should tune	a `mo.ui` element read via `.value`
Reuse across notebooks / tests	a plain function imported into a cell

Frequently asked questions

What is Marimo (the reactive Python notebook)?

Marimo is an open-source Python notebook that runs reactively: it builds a dependency graph from your code and, when you change a cell, automatically reruns every cell that depends on it. There is no hidden state, because the runtime — not your click history — decides execution order. Notebooks are stored as pure .py files, so they diff in git, import as modules, and run as scripts or apps.

How is Marimo different from Jupyter?

Jupyter executes cells in whatever order you click them and stores the notebook as JSON with baked-in outputs, which invites out-of-order execution bugs, stale variables, and unreadable diffs. Marimo derives execution order from a dataflow DAG, forbids defining the same variable in two cells, and saves the notebook as a plain Python file. The practical effect is that a Marimo notebook is reproducible by construction, whereas a Jupyter notebook is reproducible only if you remember to "Restart & Run All."

How does Marimo know which cells to rerun?

Marimo statically analyzes each cell to see which global variables it defines and which it references, then draws an edge from the defining cell to the referencing cell. When a cell changes, Marimo reruns that cell and its transitive descendants in the graph — nothing upstream and nothing on an unrelated branch. Deleting a cell removes its variables and invalidates the cells that used them, so no cell keeps running against a ghost value.

Why are Marimo notebooks stored as `.py` files?

Because a plain Python file is diffable, reviewable, importable, and executable without a kernel. A .ipynb is JSON that mixes source, outputs, and execution counts, producing noisy diffs and merge conflicts and letting stale outputs live on disk. A Marimo .py file contains only code, so one edit is one line of diff, and the same file runs as an interactive app (marimo run) or a headless script (python notebook.py).

How do Marimo UI elements work without callbacks?

A UI element such as mo.ui.slider is a Python object with a live .value. You define and display it in one cell, and any other cell that reads its .value automatically becomes a dependent in the DAG. Interacting with the widget is treated as a change to its cell, so Marimo reruns only the dependent cells — there is no on_change handler and no global mutation, because the dependency edge already declares what to recompute.

Can Marimo run SQL?

Yes — Marimo has first-class SQL cells backed by an embedded DuckDB engine, invoked through mo.sql(...). A SQL cell can query your in-memory Python dataframes by variable name (no load step) and returns the result as a dataframe that is itself a reactive variable. You can interpolate Python values, including a UI element's .value, straight into the query with an f-string, so a slider can drive a live SQL result.

Practice on PipeCode

Pipecode.ai is Leetcode for Data Engineering — every Marimo idea above, from the reactive dataflow DAG and the single-definition rule to UI elements bound by `.value` and SQL-over-dataframe with DuckDB, maps to a hands-on practice room where you build the analysis against real graded inputs. PipeCode pairs each reading with 450+ DE-focused problems and a real-time scoring engine, so your answer to "how would you make this notebook reproducible?" holds up under a senior interviewer's depth probes.

Practice data-analysis problems now →
Pipeline-design drills →

Papermill & Notebook Pipelines: Parametrized, Scheduled, Version-Controlled Notebooks

Gowtham Potureddi — Thu, 24 Sep 2026 17:43:33 +0000

papermill notebook pipelines are what you get when you stop treating a Jupyter notebook as a throwaway scratchpad and start treating it as a runnable, parametrized job. Papermill is a small open-source tool — a Python library and a command-line program — that takes a notebook, injects a fresh set of parameters into it, runs every cell top to bottom, and writes a new notebook that contains the code, the injected values, and every output the run produced. The template is never mutated; each execution leaves behind its own fully-rendered artifact.

That one move changes the economics of the work an analyst already does in a notebook. Instead of copying a notebook, hand-editing the date at the top, running it, and screenshotting a chart into an email, you run papermill report.ipynb out/2026-09-15.ipynb -p run_date 2026-09-15 and the executed notebook — charts, tables, logs, and all — becomes the deliverable. This guide walks the five things an interviewer will actually probe: the parameters cell tag and papermill.execute_notebook, the executed output notebook as an artifact plus nbconvert, collecting structured results with scrapbook.glue, orchestrating notebooks on a schedule with Airflow's PapermillOperator, and version-controlling notebooks with nbdime. Each section pairs the concept with a Solution-Tail interview answer: code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works — including the honest part, which is when notebooks-as-pipelines is the wrong tool.

When you want hands-on reps immediately after reading, drill the pipeline-design practice library →, rehearse the run-cadence decisions on the scheduling practice set →, and shape the parametrized outputs on the data-transformation practice set →.

On this page

Why notebooks-as-pipelines — Papermill's job and its limits
The parameters cell tag & execute_notebook
The executed output notebook as an artifact
Collecting outputs with scrapbook & glue
Orchestrating & version-controlling notebook pipelines
Cheat sheet — Papermill recipes
Frequently asked questions
Practice on PipeCode

1. Why notebooks-as-pipelines — Papermill's job and its limits

Papermill parametrizes and executes a notebook, producing a second notebook — that one fact decides where it fits

The one-sentence invariant: Papermill runs a notebook the way a function runs a body — you pass parameters in, it executes, and it returns a new notebook with the results baked in. The input notebook is read-only during a run; the value Papermill produces is an output notebook, an ordinary .ipynb in which every cell has been executed and its outputs captured. Everything else — scheduling, storing to S3, rendering to HTML — is built on top of that single behaviour.

What Papermill actually is (and is not).

A parametrizer. Papermill's headline feature is injecting parameters into a notebook without editing the source. You mark one cell, and Papermill overrides its variables at run time.
An executor. It runs the notebook cell-by-cell against a Jupyter kernel (python3 by default, but any installed kernel), saving outputs and cell timings as it goes.
Not a scheduler. Papermill does not have a cron, a DAG, or a UI. You invoke it from a script, a CI job, or an orchestrator. "Scheduled notebooks" means something else triggers papermill.
Not a transformation framework. Papermill neither knows nor cares what the notebook does; there is no dependency graph between cells beyond top-to-bottom order.

The good-idea case — when the artifact is the point.

The output notebook is a self-documenting audit log. Inputs, code, logs, tables, and charts live in one file for one run — reproducible and reviewable months later without re-running anything.
The author is the operator. An analyst who already lives in Jupyter can promote their notebook to a scheduled job with zero rewrite into a "real" script, which is often the difference between a report shipping and not shipping.
Parametrized fan-out is trivial. Run the same notebook once per region, per date, or per customer by looping over parameter sets — each run is an independent artifact.

The anti-pattern case — when to reach for a script or task instead.

Hidden execution state. Notebooks invite out-of-order cell runs and lingering variables; a notebook that only works if you run cell 7 before cell 3 is a landmine. Papermill always runs top-to-bottom, which helps, but complex control flow still belongs in tested modules.
Heavy DAGs and shared logic. If tasks fan out, retry independently, or share a library, a notebook is the wrong container — extract the logic into a package and call it from Airflow tasks or a script.
Team-edited notebooks. Concurrent edits to a JSON notebook merge badly; long-lived, multi-author pipeline logic wants plain .py files under review.

What interviewers listen for.

Do you say "Papermill produces a new executed notebook, it does not mutate the input" early? — senior signal.
Do you separate "Papermill executes" from "something else schedules"? — required framing.
Can you name both the good case (the artifact is an audit log) and the anti-pattern (hidden state, heavy DAGs) unprompted? — the maturity signal.
Do you mention that parameters are injected via a tagged cell, not by editing source? — the whole mechanism.

Worked example — five lines that turn a notebook into a job

Detailed explanation. The canonical Papermill "hello world" takes an existing notebook and runs it with one overridden parameter, writing the executed result somewhere new. It looks trivial, and that is the point: the same call that runs a toy report scales unchanged to a nightly job, because Papermill only ever sees "a notebook, some parameters, an output path." No cell is edited by hand; the override is injected.

Question. Run a report.ipynb for a specific date and land the executed notebook at a new path, without touching the original.

Input.

argument	value
input notebook	`report.ipynb`
output notebook	`runs/report-2026-09-15.ipynb`
parameter `run_date`	`2026-09-15`

Code.

import papermill as pm

pm.execute_notebook(
    "report.ipynb",                       # input template (never modified)
    "runs/report-2026-09-15.ipynb",       # executed output artifact
    parameters={"run_date": "2026-09-15"},
)

Step-by-step explanation. execute_notebook reads report.ipynb, finds the cell tagged parameters, and inserts a new injected-parameters cell directly beneath it that sets run_date = "2026-09-15", overriding the default. It then launches the python3 kernel and runs every cell in order, capturing each cell's outputs. Finally it writes the fully-executed notebook to runs/report-2026-09-15.ipynb. The original report.ipynb is untouched — you could run it a hundred times with a hundred dates and get a hundred artifacts.

Output.

Papermill produced	value
input notebook	`report.ipynb` (unchanged)
output notebook	`runs/report-2026-09-15.ipynb` (all cells executed)
injected cell	`injected-parameters` with `run_date = "2026-09-15"`
captured	every cell's outputs + per-cell execution time

Rule of thumb. If the work already lives in a notebook and the deliverable is "this notebook, run with today's inputs," Papermill is the smallest possible upgrade from manual to reproducible — one execute_notebook call, no rewrite.

2. The parameters cell tag & execute_notebook

One tagged cell is the whole parameter interface — Papermill injects overrides right beneath it

Papermill has exactly one mechanism for parametrizing a notebook, and an interviewer who asks "how does Papermill inject parameters?" wants it precisely. You tag one cell with the cell tag parameters. That cell holds default assignments. At run time Papermill inserts a second, machine-generated cell — labelled injected-parameters — immediately after it, containing the values you passed. Because Python executes top-to-bottom, the injected assignments run after the defaults and win.

The tagging mechanism.

The parameters tag. In JupyterLab you add it via Property Inspector → Cell Tags → add tag parameters. It is metadata on the cell, not a comment; Papermill looks for exactly this tag.
Defaults belong in the tagged cell. Assign every parameter a sensible default there (run_date = "2026-01-01", region = "us"). This keeps the notebook runnable standalone in Jupyter with no Papermill involved.
The injected cell. Papermill writes injected-parameters right below the tagged cell. If no tagged cell exists, Papermill injects at the top and warns — your defaults then never run, a common footgun.

Passing parameters — library and CLI.

Library. pm.execute_notebook(input, output, parameters={"region": "eu", "limit": 500}) passes a dict; types are preserved (int stays int, list stays list).
CLI -p (typed). papermill in.ipynb out.ipynb -p limit 500 -p region eu — Papermill infers 500 as an int and eu as a string.
CLI -r (raw string). -r zip 02139 forces a string, so a zero-padded code is not mangled into the integer 2139.
CLI -y / -f (YAML / file). -y "region: eu\nskus: [a, b]" passes inline YAML; -f params.yaml reads a file — the clean way to pass nested or list parameters.
-k kernel. -k python3 (or an R / Scala kernel) picks the kernel to execute against.

Why injection beats editing source.

The template file is never modified, so it stays clean in version control and safe to run concurrently with different parameter sets.
Types survive: a dict passed in Python arrives as a dict, not a re-parsed string.
The injected cell is visible in the output notebook, so anyone reading the artifact sees exactly which values produced it.

Worked example — a parameters cell with defaults, overridden at run time

Detailed explanation. The everyday pattern is a first code cell tagged parameters that declares every input with a default, followed by cells that use those names. Papermill overrides a subset; the rest keep their defaults. This is what lets the same notebook run interactively (defaults) and as a job (overrides).

Question. A notebook's first cell is tagged parameters with region = "us" and limit = 100. You run it with Papermill passing region="eu" only. What values do the downstream cells see?

Input.

region = "us"    # first cell of report.ipynb, tagged "parameters"
limit = 100

Code.

import papermill as pm

pm.execute_notebook(
    "report.ipynb",
    "runs/report-eu.ipynb",
    parameters={"region": "eu"},        # override region; limit keeps its default
)

The CLI form is identical: papermill report.ipynb runs/report-eu.ipynb -p region eu.

Step-by-step explanation. Papermill locates the parameters-tagged cell and inserts injected-parameters right after it containing region = "eu". When the notebook runs, the tagged cell sets region = "us" and limit = 100, then the injected cell immediately reassigns region = "eu". Because limit was not passed, it keeps its default 100. Every downstream cell reads region == "eu" and limit == 100.

Output.

variable	source of final value	value at run time
`region`	injected-parameters (override)	`"eu"`
`limit`	parameters cell (default)	`100`

Rule of thumb. Give every parameter a default in the tagged cell, then override only what changes per run — the notebook stays runnable by hand and precise as a job.

Papermill interview question on parameter injection

Question. A teammate reports that their Papermill run "ignores the parameters" — the notebook always uses the defaults no matter what -p values they pass. The notebook opens fine and runs top-to-bottom in Jupyter. What is almost certainly wrong, and how would you prove and fix it?

Solution Using the parameters cell tag correctly

Code.

import papermill as pm
import nbformat

nb = nbformat.read("report.ipynb", as_version=4)
tagged = [                                       # prove it: which cells carry the tag?
    idx for idx, c in enumerate(nb.cells)
    if "parameters" in c.get("metadata", {}).get("tags", [])
]
print("cells tagged 'parameters':", tagged)     # [] means none — that's the bug

if not tagged:                                   # fix: tag the defaults cell, then re-run
    nb.cells[0]["metadata"].setdefault("tags", []).append("parameters")
    nbformat.write(nb, "report.ipynb")

pm.execute_notebook(
    "report.ipynb", "runs/report-eu.ipynb",
    parameters={"region": "eu"},
)

Step-by-step trace.

step	condition	Papermill behaviour
1	no cell tagged `parameters`	injects `injected-parameters` at top, emits a warning
2	injected cell runs first	later default cell reassigns `region = "us"`, clobbering the override
3	tag the defaults cell	injected cell now lands after defaults
4	re-run with `-p region eu`	override runs last and wins → `region == "eu"`

Papermill only overrides variables via the injected-parameters cell it inserts right after the parameters-tagged cell.
With no tagged cell, Papermill still injects — but at the top — so any later plain cell that reassigns region overwrites the injected value; the defaults appear to "win."
The warning No cell tagged 'parameters' in the output notebook is the diagnostic; the empty tagged list is the proof.
Tagging the defaults cell places the injected cell after the defaults, so the override is the last assignment and takes effect.

Output:

state	tagged cells	`region` at run time
before fix	none	`"us"` (default wins)
after fix	cell 0	`"eu"` (override wins)

Why this works — concept by concept:

Cell tag, not comment — Papermill keys off notebook metadata tags: ["parameters"]; a # parameters comment does nothing, which is the single most common Papermill mistake.
Injection position — overrides are inserted after the tagged cell so they run last; with no tag, injection at the top is beaten by later reassignments.
Defaults keep it runnable — the tagged cell's defaults let the notebook run in plain Jupyter, while Papermill supplies production values.
Inspect to diagnose — reading cell.metadata.tags with nbformat turns a vague "it ignores params" into a one-line proof.
Cost — injection and tag lookup are O(cells), trivial against the runtime of the notebook's actual work.

Pipelines
Topic — pipelines
Parametrized pipeline-design problems

Practice →

Transform Topic — data-transformation Parameter-driven transformation problems

Practice →

3. The executed output notebook as an artifact

The output notebook is the deliverable — a fully-run, self-contained record of one execution

The feature that makes Papermill a pipeline tool rather than a runner is that every execution produces a durable, human-readable artifact: the output notebook. The input template stays pristine; the output is the same notebook with every cell run, every output captured, per-cell timing recorded, and the injected parameters visible at the top. Months later you can open it and see exactly what code ran, on what inputs, with what result — no re-execution, no guessing.

What lands in the output notebook.

Executed cells with outputs. Tables, stdout, logs, and rendered charts (as embedded images) are all stored inline, so the file is portable and complete.
Execution metadata. Papermill records per-cell start/end times and a run status in the notebook metadata, so you can see which cell was slow or where it failed.
The injected-parameters cell. The exact values that produced this run are right there in the artifact — the run is self-describing.
Errors are captured, not hidden. By default a cell that raises stops the run and the traceback is written into the output notebook; the file is saved so you can debug the failure post-mortem.

Where the output can live.

Path-based backends. The output path can be local, s3://bucket/key.ipynb, gs://bucket/key.ipynb, or Azure Blob — Papermill uses fsspec-style handlers, so the same call writes to object storage with no code change.
Per-run paths. Templating the date or run id into the output path (runs/report-{date}.ipynb) keeps every execution's artifact instead of overwriting one file.

Turning the artifact into something to share.

--report-mode. Runs Papermill with input cells hidden in the rendered result, so stakeholders see outputs (charts, tables) without the code.
jupyter nbconvert. Converts the executed notebook to HTML or PDF: jupyter nbconvert --to html runs/report.ipynb. This is the standard "email a clean report" step — nbconvert renders, Papermill executes; they compose.
--no-input. An nbconvert flag that strips code cells from the rendered HTML/PDF for a purely visual report.

Worked example — execute to S3, then render to HTML

Detailed explanation. A production pattern is: execute the notebook and land the artifact in object storage keyed by run date, then render a human-friendly HTML from that executed notebook. The execute step preserves the machine-readable record; the nbconvert step produces the shareable document. Neither step touches the template.

Question. Run report.ipynb for 2026-09-15, store the executed notebook to S3 under a dated key, and produce a code-free HTML report from it.

Input.

step	tool	target
execute	papermill	`s3://reports/2026-09-15/report.ipynb`
render	nbconvert	`report-2026-09-15.html` (no code)

Code.

import papermill as pm
import subprocess

out_nb = "s3://reports/2026-09-15/report.ipynb"

pm.execute_notebook(
    "report.ipynb", out_nb,
    parameters={"run_date": "2026-09-15"},
)

subprocess.run(                            # render the executed notebook to code-free HTML
    ["jupyter", "nbconvert", "--to", "html", "--no-input",
     out_nb, "--output", "report-2026-09-15"],
    check=True,
)

Step-by-step explanation. execute_notebook runs the parametrized notebook and writes the executed artifact straight to S3 — the s3:// prefix routes the write through the object-store handler, no extra upload code. The executed notebook contains every output. jupyter nbconvert --to html --no-input then reads that executed notebook and emits report-2026-09-15.html containing only the rendered outputs (charts, tables), because --no-input drops the code cells. The template report.ipynb remains unchanged.

Output.

artifact	contents
`s3://reports/2026-09-15/report.ipynb`	executed notebook: code + outputs + injected params + timings
`report-2026-09-15.html`	rendered outputs only (no code), ready to email

Rule of thumb. Execute once to a dated, durable path for the audit trail; convert that same artifact with nbconvert for humans — never re-run the notebook just to get a different format.

Papermill interview question on failure artifacts

Question. Your nightly Papermill job failed at cell 12 of 20. A teammate says "the run is gone, we have to reproduce it locally." Are they right? What did Papermill leave behind, and how do you use it?

Solution Using the saved output notebook on failure

Code.

import papermill as pm

try:
    pm.execute_notebook(
        "etl.ipynb",
        "runs/etl-2026-09-15.ipynb",     # saved even if a cell raises
        parameters={"run_date": "2026-09-15"},
    )
except pm.exceptions.PapermillExecutionError as err:
    # The output notebook is already on disk with the traceback in cell 12.
    print("failed cell:", err.cell_index, "->", err.ename)
    raise                                 # let the orchestrator mark the task failed

Step-by-step trace.

step	event	what Papermill does
1	cells 1–11 run	outputs captured into the output notebook in progress
2	cell 12 raises	traceback written into cell 12's output
3	run halts	output notebook saved to `runs/etl-2026-09-15.ipynb`
4	`PapermillExecutionError`	raised to the caller with `cell_index=12` and error name

Papermill executes cells in order, persisting outputs as it goes, so partial progress is not lost.
When cell 12 raises, Papermill records the full traceback inside the output notebook rather than discarding it.
It then saves the output notebook to the specified path — the failure artifact exists on disk even though the run failed.
It re-raises PapermillExecutionError so the orchestrator sees a failed task, while you open the saved notebook to debug the exact failing cell with its inputs intact.

Output:

claim	reality
"the run is gone"	false — output notebook saved with the traceback
where to debug	open `runs/etl-2026-09-15.ipynb`, jump to cell 12

Why this works — concept by concept:

Fail-with-artifact — Papermill saves the output notebook even on error, so a failed run is a debuggable record, not a void.
Traceback in place — the exception is captured in the failing cell's output, so you see the error next to the code and the injected parameters that triggered it.
PapermillExecutionError — a typed exception carrying cell_index and error name lets orchestration fail cleanly and alert precisely.
Reproducibility for free — the injected-parameters cell means re-running the exact scenario is copy-paste, no guessing which date failed.
Cost — persisting outputs incrementally is O(output size); the debugging time it saves is the real payoff.

Pipelines
Topic — pipelines
Artifact-and-observability pipeline problems

Practice →

Transform Topic — data-transformation Output-shaping and reporting problems

Practice →

4. Collecting outputs with scrapbook & glue

`scrapbook.glue` writes named values into the notebook, `read_notebook` reads them back — data escapes the notebook without a side database

An executed notebook is great for humans, but a pipeline usually needs to pull structured results back out — a row count, a model score, a small DataFrame — so a downstream step can use it. Papermill's original record / read_notebook API is deprecated; the supported way is the companion library scrapbook. The mechanism is symmetric: inside the notebook you sb.glue("name", value) to persist a value, and outside you sb.read_notebook(path).scraps to read it back. The values ("scraps") are stored inside the output notebook itself.

The glue side (inside the notebook).

sb.glue("rows", 1240). Persists a JSON-serializable value under a name; it is stored in the cell's metadata as a scrap and survives in the output notebook.
sb.glue("summary", df, encoder="pandas"). Encoders let you glue richer objects (a DataFrame, an Arrow table) that round-trip back to the same type.
sb.glue("chart", fig, display=True). With display=True the value is also rendered visibly in the notebook, so it is both machine-readable data and a human-visible output.

The read side (downstream code).

sb.read_notebook(path).scraps. Returns a name → scrap mapping; .scraps["rows"].data is the original value with its type restored.
sb.read_notebooks(dir).scraps_report() / .papermill_dataframe. Read a whole directory of executed notebooks and assemble one row per run — exactly what you want after a parametrized fan-out.
Type fidelity. Because scraps carry an encoder tag, a glued DataFrame comes back as a DataFrame, not a re-parsed string.

Why this beats the alternatives.

No side channel. The result travels with the artifact — you do not need a separate table, file, or return-value plumbing to move a scalar from a notebook to the next step.
Batch-friendly. After running one notebook per region, a single read_notebooks call collects every region's metrics into a DataFrame for comparison.
Auditable. The glued value is visible in the same artifact that shows the code and inputs that produced it.

Worked example — glue a metric, read it in the next step

Detailed explanation. The everyday scrapbook pattern is: the notebook computes something and glues the number it wants to expose; an orchestrating script runs the notebook, then reads that number back to decide what to do next. This turns a notebook into a callable step with a return value.

Question. A notebook computes row_count. Glue it, run the notebook with Papermill, then read row_count back in the orchestrator and branch on it.

Input.

import scrapbook as sb        # inside etl.ipynb, after the load cell
row_count = len(df)          # e.g. 1240
sb.glue("row_count", row_count)

Code.

import papermill as pm
import scrapbook as sb

out = "runs/etl-2026-09-15.ipynb"
pm.execute_notebook("etl.ipynb", out, parameters={"run_date": "2026-09-15"})

nb = sb.read_notebook(out)                   # read the glued value back out
rows = nb.scraps["row_count"].data           # -> 1240, as an int

if rows == 0:
    raise ValueError("load produced zero rows — failing the pipeline")
print(f"loaded {rows} rows")

Step-by-step trace.

step	actor	effect
1	notebook	`sb.glue("row_count", 1240)` stores a scrap in the output notebook
2	papermill	writes the executed notebook (with the scrap) to `out`
3	orchestrator	`sb.read_notebook(out).scraps["row_count"].data` → `1240`
4	orchestrator	`rows != 0`, so the pipeline proceeds

Inside the notebook, glue serializes row_count and stores it as a named scrap in that cell's metadata.
Papermill saves the executed notebook, so the scrap travels inside the artifact — no external write.
The orchestrator reads the notebook back and pulls scraps["row_count"].data, recovering the integer 1240 with its type intact.
Because the value is a real int, the == 0 guard works directly, and the pipeline branches on genuine notebook output.

Output:

scrap	stored in	recovered value	type
`row_count`	`runs/etl-2026-09-15.ipynb`	`1240`	`int`

Why this works — concept by concept:

glue = persist-with-artifact — the value is written into the output notebook, so results and evidence never drift apart.
scraps carry types — an encoder tag round-trips ints, dicts, and DataFrames back to their original types instead of strings.
read_notebook as return value — the downstream step treats the notebook like a function that returned data, enabling real control flow (guards, branching).
No side database — moving a scalar between steps needs no extra table or file, cutting a whole class of plumbing bugs.
Cost — glue/read are O(value size); for scalars and small frames it is negligible next to the notebook's compute.

Transform
Topic — data-transformation
Extract-and-collect result problems

Practice →

Pipelines Topic — pipelines Step-to-step handoff pipeline problems

Practice →

5. Orchestrating & version-controlling notebook pipelines

Something else schedules Papermill, and nbdime tames the diff — the two things that make notebooks production-grade

Papermill executes; it does not schedule and it does not version-control. Making notebook pipelines production-grade means solving those two separately: an orchestrator (usually Airflow) triggers papermill on a cadence with per-run parameters, and a notebook-aware diff tool (nbdime) plus output stripping (nbstripout) make the JSON reviewable in git. Say it in one breath: Airflow's PapermillOperator runs the notebook on a schedule, and nbdime/nbstripout keep the source diffable.

Scheduling with Airflow.

PapermillOperator. From airflow.providers.papermill, it wraps execute_notebook: you give it input_nb, output_nb, and parameters, and it becomes a task in a DAG.
Templated per-run paths and dates. Use Jinja templating ({{ ds }}) so each scheduled run writes its own dated output notebook and passes the run date as a parameter — one artifact per run, no overwrites.
Idempotent parameters. Parametrize by the logical run date, not datetime.now(), so a re-run of the same interval reproduces the same result — the property that makes retries and backfills safe.
Operator-level retries. Retries, alerts, and SLAs live on the Airflow task, not in the notebook; Papermill just runs the notebook and fails loudly if a cell raises.

Version-controlling notebooks.

The problem. A .ipynb is JSON containing source, outputs, execution counts, and metadata, so a one-line code change shows up as a huge, unreadable git diff full of base64 image blobs.
nbstripout. A git filter that strips outputs and execution counts on commit, so the repo stores only source — small, mergeable diffs.
nbdime. Content-aware diff/merge: nbdiff for the terminal, nbdiff-web for a rendered side-by-side, and nbdime config-git --enable to make git diff / git merge understand notebooks.
jupytext pairing. Pair each .ipynb with a synced .py "percent" script so pull requests review clean Python while the notebook stays runnable — a common belt-and-suspenders setup.

Worked example — a scheduled PapermillOperator task

Detailed explanation. The standard way to schedule a notebook is a single Airflow task built from PapermillOperator, templated so each daily run injects that day's logical date and writes a dated artifact. The DAG owns cadence and retries; the operator owns the Papermill call.

Question. Define a daily Airflow task that runs etl.ipynb, passing the run's logical date as run_date and writing the executed notebook to a dated path.

Input.

field	value
schedule	daily (`@daily`)
input notebook	`/nbs/etl.ipynb`
output notebook	`/runs/etl-{{ ds }}.ipynb`
parameter	`run_date = {{ ds }}`

Code.

from airflow import DAG
from airflow.providers.papermill.operators.papermill import PapermillOperator
import pendulum

with DAG(
    dag_id="etl_notebook",
    schedule="@daily",
    start_date=pendulum.datetime(2026, 1, 1, tz="UTC"),
    catchup=False,
) as dag:
    run_etl = PapermillOperator(
        task_id="run_etl",
        input_nb="/nbs/etl.ipynb",
        output_nb="/runs/etl-{{ ds }}.ipynb",      # dated artifact per run
        parameters={"run_date": "{{ ds }}"},        # logical date, not now()
    )

Step-by-step explanation. Airflow schedules etl_notebook daily. For each run it renders the Jinja templates: {{ ds }} becomes that interval's logical date (e.g. 2026-09-15). PapermillOperator then calls execute_notebook("/nbs/etl.ipynb", "/runs/etl-2026-09-15.ipynb", parameters={"run_date": "2026-09-15"}). Because the date comes from the scheduler's logical date, re-running the task reproduces the same output; retries are safe. Each day leaves its own dated artifact.

Output.

run interval	output notebook	injected `run_date`
2026-09-15	`/runs/etl-2026-09-15.ipynb`	`2026-09-15`
2026-09-16	`/runs/etl-2026-09-16.ipynb`	`2026-09-16`

Rule of thumb. Parametrize by the orchestrator's logical run date, never wall-clock now() — that is what makes a scheduled notebook idempotent and backfillable.

Papermill interview question on notebooks in version control

Question. Reviewers refuse to approve notebook PRs because "the diffs are unreadable" — a two-line code change shows thousands of changed lines. The team still wants to keep working in notebooks. What do you set up so notebook changes review cleanly without abandoning Jupyter?

Solution Using nbstripout and nbdime as git integrations

Code.

pip install nbstripout nbdime
nbstripout --install                       # git clean filter: strip outputs on commit
nbdime config-git --enable --global        # make git diff/merge notebook-aware
nbdiff-web notebooks/etl.ipynb             # rendered side-by-side diff in the browser

Step-by-step trace.

step	tool	effect on the repo / review
1	`nbstripout --install`	commits drop outputs + `execution_count` → tiny diffs
2	`nbdime config-git --enable`	`git diff` shows cell-level source changes, not JSON
3	`nbdiff-web`	renders a two-column diff a reviewer can actually read
4	merge conflict	`nbmerge` resolves per-cell instead of per-JSON-line

nbstripout installs a git clean filter that removes outputs and execution counts before content is staged, so the stored notebook is mostly source — the base64 image blobs that bloated the diff are gone.
nbdime config-git --enable registers nbdime as git's diff and merge driver for .ipynb, so git diff compares notebooks by cell content rather than raw JSON lines.
nbdiff-web gives reviewers a rendered, side-by-side view where a two-line change looks like a two-line change.
When two branches edit the same notebook, nbmerge reconciles cell-by-cell, avoiding the unresolvable JSON conflicts that make notebook collaboration painful.

Output:

before	after
2-line change → thousands of diff lines	2-line change → 2 diff lines
unmergeable JSON conflicts	cell-level `nbmerge` resolution

Why this works — concept by concept:

Strip outputs at the boundary — nbstripout removes the volatile, huge parts (outputs, counts) at commit time, so version control tracks intent, not render noise.
Content-aware diff — nbdime compares the notebook's cell structure rather than its serialized JSON, turning an unreadable blob diff into a code review.
Git integration — wiring both tools into git means reviewers get clean diffs with no change to how authors work in Jupyter.
Mergeable notebooks — nbmerge resolves per cell, removing the "notebooks can't be collaborated on" objection.
Cost — filters and diffs are O(notebook size) at commit/review time; the throughput cost is trivial next to the review friction they remove.

Scheduling
Topic — scheduling
Scheduled-run and idempotency problems

Practice →

Pipelines Topic — pipelines Orchestration and DAG-design problems

Practice →

Cheat sheet — Papermill recipes

Minimal execute.

import papermill as pm
pm.execute_notebook("in.ipynb", "out.ipynb", parameters={"run_date": "2026-09-15"})

CLI parametrized run.

papermill in.ipynb out.ipynb -p region eu -p limit 500 -r zip 02139 -k python3

YAML params file.

papermill in.ipynb out.ipynb -f params.yaml

where params.yaml contains:

region: eu
skus: [a, b, c]

Glue a value and read it back.

import scrapbook as sb                # inside the notebook
sb.glue("row_count", len(df))

nb = sb.read_notebook("out.ipynb")   # downstream: read it back
rows = nb.scraps["row_count"].data

Airflow PapermillOperator task.

from airflow.providers.papermill.operators.papermill import PapermillOperator
PapermillOperator(
    task_id="run_etl",
    input_nb="/nbs/etl.ipynb",
    output_nb="/runs/etl-{{ ds }}.ipynb",
    parameters={"run_date": "{{ ds }}"},
)

Clean notebook diffs in git.

nbstripout --install                 # strip outputs on commit
nbdime config-git --enable --global  # notebook-aware diff/merge
jupyter nbconvert --to html --no-input out.ipynb   # code-free report

Choosing the pattern.

Situation	Use
Run the same notebook per date / region	Papermill parameters + dated output path
Get a scalar / frame back to the caller	`scrapbook.glue` + `read_notebook`
Trigger on a schedule with retries	Airflow `PapermillOperator`
Share outputs without code	`nbconvert --no-input` (HTML/PDF)
Keep notebook PRs reviewable	`nbstripout` + `nbdime`

Frequently asked questions

What is Papermill?

Papermill is an open-source tool (a Python library and a CLI) for parametrizing and executing Jupyter notebooks. You pass parameters into a notebook without editing its source, Papermill runs every cell against a kernel, and it writes a new output notebook containing the code, the injected parameters, and every output the run produced. It does not schedule or transform — it executes a notebook and hands you the executed artifact.

How do I parametrize a Jupyter notebook with Papermill?

Tag one cell with the cell tag parameters and put default variable assignments in it. At run time Papermill injects a new injected-parameters cell immediately after the tagged cell, overriding those defaults with the values you pass via pm.execute_notebook(..., parameters={...}) or the CLI -p/-r/-y/-f flags. The defaults keep the notebook runnable by hand; the injected values win because they run last.

What is the output notebook in Papermill?

It is the executed artifact Papermill writes for each run — the input notebook with every cell run, all outputs captured inline (tables, logs, charts), per-cell timings recorded, and the injected-parameters cell visible. The input template is never modified. If a cell raises, Papermill still saves the output notebook with the traceback in place, so a failed run remains fully debuggable.

How do I get values back out of a Papermill notebook?

Use the companion library scrapbook (Papermill's old record/read_notebook API is deprecated). Inside the notebook call sb.glue("name", value) to store a value in the output notebook; downstream, call sb.read_notebook(path).scraps["name"].data to read it back with its type intact. For a batch of runs, sb.read_notebooks(dir) assembles one row per notebook so you can compare metrics across a parametrized fan-out.

How do I schedule Papermill notebooks?

Papermill has no scheduler, so you trigger it from an orchestrator. In Airflow, the PapermillOperator (from airflow.providers.papermill) wraps execute_notebook: give it input_nb, a templated output_nb, and parameters, and template the run's logical date ({{ ds }}) instead of now() so re-runs are idempotent. Retries, alerts, and SLAs live on the Airflow task; Papermill just runs the notebook and fails loudly if a cell errors.

When are notebooks-as-pipelines a bad idea?

When the logic is complex, shared, or heavily branched. Notebooks encourage hidden execution state, they are awkward to unit-test, and their JSON format makes multi-author collaboration and code review painful. Papermill mitigates the run-order problem by always executing top-to-bottom, but for reusable libraries, fan-out DAGs, or team-owned pipeline logic, extract the code into tested .py modules and call them from tasks — keep notebooks for the reporting and exploratory jobs where the rendered artifact is the deliverable.

Practice on PipeCode

Pipecode.ai is Leetcode for Data Engineering — every Papermill idea above, from the parameters cell tag to the executed output artifact, scrapbook glue, and the scheduled PapermillOperator, maps to a hands-on practice room where you design the pipeline against real graded inputs. PipeCode pairs each reading with 450+ DE-focused problems and a real-time scoring engine, so your answer to "how would you make this scheduled notebook idempotent?" holds up under a senior interviewer's depth probes.

Practice pipeline problems now →
Scheduling drills →

Streamlit for Data Engineers: Internal Data Apps & Pipeline Dashboards

Gowtham Potureddi — Thu, 24 Sep 2026 17:40:15 +0000

streamlit for data engineers answers a very specific, very common need: you have a warehouse full of pipeline metadata, freshness timestamps, and row counts, and someone — an analyst, an on-call engineer, your own future self at 3 a.m. — needs to see it and act on it without you standing up a React frontend and a Flask API. Streamlit turns a single Python script into a web app. You write import streamlit as st, add a few st. calls that print DataFrames and draw charts, run streamlit run app.py, and you have an internal tool on a URL. No HTML, no JavaScript, no callback wiring, no template engine.

That shape is a real departure from the two things data engineers reached for before it: a Jupyter notebook that only you can run and that no non-technical teammate will ever open, or a "proper" web app that costs a week of frontend work you do not want to own. Streamlit lives in between — it is code-first like the notebook and shareable like the web app. This guide walks the four ideas an interviewer or a code reviewer will actually probe when you put Streamlit in your stack — the top-to-bottom rerun model with widgets and session_state, the two caching decorators, displaying data and connecting to a warehouse, and assembling all of it into a pipeline-health / SLA dashboard — and pairs each with a Solution-Tail interview answer: code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the DataFrame shaping that feeds every widget on the data-analysis practice library →, sharpen the transforms behind your tables on the dataframe-basics practice set →, and rehearse the freshness-and-breach logic your dashboard renders on the sla-monitoring practice set →.

On this page

Why Streamlit fits data engineering in 2026
The rerun model, widgets & session_state
Caching — st.cache_data vs st.cache_resource
Displaying data & connecting to warehouses
Building a pipeline-health / SLA dashboard
Cheat sheet — Streamlit recipes
Frequently asked questions
Practice on PipeCode

1. Why Streamlit fits data engineering in 2026

Streamlit is a Python script that renders as a web app — that one fact decides where it fits and how it behaves

The one-sentence invariant: a Streamlit app is an ordinary Python script that the framework re-executes top to bottom every time the user interacts with it, turning st. calls into UI. Everything surprising about Streamlit — why a variable resets, why caching matters so much, why you need session_state — falls out of that single sentence. There is no component tree you register, no event loop you write, no separate frontend build. You write the script the way you would write an analysis, and Streamlit renders it.

What Streamlit does — and deliberately does not — do.

UI from Python. st.dataframe(df), st.line_chart(df), st.button("Run"), st.metric("Rows", 1200) — each call emits a widget. You never touch HTML or CSS for the common cases.
State and rerun handled for you. The framework owns the render loop: interact with a widget and the whole script reruns. You do not manage a DOM or diff a virtual tree.
Not a BI product, not a heavy web framework. Streamlit is not trying to be Tableau (no semantic layer, no governed metrics) and not trying to be Django (no ORM, no auth framework, no routing beyond simple multipage). It is the fastest path from "I have a DataFrame" to "my team can click on it."

Where Streamlit sits against the alternatives data engineers weigh.

vs a Jupyter notebook. A notebook is for you exploring; a Streamlit app is for others using. The notebook has hidden execution order and stale cells; Streamlit reruns cleanly every time, so what the viewer sees always matches the current inputs.
vs Flask / FastAPI + a JS frontend. A hand-rolled web app gives you total control and costs days of frontend work you must maintain. Streamlit trades that control for a Python-only script you finish in an afternoon — the right trade for an internal tool with ten users, the wrong one for a public product.
vs Dash / Panel. Dash is callback-oriented (you wire inputs to outputs explicitly); Streamlit is rerun-oriented (no callbacks needed for the basics). Streamlit is faster to write; Dash gives finer-grained control when you truly need it.
vs a BI dashboard (Looker, Metabase). BI tools own governed metrics and scheduled reports; Streamlit owns interactive tools that do things — trigger a backfill, edit a config, kick a re-run — which a read-only BI tile cannot.

What interviewers listen for.

Do you say "the whole script reruns on every interaction" in the first sentence? — this is the senior signal; everything else is a corollary.
Do you reach for st.cache_data / st.cache_resource unprompted when you mention loading data, because you know the rerun would otherwise re-query every time? — required framing.
Do you know that a plain Python variable does not survive a rerun, and that st.session_state is the fix? — the number-one beginner gap.
Do you place Streamlit as "internal tools and dashboards, not a public product" rather than as a Flask replacement for everything? — placement maturity.

Worked example — six lines that become a shareable app

Detailed explanation. The canonical Streamlit "hello world" reads a DataFrame and renders it as a title, an interactive table, and a chart. It looks trivial, and that is the point: the same six lines that render a toy DataFrame scale unchanged to a live warehouse query, because Streamlit only ever sees a DataFrame and a few st. calls. There is no server code, no route, no template — the script is the app.

Question. Turn a two-row pipeline-runs DataFrame into a titled app with an interactive table and a bar chart, runnable with one command.

Input.

pipeline	rows_loaded
orders_el	4200
events_el	9100

Code.

import streamlit as st
import pandas as pd

st.title("Pipeline runs")

df = pd.DataFrame(
    {"pipeline": ["orders_el", "events_el"], "rows_loaded": [4200, 9100]}
)

st.dataframe(df)                       # interactive, sortable table
st.bar_chart(df, x="pipeline", y="rows_loaded")

Step-by-step explanation. import streamlit as st pulls in the whole UI toolkit. st.title(...) emits an <h1> at the top of the page. Building df is ordinary pandas — Streamlit has no opinion about how you get the DataFrame. st.dataframe(df) renders a sortable, scrollable, searchable grid; st.bar_chart(...) draws a chart from the same frame with no plotting library imported. You launch it with streamlit run app.py, which starts a local server and opens the browser; every save auto-reloads.

Output.

the app shows	rendered from
page title "Pipeline runs"	`st.title(...)`
an interactive 2-row table	`st.dataframe(df)`
a bar chart, pipeline vs rows_loaded	`st.bar_chart(...)`
a live URL on `localhost:8501`	`streamlit run app.py`

Rule of thumb. If your logic ends in a DataFrame, a number, or a chart, Streamlit can display it — the toy example and the production dashboard differ only in where the DataFrame comes from, never in the st. calls that render it.

2. The rerun model, widgets & session_state

The whole script reruns top-to-bottom on every interaction — widgets return values, `session_state` is what survives

If you internalise one thing about Streamlit, make it this: there is no event loop and no callback graph; every widget interaction re-executes your entire script from the first line to the last. A button click, a slider drag, a dropdown change — each schedules a fresh top-to-bottom run. Widgets are not "handlers you attach"; they are function calls that return their current value on each run. This is why a normal variable resets and why state needs a special home.

The rerun loop, precisely.

Interaction → rerun. The user moves a slider; Streamlit re-runs app.py from the top. Your code executes again, the slider call returns the new value, and the page is redrawn from scratch.
Widgets return values, not events. value = st.slider("Days", 1, 30, 7) returns 7 on the first run and whatever the user picked on later runs. You use the return value inline — no onChange handler.
Plain variables do not persist. Any x = 0 at the top is re-initialised to 0 on every rerun. If you increment it on a click, the increment is wiped by the next rerun. This is the single most common beginner bug.

st.session_state — the store that survives reruns.

A dict scoped to the browser session. st.session_state is a dictionary-like object that persists across reruns for one user session. Write st.session_state["count"] = 0 once, read and mutate it on later runs.
Widgets can bind to it via key. Give a widget key="days" and its value is mirrored at st.session_state["days"]; reading the key elsewhere gives the live value without threading it through function arguments.
Initialise defensively. Because the script reruns, guard initialisation with if "count" not in st.session_state: so you seed the value once and never clobber it.

Callbacks and the button gotcha.

Callbacks run before the rerun. st.button("Add", on_click=fn) runs fn first, then reruns the script. Callbacks (on_click, on_change) are where you mutate session_state cleanly, and they receive args / kwargs.
st.button is transient. st.button(...) returns True only on the single rerun immediately following the click, then False again. So if st.button("Go"): x = compute() recomputes once — do not expect the True to "stick." For durable toggles use st.session_state or st.checkbox.

Worked example — a counter that actually counts

Detailed explanation. The classic demonstration of the rerun model is a click counter, because the naive version is broken in a way that teaches the whole model. A plain variable is reset by the rerun; session_state plus a callback is the fix. Getting this right is the difference between an app that "forgets" and one that holds state.

Question. Build a button that increments a counter and shows the running total, surviving every rerun.

Input. Three clicks of the "Add one" button.

Code.

import streamlit as st

if "count" not in st.session_state:     # seed once, never on later reruns
    st.session_state.count = 0

def increment():                        # callback runs BEFORE the rerun
    st.session_state.count += 1

st.button("Add one", on_click=increment)
st.write("Count:", st.session_state.count)

Step-by-step explanation. On the first run, count is absent, so it is seeded to 0. Clicking the button fires increment before the rerun, so count becomes 1; then the script reruns, the if guard is skipped (key exists), and st.write prints 1. Each subsequent click repeats: callback mutates state, script reruns, the guard leaves the existing value alone. Had we written count = 0 as a plain variable at the top, every rerun would reset it to 0 and the display would never exceed 1.

Output.

click #	value before callback	value shown
1	0	1
2	1	2
3	2	3

Rule of thumb. Anything that must outlive a rerun lives in st.session_state; mutate it in a callback, and guard its initialisation with an if key not in st.session_state check.

Streamlit interview question on the rerun model

Question. A candidate writes clicks = 0 at the top of the script and does if st.button("Click"): clicks += 1; st.write(clicks). The display never goes past 1. Explain exactly why, and fix it so the count accumulates.

Solution Using session_state seeded once with a callback

Code.

import streamlit as st

# BROKEN: `clicks` is a plain variable, reset to 0 on every rerun
# clicks = 0
# if st.button("Click"):
#     clicks += 1
# st.write(clicks)          # always 0 or 1

# FIXED: state lives in session_state, mutated in a callback
if "clicks" not in st.session_state:
    st.session_state.clicks = 0

def bump():
    st.session_state.clicks += 1

st.button("Click", on_click=bump)
st.write(st.session_state.clicks)

Step-by-step trace.

event	rerun #	`clicks` at top	after handler	displayed
initial load	1	seeded 0	0	0
click	2	0 (state)	bump → 1	1
click	3	1 (state)	bump → 2	2
click	4	2 (state)	bump → 3	3

In the broken version, clicks = 0 executes on every rerun, so the pre-increment value is always 0; the click makes it 1, and the next rerun resets it. The counter can never exceed 1.
st.button(...) returns True only on the rerun right after the click, so the += 1 fires at most once per click — but against a variable that was just reset.
The fix seeds clicks in session_state once, guarded by the not in check, so it is not clobbered on reruns.
The on_click=bump callback mutates the persisted value before the redraw, so st.write reads the accumulated total.

Output:

after 3 clicks	broken version	fixed version
displayed count	1	3

Why this works — concept by concept:

Top-to-bottom rerun — because the entire script re-executes on each interaction, any assignment at module scope is re-run and therefore re-initialised; state cannot live in a local variable.
session_state persistence — a dict scoped to the session survives reruns, so a value written once is readable and mutable on every later run.
Guarded initialisation — the if key not in st.session_state seed runs on the first rerun only, protecting the accumulated value from being reset.
Callback ordering — on_click runs before the rerun, so the mutation is visible to the redraw in the same cycle, avoiding an off-by-one lag.
Cost — the rerun is O(script length) each interaction; state access is O(1), which is why heavy work must be cached rather than repeated.

Analysis
Topic — data-analysis
Interactive data-analysis app problems

Practice →

DataFrames Topic — dataframe-basics DataFrame state-and-filter problems

Practice →

3. Caching — st.cache_data vs st.cache_resource

The rerun re-runs everything, so caching is not optional — pick `st.cache_data` for values and `st.cache_resource` for connections

Because the whole script reruns on every click, an uncached pd.read_sql(...) re-queries the warehouse every time a user drags a slider — unusable. Streamlit's answer is two decorators, and the interview question is always "which one and why." Say it in one breath: st.cache_data memoizes the return value and hands each caller a fresh copy; st.cache_resource caches a single global object and hands every caller the same instance.

st.cache_data — for data you compute or fetch.

Memoizes serializable results. Wrap a function that returns a DataFrame, a dict, a list, an API response — anything picklable. On a cache hit Streamlit skips the body and returns the stored value.
Keyed on the function's inputs. Streamlit hashes the arguments (and the function's code); same args → cache hit, different args → recompute. So load(start, end) caches per date range automatically.
Returns a copy every time. Each caller gets its own copy of the result, so one user mutating the DataFrame cannot corrupt another user's cached copy. That safety is the whole reason it is separate from cache_resource.

st.cache_resource — for global singletons.

Caches the object itself, not a copy. A database connection, a SQLAlchemy engine, an ML model, a client handle — things that are expensive to create and meant to be shared. Every caller receives the same live object.
Not serialized, not copied. Because it is shared, it must be safe for concurrent use; a connection pool or a thread-safe client is the right kind of thing to put here.
One per app, across sessions. Unlike session_state (per session), a cache_resource object is shared across all users and reruns until the app restarts or you clear it.

The knobs and the failure modes.

ttl. Expire entries after N seconds (ttl=600) so a dashboard shows data at most ten minutes stale without a manual refresh.
max_entries. Cap the number of cached results (e.g. max_entries=50) to bound memory when the argument space is large.
show_spinner and .clear(). Toggle the "Running..." spinner, and call load.clear() to evict programmatically (e.g. after a backfill writes new rows).
The classic mistake: putting a database connection in st.cache_data (it tries to pickle the connection and fails or misbehaves), or putting a DataFrame you mutate in st.cache_resource (mutations leak across users because there is no copy).

Worked example — cache the query result, cache the engine once

Detailed explanation. The everyday pattern pairs the two decorators: cache_resource opens the SQLAlchemy engine one time and shares it; cache_data runs the query and memoizes the resulting DataFrame per set of parameters. Together they turn a per-rerun round trip into a per-argument-set round trip.

Question. Write a helper that opens a Postgres engine once and a query function that caches its DataFrame for five minutes, keyed on the day range.

Input. Two reruns with the same days=7, then one with days=30.

Code.

import streamlit as st
import pandas as pd
from sqlalchemy import create_engine, text

@st.cache_resource                      # one engine, shared across all reruns/users
def get_engine():
    return create_engine(st.secrets["db_url"])

@st.cache_data(ttl=300)                 # memoize the DataFrame per `days`, 5-min TTL
def load_runs(days: int) -> pd.DataFrame:
    sql = text("SELECT pipeline, status, ended_at FROM runs "
               "WHERE ended_at > now() - make_interval(days => :d)")
    with get_engine().connect() as conn:
        return pd.read_sql(sql, conn, params={"d": days})

days = st.slider("Look-back (days)", 1, 30, 7)
st.dataframe(load_runs(days))

Step-by-step explanation. The first rerun calls get_engine(), which builds the engine and caches the object; load_runs(7) runs the SQL and caches the DataFrame under key days=7. Dragging the slider and releasing at 7 again reruns the script, but get_engine() returns the cached engine and load_runs(7) returns the cached DataFrame — zero database work. Moving the slider to 30 is a new argument, so load_runs(30) misses the cache and queries once, then caches that too. After five minutes the days=7 entry expires and the next access re-queries.

Output.

rerun	engine built?	query run?	why
1 (days=7)	yes	yes	cold cache
2 (days=7)	no	no	both cache hits
3 (days=30)	no	yes	new arg → data miss
4 (days=7, >5 min later)	no	yes	ttl expired

Rule of thumb. If the function returns data, decorate with st.cache_data; if it returns a thing you open once and reuse (a connection, an engine, a client, a model), decorate with st.cache_resource.

Streamlit interview question on choosing a cache decorator

Question. An engineer decorates their get_connection() function with @st.cache_data and their load_orders() DataFrame function with @st.cache_resource. The app is slow and occasionally shows another user's filtered data. Which decorators are swapped, and what breaks in each case?

Solution Using cache_resource for the connection and cache_data for the DataFrame

Code.

import streamlit as st
import pandas as pd

# WRONG:
# @st.cache_data           -> tries to pickle a live connection (fails / re-opens)
# def get_connection(): ...
# @st.cache_resource       -> shares ONE DataFrame object; mutations leak across users
# def load_orders(): ...

# RIGHT:
@st.cache_resource                      # the connection is a shared singleton
def get_connection():
    return open_warehouse_connection()

@st.cache_data(ttl=600)                 # each caller gets a fresh copy of the data
def load_orders(region: str) -> pd.DataFrame:
    conn = get_connection()
    return pd.read_sql(f"SELECT * FROM orders WHERE region = '{region}'", conn)

Step-by-step trace.

symptom	wrong decorator	what actually happens
app is slow	`get_connection` under `cache_data`	connection is not a picklable value; caching misbehaves, so it re-opens each rerun
user B sees user A's data	`load_orders` under `cache_resource`	the DataFrame is a shared singleton — no copy — so B reads A's filtered frame
fixed: fast	`get_connection` under `cache_resource`	one connection, shared, reused every rerun
fixed: isolated	`load_orders` under `cache_data`	each region's result is a separate copy, keyed on `region`

st.cache_data is designed for serializable return values and hands back a copy; a live DB connection is neither serializable nor safe to copy, so it belongs in cache_resource.
st.cache_resource shares one object across all sessions; a DataFrame stored there is mutated in place by whoever touches it, leaking one user's view into another's.
Swapping them restores both properties: the connection is opened once and shared, and each query result is an isolated, argument-keyed copy.
Adding ttl=600 to load_orders bounds staleness without changing correctness.

Output:

function	correct decorator	guarantee
`get_connection`	`st.cache_resource`	one shared, reused connection
`load_orders`	`st.cache_data`	per-arg copy, no cross-user leak

Why this works — concept by concept:

cache_data semantics — memoizes a function's return value keyed on its arguments and returns a fresh copy each call, which is exactly right for query results and computed frames.
cache_resource semantics — caches a single global object with no copy, which is exactly right for connections and models that are expensive to build and meant to be shared.
Copy vs share — the copy in cache_data is what isolates users; the shared instance in cache_resource is what makes a connection reusable — mixing them swaps safety for a leak and speed for a stall.
TTL invalidation — a time-to-live bounds staleness so a dashboard refreshes on its own, and .clear() lets a write path evict immediately.
Cost — a cache hit is O(1) plus a hash of the arguments, turning an O(query) round trip on every rerun into one round trip per distinct argument set.

DataFrames
Topic — dataframe-basics
DataFrame caching and memoization problems

Practice →

Analysis Topic — data-analysis Query-and-aggregate analysis problems

Practice →

4. Displaying data & connecting to warehouses

From DataFrame to dashboard is one call — `st.dataframe`, the built-in charts, and `st.connection` for the warehouse

Once you have a DataFrame, Streamlit gives you display primitives that need no plotting or web code, and a first-class way to get that DataFrame from a warehouse. The framing an interviewer wants: you render data with st.dataframe and the st.*_chart family, and you source it with st.connection, whose conn.query(...) caches results for you. No ORM, no cursor management, no manual connection pooling.

Display primitives, by intent.

st.dataframe(df) — interactive. A sortable, scrollable, searchable grid. Style it with column_config (format a number as currency, render a URL as a link, show a progress bar in a cell) and enable row selection when you need the user to pick rows.
st.data_editor(df) — editable. The same grid, but the user can edit cells; it returns the edited DataFrame, which is how you build a lightweight config or mapping editor.
st.table(df) — static. Renders the whole frame at once, no interactivity — good for small, fixed summaries.
st.metric(label, value, delta) — a KPI tile. A big number with an optional up/down delta; the building block of the header row on any dashboard.

Built-in charts, zero imports.

st.line_chart / st.area_chart / st.bar_chart / st.scatter_chart. Pass a DataFrame plus x= and y= (and optional color= to split series) and Streamlit draws an interactive chart with no Matplotlib or Plotly import. Ideal for latency-over-time and volume-by-pipeline views.
Escape hatch when you need it. For full control, st.plotly_chart(fig), st.altair_chart(chart), and st.pyplot(fig) accept figures from those libraries — you drop down only when the built-ins are not enough.

Connecting to a warehouse with st.connection.

One call, configured by secrets. conn = st.connection("warehouse", type="sql") reads its URL/credentials from .streamlit/secrets.toml under [connections.warehouse], so no credentials live in code.
conn.query(sql, ttl=...) caches results. The SQLConnection.query method runs the SQL and memoizes the DataFrame with a built-in TTL — it is st.cache_data under the hood, so you get caching without decorating anything.
Typed connections. type="sql" covers any SQLAlchemy URL (Postgres, MySQL, DuckDB, SQLite); type="snowflake" uses the Snowflake connector; BigQuery is reachable via the SQLAlchemy dialect or a cache_resource client. Parameterise with params= — never f-string user input into SQL.

Worked example — query the warehouse and render three ways

Detailed explanation. The bread-and-butter dashboard cell queries the warehouse once through st.connection, then feeds the single result DataFrame into a metric, a chart, and a table. One query, three renders — and the query is cached, so reruns are free until the TTL lapses.

Question. Using st.connection, pull daily loaded-row counts for the last 14 days and show a total-rows metric, a line chart of the trend, and the raw table.

Input. A daily_loads table with day and rows_loaded columns.

Code.

import streamlit as st

conn = st.connection("warehouse", type="sql")

df = conn.query(
    """
    SELECT day, rows_loaded
    FROM daily_loads
    WHERE day > current_date - 14
    ORDER BY day
    """,
    ttl=600,                            # cache the result for 10 minutes
)

st.metric("Rows loaded (14d)", f"{df['rows_loaded'].sum():,}")
st.line_chart(df, x="day", y="rows_loaded")
st.dataframe(df, use_container_width=True)

Step-by-step explanation. st.connection("warehouse", type="sql") builds (or reuses) a connection from the [connections.warehouse] block in secrets.toml. conn.query(..., ttl=600) executes the SQL and caches the DataFrame for ten minutes, so any rerun within that window skips the round trip. The one df then drives three widgets: st.metric shows the summed total with a thousands separator, st.line_chart plots rows_loaded against day, and st.dataframe renders the sortable grid. Nothing here imports a database driver or a charting library directly.

Output.

widget	shows
`st.metric`	"Rows loaded (14d) — 128,400"
`st.line_chart`	14-point trend line of daily volume
`st.dataframe`	sortable 14-row table

Rule of thumb. Query once with conn.query(..., ttl=...) and reuse the one DataFrame for every widget; do not run a separate query per chart, and never build SQL by concatenating user input — pass params=.

Streamlit interview question on warehouse connections

Question. A dashboard calls pd.read_sql(build_sql(region), psycopg2.connect(DSN)) at the top of the script. It re-opens a connection and re-queries on every slider move, and the DSN is hard-coded. Rewrite it the Streamlit way so the connection is reused, the query is cached per region, and credentials are out of the code.

Solution Using st.connection with a parameterised, cached query

Code.

import streamlit as st

# BEFORE: new connection + query on every rerun, credentials in code, SQL injection risk
# conn = psycopg2.connect("host=... user=... password=...")
# df = pd.read_sql(f"SELECT * FROM runs WHERE region = '{region}'", conn)

conn = st.connection("warehouse", type="sql")   # credentials from secrets.toml

region = st.selectbox("Region", ["us", "eu", "apac"])

df = conn.query(
    "SELECT pipeline, status, rows_loaded FROM runs WHERE region = :region",
    params={"region": region},          # bound param, not string interpolation
    ttl=600,                             # cache per region for 10 minutes
)

st.dataframe(df, use_container_width=True)

Step-by-step trace.

interaction	region	connection	query executed?
load	us	opened once, cached	yes (cold)
pick eu	eu	reused	yes (new param)
pick us again	us	reused	no (cache hit)
pick us, 11 min later	us	reused	yes (ttl expired)

st.connection(...) builds the connection once and caches it (via cache_resource internally), so no rerun re-opens it.
Credentials move to .streamlit/secrets.toml under [connections.warehouse], so the DSN is out of source control.
conn.query(sql, params={"region": region}, ttl=600) binds region as a parameter — closing the SQL-injection hole — and memoizes the result per region for ten minutes.
Re-selecting a region within the TTL is a cache hit, so the warehouse is queried once per distinct region per ten-minute window instead of once per interaction.

Output:

property	before	after
connections opened	one per rerun	one, reused
queries per repeat interaction	one each time	one per region per TTL
credentials	hard-coded	in `secrets.toml`
injection risk	yes (f-string)	no (bound param)

Why this works — concept by concept:

st.connection — wraps connection construction in a cached factory, so the expensive open happens once and every rerun reuses the same handle.
Cached query — conn.query(..., ttl=...) is cache_data under the hood, keying the DataFrame on the SQL and params, so identical interactions are free.
Bound parameters — passing params= sends values separately from the SQL text, which both enables correct cache keying and eliminates injection.
Secrets separation — reading credentials from secrets.toml keeps them out of the repo and lets the same code run against dev and prod by swapping the file.
Cost — the query cost collapses from O(interactions) round trips to O(distinct params per TTL), the single biggest performance win in a Streamlit data app.

Analysis
Topic — data-analysis
Query, aggregate and visualize problems

Practice →

SLA Topic — sla-monitoring Freshness and SLA-metric problems

Practice →

5. Building a pipeline-health / SLA dashboard

Metrics, a freshness table, a backfill button — assembling the primitives into an SLA dashboard with forms and fragments

Everything so far converges here: a real internal tool a data team uses. A pipeline-health dashboard shows KPI metrics up top, a freshness table with SLA breaches flagged, and controls that do something — a form that triggers a backfill. Two more primitives finish it: st.form to batch inputs so the script does not rerun on every keystroke, and st.fragment to refresh part of the page without re-running the whole thing.

The layout primitives.

st.columns for the KPI row. c1, c2, c3 = st.columns(3) then c1.metric(...) places metrics side by side. st.metric("SLA met", "96%", delta="-2%") shows the value and a coloured delta.
st.dataframe with column_config for the freshness table. Format last_run as a timestamp, render minutes_late with a progress bar, and use row styling (via a Styler or a status column) to make breaches obvious.
st.sidebar and st.tabs for structure. Filters go in st.sidebar; separate an "Overview" tab from a "Backfill" tab with st.tabs([...]).

Forms and buttons — controls that act.

st.form batches input. Widgets inside with st.form("backfill"): do not trigger a rerun individually; the script reruns only when the user clicks st.form_submit_button(...). This is essential for a backfill panel with several inputs — you do not want a rerun (or a job) fired on every field change.
The submit button gates the action. if st.form_submit_button("Run backfill"): is True on the submitting rerun, so you kick the job inside that block. Guard destructive actions with a confirmation checkbox or a typed-in pipeline name.
Long jobs get status feedback. Wrap the work in with st.status("Backfilling...", expanded=True) as s: and update it, or use st.spinner / st.progress, so the user sees progress instead of a frozen page.

Auto-refresh without re-running everything.

@st.fragment for partial reruns. Decorate a function with @st.fragment and only that block reruns when its own widgets change — the rest of the page stays put. This keeps a heavy dashboard responsive when one control should not re-query everything.
run_every for a live tile. @st.fragment(run_every="30s") re-runs that fragment on a timer, so a freshness table or a "last updated" clock refreshes itself without a full-page reload.
st.rerun() to force a cycle. After a backfill writes new rows, call load.clear() then st.rerun() to evict the cache and redraw with fresh data immediately, rather than waiting for the TTL.

Deployment, briefly.

streamlit run app.py locally; behind a reverse proxy or on Streamlit Community Cloud (point it at a GitHub repo, set secrets in the dashboard) for zero-ops sharing.
Docker for self-hosting: base image, pip install -r requirements.txt, EXPOSE 8501, CMD ["streamlit", "run", "app.py"]. Put credentials in secrets.toml or environment variables, never in the image.

Worked example — a freshness table that flags SLA breaches

Detailed explanation. The heart of a pipeline-health dashboard is a freshness table: for each pipeline, how long since its last successful run, and is that within its SLA. The logic is a per-row comparison of lateness against a threshold, rendered so breaches jump out. This is the view an on-call engineer opens first.

Question. Given a DataFrame of pipelines with minutes_since_run and an sla_minutes threshold, compute a status of OK or BREACH and show a metric for the breach count plus the flagged table.

Input.

pipeline	minutes_since_run	sla_minutes
orders_el	25	60
events_el	130	90
dim_refresh	40	45

Code.

import streamlit as st
import pandas as pd

df = load_freshness()                   # cached conn.query(...) from earlier

df["status"] = df.apply(
    lambda r: "BREACH" if r.minutes_since_run > r.sla_minutes else "OK",
    axis=1,
)
breaches = int((df["status"] == "BREACH").sum())

st.metric("SLA breaches", breaches, delta=None)
st.dataframe(
    df.style.apply(
        lambda r: ["background-color: #fee2e2" if r.status == "BREACH" else "" for _ in r],
        axis=1,
    ),
    use_container_width=True,
)

Step-by-step explanation. load_freshness() returns the cached freshness DataFrame. The apply compares minutes_since_run to each pipeline's own sla_minutes and writes BREACH or OK into a new status column. breaches counts the flagged rows for the KPI tile. st.metric surfaces that count up top, and st.dataframe renders the table with a row-level Styler that tints breach rows red, so the one late pipeline is impossible to miss.

Output.

pipeline	minutes_since_run	sla_minutes	status
orders_el	25	60	OK
events_el	130	90	BREACH
dim_refresh	40	45	OK

The metric reads SLA breaches — 1, and the events_el row is highlighted.

Rule of thumb. Compare lateness against a per-pipeline SLA, not one global threshold; render the breach as both a headline metric and a highlighted row so the summary and the detail agree.

Streamlit interview question on triggering a backfill safely

Question. Product wants a button on the dashboard that backfills a chosen pipeline over a chosen date range. A naive if st.button("Backfill"): run_backfill(...) outside a form fires on stray reruns and offers no confirmation. Build a safe backfill control that only runs on explicit submit and refreshes the data afterward.

Solution Using st.form, a submit button, and a post-run rerun

Code.

import streamlit as st

with st.form("backfill"):
    pipeline = st.selectbox("Pipeline", ["orders_el", "events_el", "dim_refresh"])
    start, end = st.date_input("Date range", [])
    confirm = st.checkbox("I understand this reprocesses data")
    submitted = st.form_submit_button("Run backfill")

if submitted and confirm:
    with st.status(f"Backfilling {pipeline}...", expanded=True) as status:
        run_backfill(pipeline, start, end)      # your job trigger
        status.update(label="Backfill complete", state="complete")
    load_freshness.clear()                       # evict stale cached data
    st.rerun()                                   # redraw with fresh numbers
elif submitted and not confirm:
    st.warning("Tick the confirmation box before running a backfill.")

Step-by-step trace.

user action	reruns fired	backfill runs?
change pipeline dropdown (in form)	0	no — form defers rerun
pick a date range (in form)	0	no — still deferred
click "Run backfill", confirm unticked	1 (on submit)	no — warning shown
tick confirm, click "Run backfill"	1 (on submit)	yes, then cache cleared + rerun

Widgets inside with st.form(...) do not trigger a rerun on change, so choosing a pipeline and a date range fires no premature runs and no stray job.
st.form_submit_button reruns the script once and returns True only on that submitting run, so the job can only start from an explicit click.
The and confirm gate requires the checkbox, turning an accidental click into a harmless warning instead of a reprocessing job.
After the job, load_freshness.clear() evicts the cached freshness data and st.rerun() forces an immediate redraw, so the dashboard reflects the backfill without waiting for the TTL.

Output:

guarantee	mechanism
no run on field edits	`st.form` defers reruns to submit
no run without confirmation	`and confirm` gate
fresh data after run	`.clear()` + `st.rerun()`
visible progress	`st.status(...)`

Why this works — concept by concept:

st.form batching — grouping inputs in a form suppresses the per-widget rerun, so multi-field controls neither thrash the script nor trigger side effects until submit.
Submit-gated action — st.form_submit_button is the only trigger, and its True is confined to the submitting rerun, so a job starts exactly once per intentional click.
Confirmation gate — requiring a checkbox before a destructive action converts the transient button click into a two-step, mistake-resistant flow.
Cache clear + rerun — evicting the memoized query and forcing a rerun makes the freshly written rows appear immediately instead of on the next TTL boundary.
Cost — the control adds O(1) work and one extra rerun per backfill, a negligible price for making a data-mutating action deliberate and its result immediately visible.

SLA
Topic — sla-monitoring
SLA-breach and freshness-dashboard problems

Practice →

Pipelines Topic — pipelines Backfill and pipeline-control problems

Practice →

Cheat sheet — Streamlit recipes

Minimal app.

import streamlit as st
st.title("Hello")
st.write("It renders.")     # run with: streamlit run app.py

Widget + session_state counter.

if "n" not in st.session_state:
    st.session_state.n = 0
st.button("＋", on_click=lambda: st.session_state.__setitem__("n", st.session_state.n + 1))
st.write(st.session_state.n)

cache_data query helper.

@st.cache_data(ttl=300)
def load(days: int):
    return conn.query("SELECT * FROM runs WHERE day > current_date - :d",
                      params={"d": days})

st.connection SQL query.

conn = st.connection("warehouse", type="sql")   # creds in .streamlit/secrets.toml
df = conn.query("SELECT * FROM daily_loads", ttl=600)
st.dataframe(df)

Form with submit (no rerun until submit).

with st.form("f"):
    region = st.selectbox("Region", ["us", "eu"])
    go = st.form_submit_button("Load")
if go:
    st.dataframe(load_region(region))

Auto-refreshing fragment.

@st.fragment(run_every="30s")
def freshness_tile():
    st.metric("Last updated", now_str())   # only this block reruns on the timer
freshness_tile()

Decorator / primitive picker.

Situation	Use
Cache a DataFrame / query result	`st.cache_data(ttl=...)`
Cache a connection / engine / model	`st.cache_resource`
Keep a value across reruns	`st.session_state`
Batch several inputs, act on submit	`st.form` + `st.form_submit_button`
Refresh part of the page on a timer	`@st.fragment(run_every=...)`

Frequently asked questions

What is Streamlit and why do data engineers use it?

Streamlit is an open-source Python framework that turns a plain script into an interactive web app — you write import streamlit as st and a few st. calls, run streamlit run app.py, and get a shareable URL. Data engineers use it because it removes the frontend entirely: a pipeline-health dashboard, an SLA monitor, or a backfill console that would take days in Flask plus React takes an afternoon in one Python file. It is built for internal tools and dashboards, not public consumer products.

How does Streamlit's rerun model work?

Streamlit re-executes your entire script from top to bottom every time the user interacts with any widget. Widgets are function calls that return their current value on each run, so you use the returned value inline rather than attaching event handlers. The consequence is that ordinary variables reset on every rerun, which is why persistent values must live in st.session_state and expensive work must be wrapped in a cache decorator.

What is the difference between st.cache_data and st.cache_resource?

st.cache_data memoizes a function's return value — DataFrames, query results, anything serializable — keyed on the function's arguments, and it hands each caller a fresh copy so users cannot corrupt each other's data. st.cache_resource caches a single global object such as a database connection, engine, or ML model and gives every caller the same shared instance, with no copy. Use cache_data for data you compute or fetch and cache_resource for things you open once and reuse.

How do I keep values between reruns in Streamlit?

Store them in st.session_state, a dictionary-like object scoped to the browser session that survives reruns. Seed a key once with a guard — if "count" not in st.session_state: st.session_state.count = 0 — so the rerun does not reset it, and mutate it inside a widget callback such as on_click. Binding a widget with key="..." also mirrors its value into st.session_state automatically.

How does Streamlit connect to a data warehouse?

Use st.connection("name", type="sql"), which reads credentials from .streamlit/secrets.toml under [connections.name] so nothing sensitive lives in code. Call conn.query(sql, params={...}, ttl=600) to run a parameterised query whose DataFrame result is cached for the TTL — it uses st.cache_data internally, so repeated interactions do not re-hit the warehouse. type="sql" covers any SQLAlchemy URL (Postgres, MySQL, DuckDB), and type="snowflake" targets Snowflake directly.

How do I build an auto-refreshing pipeline dashboard in Streamlit?

Lay out KPI tiles with st.columns and st.metric, render a freshness table with st.dataframe (highlighting SLA breaches), and put action controls like a backfill trigger inside st.form so they only fire on submit. For live updates, decorate a section with @st.fragment(run_every="30s") so just that block re-runs on a timer instead of the whole page, and after a data-mutating action call your cached loader's .clear() followed by st.rerun() to redraw with fresh numbers immediately.

Practice on PipeCode

Pipecode.ai is Leetcode for Data Engineering — every Streamlit idea above, from the DataFrame that drives `st.dataframe` and `st.line_chart` to the freshness-and-breach logic behind an SLA table, rests on the data-shaping skills you can drill as graded problems. PipeCode pairs each reading with 450+ DE-focused problems and a real-time scoring engine, so your answer to "how would you compute per-pipeline SLA status from a freshness table?" holds up under a senior interviewer's depth probes.

Practice data-analysis problems now →
SLA-monitoring drills →

Factless Fact Tables: Modeling Events, Coverage & Eligibility Without Measures

Gowtham Potureddi — Thu, 24 Sep 2026 17:35:55 +0000

factless fact table is the one design pattern that breaks the rule every beginner learns first — that a fact table is where the numbers live. A factless fact table has no numbers. It is a fact table made almost entirely of foreign keys pointing at dimensions, with not a single amount, quantity, or price column to sum. And yet it is one of the most useful shapes in a star schema, because the presence of the row is itself the fact. A row exists exactly when a student attended a class, when a product was on promotion, when a user viewed a page — and the metric you want is simply how many such rows there are, or, more subtly, which combinations have no row at all.

That second question is the one factless fact tables answer better than any other structure: what did not happen. A normal sales fact only records the sales that occurred, so it can never tell you which promoted products failed to sell — those rows were never written. This guide walks through the four ideas an interviewer will actually probe about factless fact tables — why a measureless fact table is legitimate, the event-tracking family you count with COUNT(*), the coverage / eligibility family that records the universe of what could occur, and the anti-join technique that turns the two into an answer for absence — and pairs each with a Solution-Tail interview answer: code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the factless fact practice library →, rehearse the coverage-and-absence patterns on the coverage fact practice set →, and lock in your grain decisions on the fact grain practice set →.

On this page

Why a fact table can have no measures at all
Event-tracking factless facts — the row is the event
Coverage & eligibility factless facts — what could happen
Answering "what did NOT happen" with anti-joins
Grain, keys & design pitfalls for factless facts
Cheat sheet — factless fact recipes
Frequently asked questions
Practice on PipeCode

1. Why a fact table can have no measures at all

A factless fact table is a fact table whose only measure is its own existence — that one idea explains every use of the pattern

The one-sentence invariant: a factless fact table records that an event or a condition occurred by writing a row of foreign keys, and the measure you want is derived from the rows themselves rather than stored in a column. Everything about the pattern follows from that. There is no amount, no quantity, no price — there is a set of dimension keys, maybe a date key, maybe a degenerate id, and the fact is that this particular combination of keys came together at all.

Why "factless" is not a contradiction.

The row is the fact. In a sales fact you store amount; the row plus the number is the fact. In a factless fact the row alone is the fact — one row means "this happened" or "this was true," and the numeric answer is COUNT(*) over those rows.
The measure is implicit, not absent. "Factless" is a slightly misleading name: there is a fact — an occurrence or a condition — it simply is not pre-aggregated into a stored number. The measure lives in the cardinality of the table.
It still obeys star-schema rules. A factless fact table sits at the centre of a star, joins to conformed dimensions on surrogate keys, and has a clearly declared grain. It is a fact table in every structural sense except the measure column.

The two families you must be able to name.

Event tracking. One row every time something happens: a student attends a class, a user views a page, a visitor logs in, a patient is admitted. You answer questions by counting rows and counting distinct dimension members.
Coverage / condition (eligibility). One row for every combination that is eligible or in effect, whether or not anything happened: which products are on promotion this week, which students are enrolled in which course, which sales reps cover which territory. This records the universe of the possible.
The two are complementary. Event tables tell you what occurred; coverage tables tell you what could have occurred. You need both to ask "what was possible but never happened."

What interviewers listen for.

Do you say "the existence of the row is the measure" in the first sentence? — senior signal.
Can you name both families — event tracking and coverage — unprompted? — required framing.
Do you reach for a coverage table specifically to answer "what did not happen", and explain why the event fact alone cannot? — the whole point.
Do you resist the temptation to add a dummy count = 1 column, and explain that COUNT(*) already gives it? — a common anti-pattern they probe.

Worked example — the same event, with and without a measure

Detailed explanation. The quickest way to feel why a factless design is legitimate is to put a conventional sales fact next to an attendance fact. The sales fact has a measure to sum; the attendance fact has nothing to sum, yet both answer real business questions. The attendance fact answers "how many attendances" with COUNT(*) — the same answer a stored attendances = 1 column would give, minus the redundant column.

Question. Show a normal sales fact row and a factless attendance fact row side by side, and state how you get a number out of each.

Input.

fact table	keys	measure column
`fct_sales`	product_key, date_key, store_key	`amount` = 42.50
`fct_attendance`	student_key, class_key, date_key	(none)

Code.

-- measured fact: sum the stored measure
SELECT date_key, SUM(amount) AS revenue
FROM   fct_sales
GROUP  BY date_key;

-- factless fact: the metric is the row count itself
SELECT class_key, COUNT(*) AS attendances
FROM   fct_attendance
GROUP  BY class_key;

Step-by-step explanation. fct_sales carries amount, so the metric is SUM(amount) — the number was written at load time. fct_attendance carries no numeric column, so there is nothing to SUM; instead each row is one attendance, and COUNT(*) per class_key returns how many students attended. Both queries are ordinary star-schema aggregations — the only difference is that the factless query counts rows where the measured query sums a column. Adding an attendances INTEGER DEFAULT 1 column to the factless table would let you write SUM(attendances), but it would always equal COUNT(*), so it stores nothing new.

Output.

query	grouped by	result
`SUM(amount)` on `fct_sales`	date_key	revenue per day
`COUNT(*)` on `fct_attendance`	class_key	attendances per class

Rule of thumb. If the business question is "how many times did X happen," you do not need a measure column — you need a row per occurrence and COUNT(*). Reach for a measure only when there is a genuine number to add up.

2. Event-tracking factless facts — the row is the event

An event factless fact stores one row per occurrence, joins only to dimensions, and is queried with COUNT(*) and COUNT(DISTINCT)

The event-tracking family is the one most people meet first. The grain is "one row each time the event happens," the columns are the foreign keys that describe who / what / when, and the answer to almost every question is a count. Attendance is the textbook example: a row is written the moment a student attends a class on a date, and the table never stores a quantity because the quantity is always one.

Declaring the grain.

Grain is a single sentence. "One row per student per class per day attended." Write that sentence before you write the DDL; every column exists to serve it, and no column may be finer or coarser than it.
Keys describe the event. student_key, class_key, date_key — each a surrogate key into a conformed dimension. Together they answer who attended what and when.
No numeric measure. There is deliberately nothing to sum. The event's "value" is that it happened.

The metrics you actually compute.

COUNT(*) — how many events. Attendances per class, page views per article, logins per day. This is the additive workhorse.
COUNT(DISTINCT key) — how many distinct members. Distinct students who attended (headcount, not visit-count), unique visitors, distinct products viewed. This is non-additive across some dimensions, which matters when you roll up.
Ratios across dimensions. Attendance rate = distinct attendees ÷ enrolled; both numerator and denominator often come from factless tables (the second from a coverage table, section 3).

Degenerate dimensions on event facts.

The event id has no dimension. A session_id, page_view_id, or admission_no that identifies the specific event but has no attributes of its own is stored in the fact table as a degenerate dimension — a key with no dimension to join to.
It enables COUNT(DISTINCT event_id). When a single logical event can emit multiple rows, the degenerate id lets you count events, not rows.
It is not a measure. A degenerate dimension is still a key; it never gets summed.

Worked example — a student-attendance factless fact

Detailed explanation. The canonical event factless fact is class attendance. Each time a student shows up to a class on a given day, one row is written with three foreign keys and nothing else. The table can grow to billions of rows and still stores not one number, because every question about it is a count.

Question. Model an attendance factless fact and count (a) total attendances per class and (b) distinct students who attended each class.

Input.

student_key	class_key	date_key
1	100	20260901
2	100	20260901
1	100	20260902
3	101	20260901

Code.

CREATE TABLE fct_attendance (
    student_key BIGINT NOT NULL REFERENCES dim_student(student_key),
    class_key   BIGINT NOT NULL REFERENCES dim_class(class_key),
    date_key    INT    NOT NULL REFERENCES dim_date(date_key)
    -- no measure column: the row is the attendance
);

SELECT class_key,
       COUNT(*)                     AS attendances,
       COUNT(DISTINCT student_key)  AS distinct_students
FROM   fct_attendance
GROUP  BY class_key;

Step-by-step explanation. The table holds only foreign keys, so a row is the atomic statement "this student attended this class on this date." For class_key = 100 there are three rows (student 1 twice on different days, student 2 once), so COUNT(*) = 3 attendances. But only two distinct students attended, so COUNT(DISTINCT student_key) = 2. The two numbers answer different questions — visit-count vs headcount — and both fall out of the same keys-only table.

Output.

class_key	attendances	distinct_students
100	3	2
101	1	1

Rule of thumb. For an event fact, keep COUNT(*) (how many times) and COUNT(DISTINCT member) (how many members) clearly separated in your head — they diverge exactly when a member can generate more than one event.

SQL interview question on counting events in a factless fact

Question. You have the fct_attendance table above and a dim_date with a week column. An interviewer asks for weekly attendances and weekly distinct attendees per class, sorted by class and week. Write the query and explain why the two counts can differ within a week.

Solution Using COUNT(*) versus COUNT(DISTINCT) over a joined date dimension

Code.

SELECT d.week,
       f.class_key,
       COUNT(*)                       AS attendances,
       COUNT(DISTINCT f.student_key)  AS distinct_attendees
FROM   fct_attendance f
JOIN   dim_date d ON d.date_key = f.date_key
GROUP  BY d.week, f.class_key
ORDER  BY f.class_key, d.week;

Step-by-step trace.

student_key	class_key	date_key	week
1	100	20260901	2026-W36
2	100	20260901	2026-W36
1	100	20260902	2026-W36
3	101	20260901	2026-W36

The join to dim_date attaches a week label to each attendance row without changing the grain — it is a lookup, one date key maps to one week.
GROUP BY week, class_key buckets the rows; class 100 in week W36 has three rows.
COUNT(*) returns 3 attendances for that bucket — student 1 attended on two days, counted twice, which is correct for "visits."
COUNT(DISTINCT student_key) returns 2 — student 1's two visits collapse to one person, which is correct for "unique attendees."

Output:

week	class_key	attendances	distinct_attendees
2026-W36	100	3	2
2026-W36	101	1	1

Why this works — concept by concept:

Row-as-event grain — because one row means one attendance, COUNT(*) is a fully additive measure you can roll up across any dimension (day → week → term) without a stored column.
COUNT(DISTINCT) non-additivity — distinct attendees cannot be summed across weeks (the same student in two weeks would be double-counted), so it must be recomputed at each grain, not aggregated from a lower level.
Dimension join preserves grain — joining dim_date to label the week is a many-to-one lookup, so it neither adds nor drops attendance rows; the counts stay correct.
Two questions, one table — visit-count and headcount are different business metrics that both come from the same factless table, which is exactly why the pattern is so economical.
Cost — COUNT(*) is O(rows) on a scan or index; COUNT(DISTINCT) is O(rows) plus a hash/sort on the distinct key, measurably heavier at scale.

SQL
Topic — factless-fact
Factless fact event-counting problems

Practice →

Modeling Topic — event-log Event-log and occurrence-tracking problems

Practice →

3. Coverage & eligibility factless facts — what could happen

A coverage factless fact enumerates the eligible combinations — the universe of the possible — so you can measure the gap between could and did

The second family is the one interviewers use to separate people who memorised "factless = attendance" from people who actually understand the pattern. A coverage (or condition, or eligibility) factless fact does not record events. It records which combinations are in effect: which products are on promotion in which store this week, which students are enrolled in which course this term, which insurance plan covers which procedure. Nothing has to happen for a coverage row to exist — the row asserts a condition, not an occurrence.

Why coverage tables exist at all.

Event facts only record what happened. A sales fact has rows only for products that sold. It is structurally incapable of telling you which promoted products sold nothing — those rows were never written.
Coverage records the denominator. To ask "what fraction of promoted products sold," you need the full set of promoted products (the denominator) somewhere. That set is the coverage table.
The classic case is promotion coverage. For every store / product / promotion / day that a promotion is in effect, write one coverage row — regardless of whether that product sold. Now the promoted-but-unsold products are answerable.

The grain of a coverage table.

One row per eligible combination per time bucket. "One row per product per store per promotion per day the promotion runs." The combination, not an event, defines the row.
It can be large. Coverage explodes combinatorially — every eligible product in every covered store for every active day — so it is often kept at a coarser grain (per week, per promotion period) or as date-range rows with valid_from / valid_to.
Snapshot vs range. You can materialise coverage as a daily snapshot (one row per active day, easy to join to a date-grain event fact) or as effective-dated ranges (compact, but you join on BETWEEN). Snapshots trade storage for simpler joins.

What coverage tables model in the wild.

Eligibility. Which customers are eligible for an offer, which employees are eligible for a benefit — the set you will later compare against who redeemed or enrolled.
Assignment. Which sales rep covers which account, which nurse is rostered to which ward — a many-to-many relationship captured as a factless fact, which doubles as a bridge.
Applicability. Which tax rule applies to which product, which SLA applies to which ticket class — conditions in force that downstream facts are checked against.

Worked example — a promotion-coverage factless fact

Detailed explanation. The archetypal coverage table is promotion coverage. A retailer runs a promotion; some products are included, in some stores, for some dates. The coverage table gets one row for every product-store-date the promotion is in effect — even a product that never rings up a single sale. That is the whole point: coverage captures the products that could have sold under the promotion.

Question. Model a promotion-coverage factless fact and count how many distinct products were on promotion in each store, independent of whether they sold.

Input.

product_key	store_key	promo_key	date_key
500	1	9	20260901
501	1	9	20260901
502	1	9	20260901
500	2	9	20260901

Code.

CREATE TABLE fct_promo_coverage (
    product_key BIGINT NOT NULL REFERENCES dim_product(product_key),
    store_key   BIGINT NOT NULL REFERENCES dim_store(store_key),
    promo_key   BIGINT NOT NULL REFERENCES dim_promotion(promo_key),
    date_key    INT    NOT NULL REFERENCES dim_date(date_key)
    -- factless: presence means "this product was on promotion here today"
);

SELECT store_key,
       COUNT(DISTINCT product_key) AS products_on_promo
FROM   fct_promo_coverage
GROUP  BY store_key;

Step-by-step explanation. Each row asserts that a product was on promotion in a store on a date — a condition, not a sale. Store 1 has three distinct promoted products (500, 501, 502); store 2 has one (500). COUNT(DISTINCT product_key) per store answers "how wide was the promotion here," a number the sales fact could never give you because the sales fact only knows about products that actually sold.

Output.

store_key	products_on_promo
1	3
2	1

Rule of thumb. If a question contains the word "eligible," "on offer," "covered," "assigned," or "enrolled" — anything about what is in effect rather than what occurred — you are looking at a coverage factless fact, not an event one.

SQL interview question on modeling eligibility as coverage

Question. A university needs to answer "how many students are enrolled in each course this term" and, later, "which enrolled students never attended." An interviewer asks how you model enrollment and why it must be a separate table from attendance. Give the coverage table and the enrollment count query.

Solution Using an enrollment coverage fact separate from the attendance event fact

Code.

CREATE TABLE fct_enrollment_coverage (
    student_key BIGINT NOT NULL REFERENCES dim_student(student_key),
    course_key  BIGINT NOT NULL REFERENCES dim_course(course_key),
    term_key    INT    NOT NULL REFERENCES dim_term(term_key)
    -- one row per student enrolled in a course for a term (a condition)
);

SELECT course_key,
       COUNT(DISTINCT student_key) AS enrolled_students
FROM   fct_enrollment_coverage
GROUP  BY course_key;

Step-by-step trace.

student_key	course_key	term_key	meaning
1	100	202603	enrolled (may or may not attend)
2	100	202603	enrolled
3	100	202603	enrolled
3	101	202603	enrolled

Enrollment is a condition — the student is registered for the course this term whether or not they ever walk in — so it is coverage, not an event.
Attendance is a separate event fact (fct_attendance) written only when a student shows up, so it cannot represent enrolled-but-absent students.
COUNT(DISTINCT student_key) per course over the coverage table gives the enrolled headcount — course 100 has three enrolled students.
Keeping enrollment and attendance in two tables is what later makes "enrolled but never attended" answerable: it is coverage minus events.

Output:

course_key	enrolled_students
100	3
101	1

Why this works — concept by concept:

Coverage grain — one row per enrolled student-course-term captures the eligible universe; the row asserts a standing condition, so it exists independently of any attendance event.
Denominator capture — enrollment is the denominator for attendance-rate and no-show metrics; without a coverage table there is no place that stores the students who could have attended.
Separation of concerns — modeling enrollment and attendance as distinct factless facts keeps each at its own honest grain and prevents you from faking absence with outer joins on a single table.
Reusable universe — the same coverage table feeds many questions (enrolled counts, no-show lists, capacity utilisation), which is why it earns its own place in the schema.
Cost — the coverage table is O(eligible combinations), which is bounded and usually far smaller than the event stream, so it is cheap to store and index.

Modeling
Topic — coverage-fact
Coverage and eligibility modeling problems

Practice →

SQL Topic — factless-fact Factless fact table design problems

Practice →

4. Answering "what did NOT happen" with anti-joins

Coverage LEFT JOIN events, keep the NULLs — the anti-join is how a factless design turns absence into a queryable result

Here is the payoff that makes the whole pattern worth learning. Business people constantly ask negative questions: which promoted products sold nothing, which enrolled students never attended, which eligible customers never redeemed. An event fact alone cannot answer any of them, because absence leaves no row. The technique is always the same shape: take the coverage table (what was possible), left-join it to the event table (what happened), and keep only the rows where the event side is NULL — those are the non-events.

The anti-join, three equivalent ways.

LEFT JOIN ... WHERE right.key IS NULL. Join coverage to events on the shared keys; unmatched coverage rows get NULLs on the event side; filter to IS NULL. The most portable and most readable form.
NOT EXISTS (correlated subquery). For each coverage row, check that no matching event row exists. Often the planner's favourite — it can short-circuit on the first match and handles NULLs cleanly.
EXCEPT / MINUS. Set-difference the coverage key-set minus the event key-set. Elegant when the two sides share exactly the same key columns and you only want the keys.

The correctness traps.

NOT IN with NULLs is a landmine. WHERE key NOT IN (SELECT key FROM events) returns no rows at all if the subquery yields a single NULL, because NOT IN is three-valued. Prefer NOT EXISTS or LEFT JOIN ... IS NULL, which are NULL-safe.
Join on the complete grain. The anti-join must match on every key that defines "the same thing." Match promoted products to sales on product_key AND store_key AND date_key; drop one and you either miss non-events or invent false ones.
Align the time grain. If coverage is per-day and events are per-transaction, aggregate or key both to the same date grain before the anti-join, or a same-day sale in a different hour can be mismatched.

Why you need both tables.

Coverage = the universe. Without a coverage fact there is no set of "everything that could have happened" to subtract from.
Events = what occurred. Without the event fact there is nothing to subtract.
Absence = coverage − events. The non-event is precisely the coverage rows with no matching event; that is a set difference you cannot compute from either table alone.

Worked example — promoted products that never sold

Detailed explanation. The definitive "what did not happen" query pairs promotion coverage with a sales event fact. Coverage lists every product on promotion in a store on a day; sales lists every product that actually rang up. Anti-joining coverage to sales on the full key surfaces exactly the promoted products that sold nothing — the rows a plain sales report can never show.

Question. Given fct_promo_coverage (product/store/date on promotion) and fct_sales (product/store/date that sold), list the promoted products that had zero sales.

Input.

coverage (product, store, date)	sales (product, store, date)
500, 1, 0901	500, 1, 0901
501, 1, 0901	(no row)
502, 1, 0901	502, 1, 0901
500, 2, 0901	(no row)

Code.

SELECT c.product_key, c.store_key, c.date_key
FROM   fct_promo_coverage c
LEFT   JOIN fct_sales s
       ON  s.product_key = c.product_key
       AND s.store_key   = c.store_key
       AND s.date_key    = c.date_key
WHERE  s.product_key IS NULL;   -- promoted but never sold

Step-by-step explanation. The LEFT JOIN keeps every coverage row and attaches matching sales rows where they exist. Product 500 in store 1 sold, so it matches and is discarded by the IS NULL filter; product 502 in store 1 likewise matches and drops. Product 501 in store 1 and product 500 in store 2 have no sale on that date, so their sales columns are NULL, and the WHERE s.product_key IS NULL filter keeps exactly those two — the promoted-but-unsold products.

Output.

product_key	store_key	date_key
501	1	20260901
500	2	20260901

Rule of thumb. "What did not happen" is always coverage LEFT JOIN events ... WHERE events.key IS NULL. If you find yourself trying to answer it from the event table alone, you are missing a coverage fact.

SQL interview question on absence and the NOT IN pitfall

Question. A colleague wrote WHERE student_key NOT IN (SELECT student_key FROM fct_attendance) to find enrolled students who never attended, and it returns zero rows even though many students were absent. Explain the bug and give a correct, NULL-safe query for "enrolled but never attended," per course.

Solution Using NOT EXISTS instead of NOT IN

Code.

SELECT e.course_key, e.student_key
FROM   fct_enrollment_coverage e
WHERE  NOT EXISTS (
           SELECT 1
           FROM   fct_attendance a
           WHERE  a.student_key = e.student_key
             AND  a.class_key   = e.course_key   -- align to the same grain
       );

Step-by-step trace.

enrolled (student, course)	attended?	NOT EXISTS result
1, 100	yes	excluded
2, 100	no	kept (never attended)
3, 100	no	kept
3, 101	yes	excluded

The bug: fct_attendance.student_key contains a NULL (or the subquery can), so NOT IN evaluates to UNKNOWN for every row and the whole result collapses to zero — three-valued logic, not a data problem.
NOT EXISTS asks a yes/no question per enrolled row — "is there any attendance for this student in this course?" — and NULLs inside the subquery simply fail to match, so they do not poison the result.
The correlation matches on both keys (student_key and course/class), so it checks attendance for the right course, not attendance anywhere.
Students 2 and 3-in-course-100 have no matching attendance row, so NOT EXISTS is true and they are correctly returned as never-attended.

Output:

course_key	student_key
100	2
100	3

Why this works — concept by concept:

Anti-join semantics — "never attended" is a set difference (enrolled minus attended); NOT EXISTS expresses that difference directly and returns the coverage rows with no event match.
NULL-safety — NOT EXISTS uses two-valued existence checks, so a NULL in the attendance keys cannot wipe the result the way NOT IN's three-valued logic does.
Full-grain correlation — matching on student and course guarantees you are testing attendance for the same course the student is enrolled in, so you neither miss absentees nor flag the wrong ones.
Coverage as the driving table — the query iterates the enrollment universe and probes events, which is the only order that can surface absence; driving from events could never see a student who generated no row.
Cost — NOT EXISTS is typically an anti-semi-join, O(coverage rows) probing an index on the event keys; far cheaper and safer than materialising a NOT IN list.

SQL
Topic — factless-fact
Anti-join and "what did not happen" problems

Practice →

Modeling Topic — coverage-fact Coverage-vs-event absence problems

Practice →

5. Grain, keys & design pitfalls for factless facts

Declare the grain, count the rows, and never invent a dummy measure — the discipline that keeps a keys-only table honest

A factless fact table is simple to draw and easy to get subtly wrong. Because it has no measure to anchor it, the grain is the only thing keeping the table meaningful, and a sloppy join can silently multiply your "counts." This section is the checklist an interviewer expects you to run through when they say "design me a factless fact."

Grain first, always.

One sentence, no exceptions. State the grain — "one row per student per class per day attended" — before any DDL. Every foreign key is there to satisfy that sentence; if a column does not help identify one row, it does not belong.
Mixed grains corrupt counts. If some rows are per-day and others per-week in the same table, COUNT(*) means nothing. Keep one grain per table.
Choose event or condition, not both. A single factless table is either event-tracking or coverage. Cramming both into one table (with a type flag) destroys the clean COUNT(*) and forces filters everywhere.

Counting safely.

COUNT(*) is additive; COUNT(DISTINCT) is not. You can sum COUNT(*) up a hierarchy; you cannot sum distinct counts across buckets. Recompute distincts at each grain.
Beware join fan-out. Joining a factless fact to a dimension that is not many-to-one (e.g. a bridge, or a mis-keyed dimension with duplicates) multiplies rows and inflates COUNT(*). Verify every fact-to-dimension join is many-to-one, or count a degenerate id with COUNT(DISTINCT) to stay safe.
Degenerate ids protect event counts. When one logical event can produce several rows, store the event id in the fact and COUNT(DISTINCT event_id) so a downstream join cannot inflate the metric.

Anti-patterns to name and reject.

The constant 1 measure. Adding attendances INTEGER DEFAULT 1 so you can "SUM a fact" is redundant — COUNT(*) already returns it. It wastes space and invites bugs when someone loads a 2.
Storing derived attributes on the fact. Do not denormalise dimension attributes (student name, class title) into the factless fact; keep it keys-only and join to conformed dimensions.
Using a coverage table as an event table (or vice versa). Counting coverage rows as if they were events overstates activity; both must exist and be used for what they model.

Worked example — how a fan-out join inflates a factless count

Detailed explanation. The most common factless-fact bug is a count that comes back too high because a join multiplied the rows. Here an attendance fact is joined to a class-instructor bridge where a class has two co-instructors; the join doubles every attendance row, and a naive COUNT(*) reports twice the real attendance.

Question. An attendance fact is joined to bridge_class_instructor (a class can have several instructors) to attribute attendances to instructors. Show how COUNT(*) inflates and how to count correctly.

Input.

attendance (student, class)	class → instructors
1, 100	100 → {A, B}
2, 100	100 → {A, B}

Code.

-- WRONG: the bridge fans out each attendance into one row per instructor
SELECT COUNT(*) AS attendances          -- returns 4, not 2
FROM   fct_attendance a
JOIN   bridge_class_instructor b ON b.class_key = a.class_key;

-- RIGHT: count distinct attendance events, immune to the fan-out
SELECT COUNT(DISTINCT a.attendance_id) AS attendances   -- returns 2
FROM   fct_attendance a
JOIN   bridge_class_instructor b ON b.class_key = a.class_key;

Step-by-step explanation. The bridge maps class 100 to two instructors, so the join turns each of the two attendance rows into two rows (one per instructor) — four rows total. Plain COUNT(*) now reports 4 attendances, which is wrong: only two students attended. Counting DISTINCT attendance_id — a degenerate dimension carried on the fact — collapses the fan-out back to the two real events, giving the correct 2 regardless of how many instructors the bridge attaches.

Output.

query	result	correct?
`COUNT(*)` after bridge join	4	no (inflated)
`COUNT(DISTINCT attendance_id)`	2	yes

Rule of thumb. Any join to a non-many-to-one table can inflate COUNT(*); carry a degenerate event id on the factless fact and count it distinctly whenever a bridge or multi-valued dimension is in the join path.

SQL interview question on grain and additivity

Question. You are asked to report "distinct products on promotion per store per week" from fct_promo_coverage (grain: product/store/promo/day), and a teammate proposes summing a daily products_on_promo count up to the week. Explain why that is wrong and write the correct weekly query.

Solution Using COUNT(DISTINCT) at the target grain instead of summing lower-grain counts

Code.

SELECT d.week,
       c.store_key,
       COUNT(DISTINCT c.product_key) AS products_on_promo
FROM   fct_promo_coverage c
JOIN   dim_date d ON d.date_key = c.date_key
GROUP  BY d.week, c.store_key;

Step-by-step trace.

product_key	store_key	date_key	week
500	1	20260901	2026-W36
500	1	20260902	2026-W36
501	1	20260902	2026-W36

The teammate's plan computes a daily distinct-product count (day 0901 = 1, day 0902 = 2) and sums them to 3 for the week — but product 500 was on promotion on both days and gets counted twice.
Distinct counts are non-additive: you cannot sum a distinct measure from a lower grain to a higher one without double-counting members that span buckets.
The correct query recomputes COUNT(DISTINCT product_key) directly at the week grain, so product 500 (present on two days) is counted once.
The result is 2 distinct products on promotion in store 1 for the week, not the inflated 3.

Output:

week	store_key	products_on_promo
2026-W36	1	2

Why this works — concept by concept:

Grain discipline — recomputing the distinct at the reporting grain respects that the coverage grain is per-day; you never sum across the boundary that would double-count a member.
Non-additive measures — COUNT(DISTINCT) cannot be rolled up like SUM, so the only correct way to get a weekly distinct is to aggregate the base rows to the week directly.
Dimension-driven rollup — joining dim_date to attach the week lets one query serve any time grain by changing the GROUP BY, without pre-aggregated intermediate tables that would bake in the additivity bug.
Keys-only integrity — because the coverage fact stores only keys, there is no stored count to be tempted into summing; the metric is always derived at query time at the right grain.
Cost — a single COUNT(DISTINCT) over the base rows is O(rows) with a hash aggregate, and it avoids the two-pass "aggregate then re-aggregate" that produces the wrong answer anyway.

Grain
Topic — fact-grain
Fact-table grain and additivity problems

Practice →

SQL Topic — factless-fact Factless fact modeling and pitfall problems

Practice →

Cheat sheet — factless fact recipes

Event factless fact (keys only, count the rows).

CREATE TABLE fct_attendance (
    student_key BIGINT NOT NULL,
    class_key   BIGINT NOT NULL,
    date_key    INT    NOT NULL
);
SELECT class_key, COUNT(*) AS attendances
FROM   fct_attendance GROUP BY class_key;

Coverage / eligibility factless fact (the universe of the possible).

CREATE TABLE fct_promo_coverage (
    product_key BIGINT NOT NULL,
    store_key   BIGINT NOT NULL,
    promo_key   BIGINT NOT NULL,
    date_key    INT    NOT NULL
);

"What did NOT happen" — LEFT JOIN … IS NULL.

SELECT c.product_key, c.store_key, c.date_key
FROM   fct_promo_coverage c
LEFT   JOIN fct_sales s
       ON s.product_key = c.product_key
      AND s.store_key   = c.store_key
      AND s.date_key    = c.date_key
WHERE  s.product_key IS NULL;

Absence — the NULL-safe NOT EXISTS form.

SELECT e.course_key, e.student_key
FROM   fct_enrollment_coverage e
WHERE  NOT EXISTS (
    SELECT 1 FROM fct_attendance a
    WHERE a.student_key = e.student_key
      AND a.class_key   = e.course_key
);

Attendance rate = distinct attendees ÷ enrolled (event ÷ coverage).

SELECT en.course_key,
       COUNT(DISTINCT at.student_key)::numeric
         / COUNT(DISTINCT en.student_key) AS attendance_rate
FROM   fct_enrollment_coverage en
LEFT   JOIN fct_attendance at
       ON at.student_key = en.student_key
      AND at.class_key   = en.course_key
GROUP  BY en.course_key;

Coverage vs event — which family to build.

Question shape	Family	Metric
"How many times did X happen?"	event tracking	`COUNT(*)`
"How many distinct members did X?"	event tracking	`COUNT(DISTINCT member)`
"What was eligible / on offer / assigned?"	coverage	`COUNT(DISTINCT combo)`
"What was possible but never happened?"	coverage anti-join event	`LEFT JOIN … IS NULL`

Frequently asked questions

What is a factless fact table?

A factless fact table is a fact table that contains only foreign keys to dimensions (plus optionally a date key and a degenerate id) and no numeric measure column. The presence of a row is the fact: it records that an event occurred or that a condition was in effect. You derive metrics by counting rows with COUNT(*) and COUNT(DISTINCT ...) rather than summing a stored measure, so "factless" means "no stored measure," not "no meaning."

What are the two types of factless fact tables?

There are two families. Event tracking stores one row every time something happens — a student attends a class, a user views a page, a visitor logs in — and you answer questions by counting those rows. Coverage (or condition / eligibility) stores one row for every combination that is in effect — a product on promotion, a student enrolled in a course, a rep assigned to a territory — regardless of whether anything happened. Event tables tell you what occurred; coverage tables tell you what could have occurred.

How do you get a measure from a factless fact table?

You count. COUNT(*) over the rows gives the number of events (attendances, views, logins), and COUNT(DISTINCT some_key) gives the number of distinct members (unique attendees, unique visitors). Because the grain is "one row per occurrence," the row count is the measure — adding a constant 1 column so you can SUM it is redundant, since it always equals COUNT(*).

How do you answer "what did not happen" in a star schema?

You need two tables: a coverage factless fact for what was possible and an event fact for what occurred. Then you anti-join them — coverage LEFT JOIN events ON <full key> WHERE events.key IS NULL, or the NULL-safe NOT EXISTS form. The coverage rows with no matching event are exactly the things that did not happen: promoted products that sold nothing, enrolled students who never attended. An event fact alone can never answer this because absence leaves no row.

What is a coverage fact table?

A coverage fact table is a factless fact that enumerates the universe of eligible or in-effect combinations — every product on promotion in every store on every active day, for example — whether or not an event occurred against them. It captures the denominator you need for rates (attendance rate, redemption rate) and the reference set you anti-join against to find non-events. It typically lives at a per-day snapshot grain or as effective-dated valid_from / valid_to ranges.

Can a factless fact table have a degenerate dimension?

Yes, and it often should. A degenerate dimension is a key that identifies the specific event — an attendance_id, session_id, or admission_no — but has no attributes of its own, so it is stored in the fact table with no dimension to join to. It lets you COUNT(DISTINCT event_id) to get an accurate event count even when a join (to a bridge or multi-valued dimension) fans a single event out into several rows. It is still a key, never a measure.

Practice on PipeCode

Pipecode.ai is Leetcode for Data Engineering — every factless-fact idea above, from the event-counting attendance table to the coverage universe and the anti-join for absence, maps to a hands-on practice room where you build the model and the query against real graded inputs. PipeCode pairs each reading with 450+ DE-focused problems and a real-time scoring engine, so your answer to "how would you find the products that were on promotion but never sold?" holds up under a senior interviewer's depth probes.

Practice factless-fact problems now →
Coverage-fact drills →

Bridge Tables & Many-to-Many Dimensions: Modeling Hierarchies and Multi-Valued Attributes

Gowtham Potureddi — Wed, 23 Sep 2026 13:52:42 +0000

bridge table is the piece of a star schema you reach for the moment a relationship refuses to be many-to-one. A textbook fact row joins to exactly one row in each dimension — one order, one customer, one date. But reality is full of relationships that are many-to-many: a bank account can be owned by several customers, a customer can hold several accounts; a hospital visit can carry several diagnoses; a product can wear several category tags; an employee can sit at any depth beneath a manager in an org chart. A bridge table is the associative table you slot between the two sides so the model can express "many on both ends" without either overwriting data or silently multiplying your numbers.

The reason interviewers keep asking about bridges is that they sit exactly where correctness breaks. Join a fact to a dimension through a bridge naively and a joint account's balance shows up once per owner, so the bank's total balance inflates — the classic double-counting fan trap. This guide walks the four ideas an interviewer will actually probe — the fact-to-dimension many-to-many and where the bridge goes, the weighting or allocation factor that makes shared measures sum correctly, multi-valued attributes and multi-valued dimensions, and hierarchy bridges / closure tables for variable-depth trees like org charts and bills of materials — and pairs each with a Solution-Tail interview answer: SQL, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the bridge-table practice library →, rehearse the join shapes on the many-to-many practice set →, and harden your tree queries on the recursive-hierarchy practice set →.

On this page

Why bridge tables exist — the many-to-many problem
Bridge tables between a fact and a dimension
Weighting factors — allocation without double-counting
Multi-valued attributes & multi-valued dimensions
Hierarchy bridges & closure tables for ragged trees
Cheat sheet — bridge & hierarchy recipes
Frequently asked questions
Practice on PipeCode

1. Why bridge tables exist — the many-to-many problem

A star schema assumes many-to-one — a bridge is what you add when both sides are "many"

The one-sentence invariant: a bridge table is an associative table placed between two entities that have a many-to-many relationship, holding one row for each valid pair of keys. Everything about bridges follows from the fact that a normal dimensional join cannot store "many on both ends." A fact table carries a foreign key to each dimension; that foreign key holds exactly one value, so the fact-to-dimension grain is many facts to one dimension member. The instant a single fact (or a single dimension member) legitimately relates to several members on the other side, that single-valued foreign key runs out of room.

Where many-to-many shows up in real models.

Shared ownership. One bank account, several account holders (a joint account); one insurance policy, several covered members; one property, several owners.
Multi-valued classification. One patient visit, several diagnoses; one support ticket, several tags; one product, several categories or attributes.
Ragged hierarchies. One employee reports through a chain of managers of unknown length; one assembly contains sub-assemblies to arbitrary depth (a bill of materials).
The symmetry that hurts. In every case the relationship is many-to-many both ways — a customer also holds many accounts — so neither side can host the foreign key.

The three wrong answers a bridge replaces.

Overwrite (lose data). Put a single customer_key on the account and you can record only one owner; the second holder vanishes. Correct total, wrong coverage.
Grain explosion (duplicate facts). Duplicate the fact once per owner and you have changed the grain of the fact table; every downstream SUM now double-counts unless every query remembers to guard against it.
Comma-list / repeating groups (unqueryable). Stuff "C1,C2" into one column, or add owner_1, owner_2, owner_3 columns. Both are anti-patterns: you cannot join, filter, or aggregate on them, and the fixed columns cap the cardinality arbitrarily.

What a bridge actually is.

An associative table at the grain of the relationship. One row per related pair — (account_key, customer_key) — and nothing else except optional attributes of the relationship (a weighting factor, an effective-date range, a role like "primary owner").
A join hub, not a fact. A bridge usually has no additive measures of its own; it exists to be joined through. Its own grain is "one row per pair," which is exactly what lets you fan a query out and — with care — fan it back in.
The place correctness is won or lost. Because a query passes through the bridge, the bridge is where you attach the weighting factor that prevents double-counting and the flags that let you pick a rollup level.

What interviewers listen for.

Do you name the relationship as many-to-many both ways before proposing a table? — the framing that earns trust.
Do you put the bridge at the grain of the relationship ("one row per account-customer pair"), not at the grain of a fact? — the core idea.
Do you volunteer the double-counting trap and the weighting factor that fixes it before being asked? — senior signal.
Do you distinguish a bridge (resolves M:N, no measures) from a many-to-many fact (has its own measures) and a junk dimension (collapses low-cardinality flags)? — vocabulary that separates levels.

Worked example — the moment a single foreign key runs out of room

Detailed explanation. The cleanest way to feel why bridges exist is to try to model a joint account without one and watch the model fail. We have three accounts and three customers; account A3 is a joint checking account owned by two people. A single customer_key on the account dimension can hold only one of them, so the model is forced to either drop an owner or invent repeating columns.

Question. Three accounts (A1, A2, A3) and three customers (C1 Ada, C2 Linus, C3 Grace). A1 is owned by C1 alone, A2 by C2 alone, and A3 jointly by C1 and C2. Show why a single owner_customer_key column on dim_account cannot represent this, and state the grain of the table that can.

Input.

account	type	owners
A1	checking	C1
A2	savings	C2
A3	joint checking	C1, C2

Code.

-- ATTEMPT (broken): one owner column on the account dimension
CREATE TABLE dim_account (
    account_key      INT PRIMARY KEY,
    account_type     TEXT,
    owner_customer_key INT   -- can hold ONE value; A3 has two owners
);

-- The fix: an associative bridge at the grain of the relationship
CREATE TABLE bridge_account_customer (
    account_key   INT NOT NULL,
    customer_key  INT NOT NULL,
    PRIMARY KEY (account_key, customer_key)   -- one row per (account, customer) pair
);

INSERT INTO bridge_account_customer (account_key, customer_key) VALUES
    (1, 1),           -- A1 -> C1
    (2, 2),           -- A2 -> C2
    (3, 1), (3, 2);   -- A3 -> C1 and C2  (the joint account)

Step-by-step explanation. The dim_account.owner_customer_key column is single-valued, so A3 can name only one of its two owners — the model is already lossy before a single query runs. Moving ownership out to bridge_account_customer changes the grain to "one row per account-customer pair," so A3 simply gets two rows. The composite primary key (account_key, customer_key) both enforces that each pair appears once and documents the grain. Nothing about the account or the customer is stored here — only the fact that they are related.

Output.

bridge row	account_key	customer_key	meaning
1	1	1	A1 owned by C1
2	2	2	A2 owned by C2
3	3	1	A3 co-owned by C1
4	3	2	A3 co-owned by C2

Rule of thumb. If a foreign-key column would ever need to hold more than one value, the relationship is many-to-many and belongs in a bridge whose grain is "one row per related pair."

2. Bridge tables between a fact and a dimension

The account↔customer bridge — how a fact reaches a many-to-many dimension, and the trap on the way

Once the bridge exists, the interesting question is how a fact measured at one grain (a daily account balance) is reported along a dimension it relates to many-to-many (the customers who own the account). The fact stays at its natural grain — one row per account per day — and the customer is reached by hopping through the bridge. Get the join path right and you can slice balance by customer; get it wrong and the bank's numbers inflate.

The join path, one hop at a time.

Fact → account. fact_balance carries account_key at its own grain (account × day). This is a clean many-to-one join, exactly as in any star schema.
Account → bridge. bridge_account_customer explodes each account into one row per owner. This is where a single fact row can fan into several.
Bridge → customer. The bridge's customer_key reaches dim_customer for names, segments, and other attributes.
The shape. fact → dim_account → bridge → dim_customer is the canonical four-table path for a fact reported through a many-to-many dimension.

Two places to put the bridge (and why it matters).

Directly between two dimensions — bridge_account_customer links dim_account and dim_customer. Simple, and the pattern shown here.
Between a dimension and a "group" — Kimball's multivalued-dimension bridge: the account points at a customer_group_key, and the bridge maps each group to its members. This lets many accounts that share the exact same owner set reuse one group, shrinking the bridge — worth it when groups repeat heavily.
The rule. Use a direct bridge when pairs are mostly unique; introduce a group key when the same set of members recurs across many parents.

The double-counting trap, stated precisely.

A fan-out join multiplies the measure. Joining fact_balance through the bridge turns A3's one balance row into two rows (one per owner). SUM(balance) across the exploded result now counts A3's balance twice.
It is grain, not a bug in SQL. The join is doing exactly what you asked; the result set's grain became "balance per account-owner," so a naive total over it is meaningless.
When it is actually fine. A per-customer coverage or impact report — "which customers are exposed to this account" — deliberately wants the full value against each owner. The trap only bites when you then sum across customers expecting the bank total.

Worked example — slicing balance by customer through the bridge

*Detailed explanation. * The everyday query is "total balance held by each customer." It reads naturally — join the fact to the account, hop the bridge to the customer, group by customer — and it is correct per customer. The subtlety is only in what happens when you sum those per-customer numbers back up, which the next section fixes with a weight. Here we establish the honest per-customer view and expose the inflated grand total.

Question. With balances A1 = 1000, A2 = 4000, A3 = 600, report total balance per customer by joining fact_balance through the bridge. Then sum those per-customer totals and compare to the true bank total of 5600.

Input.

account_key	balance
1 (A1)	1000
2 (A2)	4000
3 (A3)	600

Code.

SELECT  c.customer_key,
        c.name,
        SUM(f.balance) AS balance_by_customer
FROM        fact_balance          f
JOIN        bridge_account_customer b ON b.account_key = f.account_key
JOIN        dim_customer          c ON c.customer_key = b.customer_key
GROUP BY    c.customer_key, c.name
ORDER BY    c.customer_key;

Step-by-step explanation. The join fact_balance → bridge fans A3's single 600 row into two rows — one for C1, one for C2. C1 therefore accumulates A1 (1000) + A3 (600) = 1600, and C2 accumulates A2 (4000) + A3 (600) = 4600. Both per-customer numbers are legitimate: C1 really does have access to 1600 of balance. But summing the customer column gives 1600 + 4600 = 6200, which overshoots the real 5600 by exactly A3's 600 — the joint balance counted once per owner.

Output.

customer	accounts summed	balance_by_customer
C1 Ada	A1 + A3	1600
C2 Linus	A2 + A3	4600
grand total (naive)	—	6200 ✗ (true = 5600)

Rule of thumb. A fact summed through a bridge is correct along the grouped dimension but not across it — the per-customer numbers are trustworthy, the grand total is inflated by every shared parent.

SQL interview question on a fact-to-dimension bridge

Question. You are handed fact_balance(account_key, balance), bridge_account_customer(account_key, customer_key), and dim_customer(customer_key, name). An analyst wrote SELECT SUM(balance) FROM fact_balance JOIN bridge_account_customer USING(account_key) to get total bank balance and got 6200 instead of 5600. Explain the defect and write a query that returns the correct bank total while still supporting a per-customer breakdown.

Solution Using a pre-aggregated fact to protect the grain

Code.

-- Correct bank total: aggregate the fact at its OWN grain, never through the bridge
SELECT SUM(balance) AS bank_total
FROM   fact_balance;                     -- 1000 + 4000 + 600 = 5600

-- Per-customer breakdown: fan through the bridge on purpose, but label it "exposure"
SELECT  c.name,
        SUM(f.balance) AS customer_exposure   -- intentionally counts shared balance per owner
FROM        fact_balance           f
JOIN        bridge_account_customer b ON b.account_key = f.account_key
JOIN        dim_customer           c ON c.customer_key = b.customer_key
GROUP BY    c.name;

Step-by-step trace.

step	rows	balance seen
1. fact at native grain	A1, A2, A3	1000, 4000, 600
2. SUM without joining bridge	3 rows	5600 (correct total)
3. join bridge (exposure query)	A1→C1, A2→C2, A3→C1, A3→C2	4 rows
4. group by customer	C1=1600, C2=4600	per-customer exposure

The bank total must be computed before the bridge fans the fact out; SUM(balance) over fact_balance alone touches each account once, so A3's 600 is counted once → 5600.
The moment the bridge joins in, A3 becomes two rows, so any total over that result double-counts by design.
The per-customer query keeps the fan-out, but you name the measure customer_exposure so no one mistakes it for an additive, cross-customer total.
Two questions, two grains, two queries — the fix is recognizing they are different questions, not forcing one query to answer both.

Output:

result	value	grain
bank_total	5600	fact grain (correct)
exposure C1	1600	per account-owner
exposure C2	4600	per account-owner

Why this works — concept by concept:

Grain discipline — the fact's additive grain is "one row per account"; any SUM that must equal the true total has to be taken at that grain, before a bridge multiplies rows.
Fan-out is intentional — joining through the bridge is the right move for a per-customer exposure view; it is only wrong when you then aggregate across the fanned dimension.
Name the measure by its grain — calling the fanned number customer_exposure rather than balance stops a downstream consumer from summing it into a meaningless total.
Two questions, two queries — "what is the bank total" and "what is each customer exposed to" have different grains and deserve separate SQL rather than one query patched to fake both.
Cost — the total is O(fact rows); the exposure query is O(fact rows × avg owners per account), bounded by the bridge fan-out factor.

SQL
Topic — bridge-table
Bridge-table join and grain problems

Practice →

Modeling Topic — many-to-many Many-to-many relationship modeling problems

Practice →

3. Weighting factors — allocation without double-counting

An allocation factor makes a shared measure sum correctly — one column that turns a fan trap into a clean total

The fix for the double-counting you just saw is a single extra column on the bridge: a weighting factor (also called an allocation factor) that says what fraction of a parent's measure belongs to each related member. Store weights that sum to 1.0 per parent, multiply the measure by the weight in your aggregate, and the shared value is split instead of duplicated — so a cross-member total lands exactly on the true whole.

The mechanism.

A weight per bridge row. bridge_account_customer gains a weighting_factor column. For a sole-owner account it is 1.0; for a 50/50 joint account each owner's row carries 0.5.
The invariant. For any given parent (account), the weights across all its bridge rows sum to exactly 1.0. This is the property that guarantees the allocated total equals the original.
The corrected expression. Replace SUM(f.balance) with SUM(f.balance * b.weighting_factor). Now A3's 600 contributes 300 to C1 and 300 to C2 — 600 in total, not 1200.
Where weights come from. Ownership percentages, headcount splits, revenue-share agreements, or a default equal split (1.0 / number_of_owners) when no business rule exists.

Two legitimate reports, two different measures.

Impact / coverage report (unweighted). Uses the raw measure against every member — "how much balance does each customer have access to." Correct per member; must not be summed across members.
Allocated report (weighted). Uses measure × weight — "how much of the balance is attributable to each customer." Sums correctly to the whole and is the one finance signs off on.
Ship both, label both. Interviewers love that you keep them distinct: an impact number and an allocated number answer different business questions and both are valid.

Traps around weighting.

Weights that don't sum to 1.0. If a parent's weights sum to 0.9 or 1.1, your allocated total drifts below or above the truth; enforce the invariant with a validation query or a CHECK on a materialized rollup.
Weighting a non-additive measure. Allocation is meaningful for additive measures (balance, revenue). A weighted average of a ratio (e.g. interest rate) needs a weighted-average formula, not a straight measure × weight sum.
Missing bridge rows. An account with no bridge rows drops out of every through-the-bridge query entirely; guard with an outer join or a data-quality check for orphan accounts.

Worked example — the same query, corrected by a weight

Detailed explanation. We take the exact per-customer query from section 2 and add * b.weighting_factor. A3's 600 now splits 300/300 across its two owners, so the per-customer numbers change and — crucially — their sum collapses onto the real 5600. This is the whole trick: one column, one multiplication.

Question. Add a weighting_factor to the bridge (sole accounts 1.0; A3 split 0.5/0.5) and report allocated balance per customer. Confirm the allocated total equals the true bank total of 5600.

Input.

account_key	customer_key	weighting_factor
1 (A1)	1 (C1)	1.0
2 (A2)	2 (C2)	1.0
3 (A3)	1 (C1)	0.5
3 (A3)	2 (C2)	0.5

Code.

SELECT  c.name,
        SUM(f.balance * b.weighting_factor) AS allocated_balance
FROM        fact_balance            f
JOIN        bridge_account_customer  b ON b.account_key = f.account_key
JOIN        dim_customer             c ON c.customer_key = b.customer_key
GROUP BY    c.name
ORDER BY    c.name;

Step-by-step explanation. For C1: A1 contributes 1000 × 1.0 = 1000 and A3 contributes 600 × 0.5 = 300, so C1 = 1300. For C2: A2 contributes 4000 × 1.0 = 4000 and A3 contributes 600 × 0.5 = 300, so C2 = 4300. The joint account's 600 is now split rather than duplicated, so 1300 + 4300 = 5600 — exactly the bank total. The only change from the section-2 query is the * b.weighting_factor factor inside the SUM.

Output.

customer	allocated_balance	contribution from A3
C1 Ada	1300	300
C2 Linus	4300	300
allocated total	5600 ✓	600 (split, not doubled)

Rule of thumb. When weights on a bridge sum to 1.0 per parent, SUM(measure × weight) is safe to total across the many-to-many dimension; without the weight, it never is.

SQL interview question on allocation

Question. Marketing wants "attributed revenue per customer" from fact_revenue(account_key, revenue) through the account↔customer bridge, and the CFO insists the per-customer numbers must add up to total revenue. Some accounts are joint. Write the query, and describe how you would validate that the weighting factors are trustworthy.

Solution Using weighted allocation with a weight-integrity check

Code.

-- Attributed revenue that sums to the whole
SELECT  c.customer_key,
        c.name,
        SUM(f.revenue * b.weighting_factor) AS attributed_revenue
FROM        fact_revenue            f
JOIN        bridge_account_customer  b ON b.account_key = f.account_key
JOIN        dim_customer             c ON c.customer_key = b.customer_key
GROUP BY    c.customer_key, c.name;

-- Weight-integrity check: every account's weights must sum to 1.0
SELECT  account_key,
        SUM(weighting_factor) AS weight_sum
FROM        bridge_account_customer
GROUP BY    account_key
HAVING      ABS(SUM(weighting_factor) - 1.0) > 0.0001;   -- rows here are BAD

Step-by-step trace.

account	owners	weights	revenue	contribution split
A1	C1	1.0	1000	C1 += 1000
A2	C2	1.0	4000	C2 += 4000
A3	C1, C2	0.5, 0.5	600	C1 += 300, C2 += 300

The allocation query multiplies each account's revenue by the owner's weighting_factor, so a joint account's revenue is divided among owners instead of replicated.
Because weights per account sum to 1.0, the sum of all attributed_revenue equals SUM(revenue) over the fact — the CFO's invariant holds by construction.
The integrity check groups the bridge by account_key and flags any account whose weights do not sum to 1.0 (within a tiny epsilon for floating point); an empty result means the weights are trustworthy.
Run the check as a data-quality test in CI or the pipeline so a bad weight fails the build rather than silently skewing attributed revenue.

Output:

result	value
attributed C1	1300
attributed C2	4300
sum attributed	5600 (= total revenue)
integrity check rows	0 (all accounts sum to 1.0)

Why this works — concept by concept:

Weighting factor — a per-relationship fraction stored on the bridge; multiplying the measure by it converts duplication into division.
Sum-to-one invariant — enforcing that each parent's weights total 1.0 is precisely what makes the allocated grand total equal the un-allocated one.
Allocation vs impact — weighted gives attributable revenue that adds up; unweighted gives exposure per member; naming them apart prevents a wrong total.
Weight integrity as a test — a HAVING SUM(weight) <> 1.0 query turns a subtle correctness property into an automatable data-quality gate.
Cost — one extra multiply per bridged row, O(fact rows × fan-out); the integrity check is O(bridge rows) and runs once per load, not per query.

Allocation
Topic — allocation-factor
Allocation-factor and weighted-rollup problems

Practice →

SQL Topic — bridge-table Bridge-table double-counting problems

Practice →

4. Multi-valued attributes & multi-valued dimensions

One member, many values — a multi-valued dimension attaches several attribute values through a bridge

The other half of the bridge story is the multi-valued dimension: a single dimension member that legitimately carries several values of one attribute. A customer belongs to several marketing segments; a patient visit carries several diagnoses; a product wears several category tags; a job posting lists several required skills. The wrong instinct is a comma-separated string or a fixed set of segment_1..segment_n columns; the right pattern is a bridge from the member to a value dimension.

The shape of a multi-valued dimension.

Member → bridge → value dimension. dim_customer → bridge_customer_segment → dim_segment. The bridge holds one row per (customer_key, segment_key) pair — the same associative grain as before, now between a dimension and its multi-valued attribute.
Optional group dimension. When many members share the exact same set of values (thousands of customers all tagged {premium, digital}), insert a segment_group between the member and the bridge so identical sets are stored once. The customer points at a segment_group_key; the bridge maps group → segments.
Optional weight. Just like ownership, a multi-valued attribute can carry a weight (primary vs secondary diagnosis, a confidence score), reusing the allocation machinery from section 3.

Querying a multi-valued dimension correctly.

Filtering is easy and safe. "Customers in the premium segment" is a straightforward join dim_customer → bridge → dim_segment WHERE segment = 'premium'; filtering does not double-count.
Counting members needs COUNT(DISTINCT). "How many customers per segment" fans a customer into one row per segment; counting customers across all segments must use COUNT(DISTINCT customer_key) or you count a two-segment customer twice.
Summing a member's measure needs a weight. If you attach a customer-level measure (lifetime value) and report it by segment, you are back in double-counting territory — allocate with a weight exactly as in section 3.

Multi-valued attribute vs the alternatives.

vs comma-list. A 'premium,digital' string can't be joined, indexed, or filtered without fragile string matching; a bridge makes every value a first-class, queryable row.
vs pivoted columns. is_premium, is_digital boolean columns work only for a small, fixed, rarely-changing value set; a new segment means a schema change. A bridge scales to any number of values with no DDL.
vs a junk dimension. A junk dimension collapses several low-cardinality, unrelated flags into one row; it is not for a single attribute with many values that relate a member to a growing list. Different tool, different problem.

Worked example — customers with several segments

Detailed explanation. We tag customers with marketing segments through a bridge. C1 Ada is {premium, digital}, C2 Linus is {retail}, C3 Grace is {premium, retail}. The example shows that filtering by segment is safe, but counting customers across segments needs COUNT(DISTINCT) because a multi-segment customer appears in several bridge rows.

Question. Given the segment tags above, (a) list customers in the premium segment, and (b) count distinct customers per segment. Show why a plain COUNT(*) on a cross-segment total would over-count.

Input.

customer_key	segment_key
1 (C1)	premium
1 (C1)	digital
2 (C2)	retail
3 (C3)	premium
3 (C3)	retail

Code.

-- (a) customers in the 'premium' segment  (filtering is safe)
SELECT  c.name
FROM        dim_customer          c
JOIN        bridge_customer_segment b ON b.customer_key = c.customer_key
JOIN        dim_segment           s ON s.segment_key = b.segment_key
WHERE       s.segment_key = 'premium';

-- (b) distinct customers per segment  (COUNT DISTINCT guards the fan-out)
SELECT  s.segment_key,
        COUNT(DISTINCT b.customer_key) AS customers
FROM        bridge_customer_segment b
JOIN        dim_segment           s ON s.segment_key = b.segment_key
GROUP BY    s.segment_key;

Step-by-step explanation. Query (a) filters the bridge to premium rows and returns C1 and C3 — filtering never duplicates because we are selecting members, not aggregating a measure across the fan-out. Query (b) groups by segment; premium has two rows (C1, C3), retail has two (C2, C3), digital has one (C1). COUNT(DISTINCT customer_key) counts members correctly within each segment. If you instead summed those segment counts to get "total customers" you would get 2 + 2 + 1 = 5 for only 3 real customers — because C1 and C3 each sit in two segments — which is why a cross-segment headcount must COUNT(DISTINCT customer_key) over the whole set, not add the per-segment counts.

Output.

segment	customers	members
premium	2	C1, C3
digital	1	C1
retail	2	C2, C3

Rule of thumb. Filtering a multi-valued dimension is free; counting members across it always needs COUNT(DISTINCT), and summing a member measure across it always needs a weight.

SQL interview question on a multi-valued dimension

Question. Each product has one or more category tags via bridge_product_category, and fact_sales(product_key, revenue) records sales at the product grain. A PM asks for "revenue by category." Some products have several categories. Give the report that (a) shows revenue exposure per category and (b) also produces category revenue that sums to total sales, and explain the difference.

Solution Using COUNT(DISTINCT) plus weighted category allocation

Code.

-- (a) exposure: full product revenue against every category it belongs to
SELECT  cat.category_name,
        SUM(f.revenue) AS revenue_exposure     -- sums > total sales if products are multi-category
FROM        fact_sales            f
JOIN        bridge_product_category b ON b.product_key = f.product_key
JOIN        dim_category          cat ON cat.category_key = b.category_key
GROUP BY    cat.category_name;

-- (b) allocated: split each product's revenue evenly across its categories
WITH cat_counts AS (
    SELECT product_key, COUNT(*) AS n_categories
    FROM   bridge_product_category
    GROUP  BY product_key
)
SELECT  cat.category_name,
        SUM(f.revenue * 1.0 / cc.n_categories) AS allocated_revenue
FROM        fact_sales            f
JOIN        bridge_product_category b ON b.product_key = f.product_key
JOIN        cat_counts            cc ON cc.product_key = f.product_key
JOIN        dim_category          cat ON cat.category_key = b.category_key
GROUP BY    cat.category_name;

Step-by-step trace.

product	categories	revenue	exposure adds	allocated adds
P1	electronics	1000	electronics += 1000	electronics += 1000
P2	electronics, home	600	each += 600	each += 300

The exposure query (a) sums full product revenue against every category, so P2's 600 lands in both electronics and home — great for "which categories touch this revenue," but the category column now totals more than sales.
The allocated query (b) derives n_categories per product in a CTE, then divides revenue by that count — an equal-split weighting factor computed on the fly (1.0 / n_categories).
Because each product's split weights sum to 1.0, the allocated category revenue sums back to total sales exactly, satisfying "must add up."
The two answers are both correct for different questions: exposure measures reach, allocation measures attributable revenue.

Output:

category	revenue_exposure (a)	allocated_revenue (b)
electronics	1600	1300
home	600	300
column total	2200 (> 1600 sales)	1600 (= total sales)

Why this works — concept by concept:

Multi-valued dimension — modeling category as a bridge, not a column, lets a product carry any number of categories and keeps every one queryable.
Exposure vs allocation — exposure keeps the full measure per value (reach); allocation splits it (attribution); each answers a real, distinct question.
On-the-fly weight — 1.0 / n_categories is a default equal-split weighting factor derived when the bridge has no stored weight, reusing the section-3 machinery.
DISTINCT for headcounts — whenever the deliverable is a count of members rather than a measure, COUNT(DISTINCT member_key) is the fan-out guard.
Cost — the CTE is O(bridge rows) to compute counts; the allocation join is O(fact rows × avg categories per product).

Modeling
Topic — many-to-many
Multi-valued dimension modeling problems

Practice →

SQL Topic — bridge-table Bridge-table aggregation problems

Practice →

5. Hierarchy bridges & closure tables for ragged trees

A closure table stores every ancestor-descendant pair — one join rolls up a fact at any level of a ragged tree

The hardest many-to-many is a member relating to itself at unknown depth: an org chart, a bill of materials, a category tree. An adjacency list (employee.manager_key) captures each parent-child edge but can't be flattened by a fixed number of joins because the depth is variable — a ragged hierarchy. The bridge that solves this is a closure table (Kimball calls it a hierarchy bridge): one row for every ancestor-descendant pair, including each node to itself, tagged with the depth between them.

What a closure table holds.

One row per ancestor→descendant pair. For every node, a row to each of its descendants at any depth, plus a depth-0 self row. E1's rows reach E1 (0), E2 (1), E3 (1), E4 (2), E5 (2).
A depth column. depth (a.k.a. distance) is the number of edges between ancestor and descendant; the self row has depth 0. It lets you query "direct reports only" (depth = 1) or "everyone below" (depth >= 1).
Optional flags. is_leaf, is_root, or Kimball's lowest_flag mark where a node sits, which is handy for stopping double-counts when a fact can attach at multiple levels.
No measures. Like every bridge, it stores structure, not facts; you join facts through it.

Building and using it.

Build with a recursive CTE. Walk the adjacency list from every node downward, accumulating depth; the anchor emits self rows at depth 0 and the recursive member extends each path by one edge.
Roll up a subtree with one join. To total a measure for "everyone under E2," join the fact on descendant and filter ancestor = E2. One join replaces an unbounded chain of self-joins.
Direct vs full rollup by depth. WHERE ancestor = X AND depth = 1 gives direct children; dropping the depth filter (or depth >= 0) gives the whole subtree including X itself.

Traps in hierarchy rollups.

Double-counting across ancestors. Every node has multiple ancestors (itself, its manager, its manager's manager). If you join a fact to the closure table and forget to fix a single ancestor, each fact row is counted once per ancestor — a massive over-count. Always filter or group by one ancestor level.
Facts attached at inner nodes. If measures live only at leaves, a plain descendant rollup is clean; if inner nodes also carry measures (a manager has personal sales), the depth-0 self rows correctly include them — which is exactly why the self row exists.
Cycles. A malformed hierarchy with a cycle makes the recursive CTE loop; guard with a depth cap or a visited-path check, and validate that the source is a true tree/DAG.

Worked example — building a closure table from an org chart

Detailed explanation. We have a five-person org: E1 (CEO) manages E2 and E3; E2 manages E4 and E5. The adjacency list stores only the direct manager_key. A recursive CTE expands it into a closure table of every ancestor-descendant pair with depth, which is the structure we then roll facts up through.

Question. From dim_employee(employee_key, manager_key) with edges E1→E2, E1→E3, E2→E4, E2→E5, build the closure table of (ancestor, descendant, depth) including depth-0 self rows.

Input.

employee_key	manager_key
E1	(none)
E2	E1
E3	E1
E4	E2
E5	E2

Code.

WITH RECURSIVE closure AS (
    -- anchor: every node is its own ancestor at depth 0
    SELECT  employee_key AS ancestor,
            employee_key AS descendant,
            0            AS depth
    FROM    dim_employee

    UNION ALL

    -- recursive: extend each known pair down one edge
    SELECT  c.ancestor,
            e.employee_key AS descendant,
            c.depth + 1    AS depth
    FROM        closure       c
    JOIN        dim_employee  e ON e.manager_key = c.descendant
)
SELECT ancestor, descendant, depth
FROM   closure
ORDER  BY ancestor, depth, descendant;

Step-by-step explanation. The anchor emits five self rows at depth 0 (E1→E1, …, E5→E5). The recursive member joins each existing pair's descendant to employees who report to it, emitting a new pair with depth + 1: from E1→E1 it finds E2 and E3 (depth 1), then from E1→E2 it finds E4 and E5 (depth 2), and independently E2→E2 yields E2→E4 and E2→E5 (depth 1). Recursion stops when no node reports to the current descendants (leaves E3, E4, E5). The result is every ancestor-descendant pair with its depth.

Output.

ancestor	descendant	depth
E1	E1	0
E1	E2	1
E1	E3	1
E1	E4	2
E1	E5	2
E2	E2	0
E2	E4	1
E2	E5	1
E3	E3	0
E4	E4	0
E5	E5	0

Rule of thumb. A closure table trades storage (roughly nodes × average depth rows) for O(1)-join rollups at any level — build it once per hierarchy load, query it cheaply forever.

SQL interview question on hierarchy rollup

Question. Using the closure table above and fact_sales(employee_key, sales) with E1=0, E2=100, E3=200, E4=50, E5=70, return total sales for the entire subtree under a given manager (including the manager's own sales). Show the result for E2, and explain how you avoid counting a person's sales once per ancestor.

Solution Using a descendant join filtered to one ancestor

Code.

-- Subtree rollup: everyone at or below the chosen manager
SELECT  cl.ancestor      AS manager_key,
        SUM(f.sales)     AS subtree_sales
FROM        closure     cl
JOIN        fact_sales   f ON f.employee_key = cl.descendant
WHERE       cl.ancestor = 'E2'          -- one ancestor => no cross-ancestor double count
GROUP BY    cl.ancestor;

-- Direct reports only would add:  AND cl.depth = 1

Step-by-step trace.

closure row (ancestor=E2)	descendant	sales joined
E2 → E2 (depth 0)	E2	100 (manager's own)
E2 → E4 (depth 1)	E4	50
E2 → E5 (depth 1)	E5	70

Filtering ancestor = 'E2' selects exactly the rows whose descendants form E2's subtree: E2 (self, depth 0), E4, and E5.
Joining fact_sales on descendant attaches each person's sales to precisely one row, because within a single ancestor each descendant appears once.
SUM(f.sales) totals 100 + 50 + 70 = 220 — the manager's own sales plus every report at any depth.
The double-count trap is avoided by fixing a single ancestor; had we joined the fact to the whole closure table and grouped by employee, each person's sales would be added once per ancestor above them (E4's 50 would count under E4, E2, and E1). Pinning one ancestor per query is the discipline.

Output:

manager	subtree_sales	includes
E2	220	E2 (100) + E4 (50) + E5 (70)

Why this works — concept by concept:

Closure table — precomputing every ancestor-descendant pair turns a variable-depth walk into a flat join, so any subtree is reachable in one hop.
Depth column — depth = 1 isolates direct reports, depth >= 0 takes the full subtree; one table answers both without restructuring.
Self rows — the depth-0 row is what folds a manager's own measure into the subtree total, so inner-node facts are not lost.
One ancestor per query — fixing a single ancestor is the guard that stops a fact from being counted once per level of the tree above it.
Cost — rollup is O(descendants of the chosen node), a single indexed join; the build is O(nodes × average depth) done once per load, not per query.

Hierarchy
Topic — recursive-hierarchy
Recursive-hierarchy and org-chart rollup problems

Practice →

Modeling Topic — closure-table Closure-table and ragged-hierarchy problems

Practice →

Cheat sheet — bridge & hierarchy recipes

Bridge table DDL (grain = one row per pair).

CREATE TABLE bridge_account_customer (
    account_key      INT NOT NULL,
    customer_key     INT NOT NULL,
    weighting_factor NUMERIC(5,4) NOT NULL DEFAULT 1.0,
    PRIMARY KEY (account_key, customer_key)
);

Weighted rollup (sums correctly across the M:N dimension).

SELECT c.name, SUM(f.balance * b.weighting_factor) AS allocated
FROM   fact_balance f
JOIN   bridge_account_customer b ON b.account_key = f.account_key
JOIN   dim_customer c            ON c.customer_key = b.customer_key
GROUP  BY c.name;

Weight-integrity check (each parent's weights sum to 1.0).

SELECT account_key, SUM(weighting_factor) AS w
FROM   bridge_account_customer
GROUP  BY account_key
HAVING ABS(SUM(weighting_factor) - 1.0) > 0.0001;   -- rows = bad

Multi-valued dimension count (guard the fan-out).

SELECT s.segment_key, COUNT(DISTINCT b.customer_key) AS customers
FROM   bridge_customer_segment b
JOIN   dim_segment s ON s.segment_key = b.segment_key
GROUP  BY s.segment_key;

Closure table build (recursive CTE over an adjacency list).

WITH RECURSIVE closure AS (
    SELECT employee_key AS ancestor, employee_key AS descendant, 0 AS depth
    FROM   dim_employee
    UNION ALL
    SELECT c.ancestor, e.employee_key, c.depth + 1
    FROM   closure c
    JOIN   dim_employee e ON e.manager_key = c.descendant
)
SELECT * FROM closure;

Subtree rollup (one join, one fixed ancestor).

SELECT SUM(f.sales) AS subtree_sales
FROM   closure cl
JOIN   fact_sales f ON f.employee_key = cl.descendant
WHERE  cl.ancestor = :manager_key;      -- add AND cl.depth = 1 for direct reports

Pattern picker.

Situation	Pattern
Fact relates M:N to a dimension	Bridge between fact's dim and the other dim
Shared measure must sum to the whole	Bridge + `weighting_factor` (sums to 1.0)
One member, many attribute values	Multi-valued dimension bridge
Same value set reused by many members	Add a group dimension
Variable-depth tree (org / BOM)	Closure table + recursive CTE
Count members across a bridge	`COUNT(DISTINCT member_key)`

Frequently asked questions

What is a bridge table in dimensional modeling?

A bridge table is an associative table placed between two entities that have a many-to-many relationship, holding one row for each valid pair of keys — for example one row per (account_key, customer_key) when accounts and customers own each other many-to-many. It stores structure, not measures: you join through it to reach a dimension a fact relates to on both ends. Optional columns on the bridge describe the relationship itself, such as a weighting factor or an effective-date range.

How do you avoid double-counting through a bridge table?

Double-counting happens because joining a fact through a bridge fans one fact row into several, so a plain SUM counts a shared value once per related member. Fix it by either aggregating the fact at its own grain before the bridge (for a true grand total), or by adding a weighting factor that sums to 1.0 per parent and summing measure × weight. For headcounts rather than measures, use COUNT(DISTINCT member_key) so a member in several groups is counted once.

What is a weighting (allocation) factor?

A weighting or allocation factor is a fraction stored on each bridge row that says what share of a parent's measure belongs to that related member. The weights for any one parent must sum to exactly 1.0 — a 50/50 joint account gives each owner 0.5 — so that SUM(measure × weight) across the many-to-many dimension lands on the true total instead of a duplicated one. It lets you produce an "allocated" report that adds up alongside an unweighted "impact" report that measures reach.

What is a multi-valued dimension?

A multi-valued dimension is a dimension member that carries several values of one attribute — a customer in several marketing segments, a visit with several diagnoses, a product with several category tags. You model it with a bridge from the member to a value dimension (dim_customer → bridge_customer_segment → dim_segment) instead of a comma-separated string or fixed attr_1..attr_n columns. Filtering on it is safe; counting members across it needs COUNT(DISTINCT), and summing a member measure across it needs a weight.

What is a closure table and when do I use one?

A closure table (or hierarchy bridge) stores one row for every ancestor-descendant pair in a tree, including each node to itself at depth 0, tagged with the depth between them. Use it for variable-depth, ragged hierarchies — org charts, bills of materials, category trees — where an adjacency list can't be flattened by a fixed number of joins. You build it once with a recursive CTE, then roll a fact up for any subtree with a single join filtered to one ancestor, avoiding an unbounded chain of self-joins.

Bridge table vs junk dimension vs many-to-many fact — how do I tell them apart?

A bridge resolves a many-to-many relationship and carries no measures; you join through it. A junk dimension collapses several low-cardinality, unrelated flags (yes/no indicators, small enums) into one dimension to avoid littering the fact with columns — it is not about many-to-many. A many-to-many fact (a factless or transaction fact) records the relationship as an event and may carry its own additive measures. Reach for a bridge when you need to attach a dimension that relates many-to-many; reach for a factless fact when the relationship itself is the thing you are measuring or counting.

Practice on PipeCode

Pipecode.ai is Leetcode for Data Engineering — every idea above, from the account-customer bridge and the weighting factor that stops double-counting to the multi-valued dimension and the closure-table subtree rollup, maps to a hands-on practice room where you write the SQL against real graded inputs. PipeCode pairs each reading with 450+ DE-focused problems and a real-time scoring engine, so your answer to "how do you model a many-to-many between a fact and a dimension without double-counting?" holds up under a senior interviewer's depth probes.

Practice bridge-table problems now →
Many-to-many modeling drills →

Special Dimensions: Junk, Degenerate, Role-Playing & Conformed Dimensions

Gowtham Potureddi — Wed, 23 Sep 2026 13:46:46 +0000

special dimensions are the four named refinements the Kimball method reaches for when a plain star schema starts to sag — when you have a fistful of yes/no flags with nowhere clean to put them, an operational key like order_number that everybody groups by but that has no attributes to describe, a single calendar that three different date columns all want to join to, and a customer table that has quietly been redefined in every data mart until no two reports agree. Each smell has a canonical fix, and the fix has a name: the junk dimension, the degenerate dimension, the role-playing dimension, and the conformed dimension.

None of these is an exception to dimensional modeling; they are dimensional modeling working as intended. The grain of the fact table is still the anchor, surrogate keys still connect facts to dimensions, and a business user can still slice by any attribute. The specials just keep the model honest under pressure — they stop flag columns from sprawling, they stop you from building empty dimension tables around keys, they stop you from copying the date dimension four times, and they stop every mart from inventing its own truth. This guide walks the four patterns an interviewer will actually make you name and defend, and pairs each with a Solution-Tail answer: the DDL, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the junk-dimension practice set →, rehearse operational-key modelling on the degenerate-dimension practice set →, and wire up multi-role calendars on the role-playing-dimension practice set →.

On this page

Why special dimensions exist in a star schema
Junk dimensions — collapse low-cardinality flags into one dim
Degenerate dimensions — operational keys with no dim table
Role-playing dimensions — one dim, many roles via views
Conformed dimensions — shared across facts for drill-across
Cheat sheet — special-dimension recipes
Frequently asked questions
Practice on PipeCode

1. Why special dimensions exist in a star schema

Special dimensions are pattern names for star-schema hygiene problems — learn the smell, then the fix

The one-sentence invariant: each special dimension is a named answer to a specific way a naive star schema goes wrong, so the skill an interviewer is testing is "hear the smell, name the pattern, write the DDL." A star schema is a central fact table (measurements at a declared grain) surrounded by dimension tables (the context you filter and group by), joined through surrogate keys. That works beautifully until the real world hands you attributes that do not fit the tidy "one dimension per business entity" story. The four special dimensions are the tidy answers to those four misfits.

The four smells and their fixes.

Flag sprawl → junk dimension. You have several low-cardinality, unrelated indicators (is_gift, is_priority, order_channel, payment_type). Giving each its own dimension is silly (one attribute each), and leaving them as columns on the fact bloats the grain with degenerate text. The junk dimension folds them into one small table.
Orphan operational key → degenerate dimension. You have a key like order_number that you must group and count by, but it has no attributes of its own worth a table. You store it as a column on the fact with no dimension behind it.
Repeated date joins → role-playing dimension. order_date, ship_date, and due_date all point at the same calendar. You keep one physical dim_date and expose it under several view aliases, one per role.
Siloed marts → conformed dimension. Sales and returns each built their own customer table, so cross-process numbers never reconcile. A conformed dimension is one shared, identically-keyed dimension used by every fact, which is what makes drill-across correct.

What stays constant across all four.

Grain first. You still declare "one row per ___" for the fact before anything else; the specials never change the grain, they change how context attaches to it.
Surrogate keys still rule. Junk, role-playing, and conformed dimensions all connect via integer surrogate keys. The degenerate dimension is the deliberate exception — it is a natural key with no surrogate because there is no table to key into.
Additivity is unaffected. Special dimensions are about context columns, not measures, so they never touch whether a measure is additive, semi-additive, or non-additive.

What interviewers listen for.

Can you name the pattern from the symptom without being handed the term? — the core signal.
Do you say "a degenerate dimension has no dimension table" and not accidentally build one? — a classic trap.
Do you explain role-playing as "one physical table, many views" rather than "copy the date dimension"? — senior signal.
Do you connect conformed dimensions to drill-across and the bus matrix, not just "shared table"? — the whole point of conforming.

Worked example — one order row that needs all four patterns

Detailed explanation. A single fact table can exhibit all four smells at once, which is why they are taught together. Picture fact_sales_line at the grain of one order line. It carries a handful of flags (junk), an order_number you group by (degenerate), three dates that all reference the calendar (role-playing), and a customer_key that must mean the same thing here as it does in fact_returns (conformed). Seeing all four on one row is the fastest way to internalise that they are complementary, not competing.

Question. Given a raw order-line record, label which special-dimension pattern each field belongs to.

Input.

field	example value	nature
order_number	SO-1007	operational key, no attributes
is_gift / is_priority / channel	true / false / web	low-cardinality flags
order_date / ship_date / due_date	2026-03-01 / 03-03 / 03-05	three refs to one calendar
customer_id	C-42	shared entity across processes

Code.

-- The line-item grain, annotated by pattern
CREATE TABLE fact_sales_line (
  order_number   VARCHAR,          -- degenerate dimension (lives here, no dim table)
  junk_key       BIGINT,           -- junk dimension (is_gift + is_priority + channel folded)
  order_date_key INT,              -- role-playing dim_date (role 1)
  ship_date_key  INT,              -- role-playing dim_date (role 2)
  due_date_key   INT,              -- role-playing dim_date (role 3)
  customer_key   BIGINT,           -- conformed dimension (same dim in every fact)
  product_key    BIGINT,
  quantity       INT,
  extended_amount NUMERIC(12,2)
);

Step-by-step explanation. order_number stays as a bare column — it is the degenerate dimension, no dim_order exists. The three flags collapse into a single junk_key pointing at dim_order_junk. The three date columns are three separate foreign keys into the same dim_date, disambiguated by role. customer_key points at a dim_customer that is byte-for-byte the same dimension fact_returns uses, so the two facts can be joined on it. Every other column is a normal measure or a standard dimension key.

Output.

field	pattern	has its own dim table?
order_number	degenerate	no
junk_key	junk	yes (`dim_order_junk`)
order/ship/due_date_key	role-playing	yes, one shared `dim_date`
customer_key	conformed	yes, shared across facts

Rule of thumb. When a field does not fit "one clean dimension per business entity," it is almost always one of these four — walk the list in order (junk, degenerate, role-playing, conformed) and one of them will fit.

SQL
Topic — dimensional-modeling
Dimensional-modeling and star-schema design problems

Practice →

Schema Topic — star-schema Star-schema fact-and-dimension modelling problems

Practice →

2. Junk dimensions — collapse low-cardinality flags into one dim

A junk dimension folds unrelated low-cardinality flags into one table keyed by a single surrogate

The feature that makes a junk dimension worth naming is that it removes clutter from two places at once: it keeps the fact table narrow (one junk_key instead of five text flags) and it avoids a swarm of one-column dimension tables. "Junk" is Kimball's own slightly jokey term — nothing about the data is low-quality; it is a grab-bag dimension that gathers the leftover indicators that do not deserve a dimension each.

What belongs in a junk dimension.

Low-cardinality flags. Booleans (is_gift, is_priority, is_returned) and short enumerations (order_channel ∈ {web, phone, store}, payment_type ∈ {card, cash, wire}).
Unrelated by nature. The flags need not correlate; the junk dimension is a convenience container, not a business entity. It is fine that channel and payment_type have nothing to do with each other.
Text you would otherwise dump on the fact. Anything that is really a small code or indicator, where storing the raw string on every fact row would bloat the table and slow scans.

How it is populated — observed vs cartesian.

Cartesian (full cross-product). Enumerate every possible combination up front. For 2 booleans × a 3-value channel × a 3-value payment type that is 2×2×3×3 = 36 rows. Fine when the product is small and bounded.
Observed-only (recommended for wide products). Insert a combination the first time it actually appears in the data. If only 12 of the 36 combinations ever occur, you carry 12 rows. This is the safer default because the cross-product explodes with each added flag.
The junk_key is a surrogate. Each distinct combination gets one integer junk_key; the fact stores that key, never the raw flags.

When NOT to junk.

High-cardinality attributes. If a "flag" has thousands of values (a free-text note, a user id), it is not junk — it belongs in its own dimension or is degenerate.
Attributes analysed independently at scale. If the business constantly filters only by channel across huge scans, a dedicated small dimension (or keeping it standalone) can index better than a combination table.
Rapidly changing correlated attributes. Those point toward a mini-dimension, not a junk dimension.

Worked example — building dim_order_junk from observed combinations

Detailed explanation. The everyday build is: take the distinct combinations of the flags that actually occur in the source, assign each a surrogate junk_key, and then look that key up when loading the fact. Below, three flags (is_gift, is_priority, channel) that would otherwise be three columns on every fact row become one small dimension.

Question. Build a junk dimension for is_gift, is_priority, and channel, populated only with combinations that appear in the orders source, and show the resulting table.

Input.

order	is_gift	is_priority	channel
1	true	false	web
2	false	false	web
3	true	false	web
4	false	true	phone

Code.

CREATE TABLE dim_order_junk (
  junk_key     BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  is_gift      BOOLEAN NOT NULL,
  is_priority  BOOLEAN NOT NULL,
  channel      VARCHAR(10) NOT NULL,
  UNIQUE (is_gift, is_priority, channel)   -- one row per distinct combination
);

-- Observed-only population: insert combinations that actually occur
INSERT INTO dim_order_junk (is_gift, is_priority, channel)
SELECT DISTINCT is_gift, is_priority, channel
FROM   stg_orders
ON CONFLICT (is_gift, is_priority, channel) DO NOTHING;

Step-by-step explanation. The UNIQUE constraint on the three flag columns is the heart of the pattern — it guarantees exactly one row per distinct combination, so junk_key is a clean surrogate for "this bundle of flags." The SELECT DISTINCT ... ON CONFLICT DO NOTHING inserts only combinations present in staging and silently skips ones already loaded, which makes the load idempotent and observed-only. Order 1 and order 3 share the same combination (true,false,web), so they collapse to one dimension row.

Output.

junk_key	is_gift	is_priority	channel
1	true	false	web
2	false	false	web
3	false	true	phone

Rule of thumb. Populate observed-only unless the full cross-product is tiny and bounded — with each extra flag the cartesian product multiplies, and most of those combinations will never occur.

SQL interview question on junk dimensions

Question. You are loading fact_orders. The source has is_gift, is_priority, and channel on each order, plus a fully-populated dim_order_junk. Write the load that replaces those three columns on the fact with a single junk_key, and explain what happens when a brand-new flag combination shows up.

Solution Using a lookup join to resolve junk_key

Code.

-- Resolve the three flags to one surrogate as the fact loads
INSERT INTO fact_orders (order_number, customer_key, order_date_key, junk_key, amount)
SELECT s.order_number,
       c.customer_key,
       d.date_key,
       j.junk_key,                       -- the folded flags
       s.amount
FROM   stg_orders s
JOIN   dim_customer   c ON c.customer_id = s.customer_id
JOIN   dim_date       d ON d.full_date   = s.order_date
JOIN   dim_order_junk j                  -- match on the flag bundle
       ON j.is_gift     = s.is_gift
      AND j.is_priority = s.is_priority
      AND j.channel     = s.channel;

Step-by-step trace.

stg_orders row (gift, prio, channel)	matched junk_key	fact junk_key stored
(true, false, web)	1	1
(false, false, web)	2	2
(false, true, phone)	3	3
(true, true, store)	none → row dropped by INNER JOIN	—

The JOIN dim_order_junk ON (is_gift, is_priority, channel) translates the three source flags into one junk_key, which is all the fact stores.
Existing combinations resolve cleanly; the fact row carries a single integer instead of three columns.
The new combination (true, true, store) has no matching junk row, so an INNER JOIN silently drops that fact row — a real bug. The fix is to insert unseen combinations into dim_order_junk before the fact load (the observed-only INSERT ... DISTINCT from the worked example) or to use a LEFT JOIN to a "late-arriving" default key.
With the dimension populated first, every fact row finds its junk_key and the flags never touch the fact table.

Output:

fact_orders column	before pattern	after pattern
flag columns	is_gift, is_priority, channel	(removed)
key column	—	junk_key (one integer)

Why this works — concept by concept:

Junk dimension — one small table gathers unrelated low-cardinality flags so the fact stays narrow and you avoid a dimension per flag.
Surrogate junk_key — the fact references a single integer, so a scan reads one column instead of three text/boolean columns per row.
Populate-before-load — the dimension must contain a combination before the fact references it, or the inner join drops rows; observed-only insertion up front guarantees coverage.
Observed vs cartesian — inserting only seen combinations keeps the dimension tiny, where the full cross-product would grow multiplicatively with each flag.
Cost — the load is O(fact rows) with an index lookup on (is_gift, is_priority, channel); the dimension itself is bounded by the number of distinct combinations, typically tens of rows.

SQL
Topic — junk-dimension
Junk-dimension and flag-folding problems

Practice →

Modelling Topic — low-cardinality Low-cardinality attribute modelling problems

Practice →

3. Degenerate dimensions — operational keys with no dim table

A degenerate dimension is an operational key that lives on the fact with no dimension table behind it

The defining property, and the one interviewers trap on: a degenerate dimension has no dimension table at all — it is a dimension key stored directly in the fact because it has no attributes worth a table. The classic examples are transaction identifiers: order_number, invoice_number, ticket_id, shipment_id, pos_transaction_id. You genuinely need them — to group line items back into their header, to count distinct orders, to reconstruct a receipt — but there is nothing to describe about SO-1007 beyond the number itself, so building a dim_order with a single column would be pure overhead.

Why it is called "degenerate."

It is a dimension that degenerated to just its key. Every attribute that would have lived in a dim_order has already moved elsewhere: the date went to dim_date, the customer to dim_customer, the channel into the junk dimension. What remains is the bare identifier.
It stays in the fact, unmodified. No surrogate key, no lookup — the natural operational key (order_number) sits in the fact table as a VARCHAR (or the source's native type). This is the one place in a star schema where a fact legitimately holds a natural key.
It is still a dimension conceptually. You group by it, filter by it, and count distinct values of it, exactly as you would a normal dimension attribute — it simply has no row storage of its own.

What a degenerate dimension is for.

Grain marker for header/line models. At the order-line grain, order_number is what ties the lines of one order together. Grouping fact_sales_line by order_number reconstructs the order header without a separate header table.
Distinct counts of transactions. COUNT(DISTINCT order_number) gives you order count from a line-grain fact — a metric you could not compute if the identifier were thrown away.
Traceability back to the source system. The operational id lets you join back to the OLTP system for audit or drill-to-detail.

Degenerate vs the alternatives.

Degenerate vs a real dimension. If the identifier gains attributes (an order got a status, a priority, a sales rep), those attributes go into a junk dimension or their own dimensions; the id itself stays degenerate. Do not "promote" it to a dimension table just to hang one attribute off it.
Degenerate vs junk. Junk folds several low-cardinality flags into a table; degenerate keeps one high-cardinality operational key with no table. Cardinality and table-existence are the discriminators.

Worked example — order_number on a line-grain fact

Detailed explanation. The canonical setup is a sales fact at the line-item grain. Each row is one product on one order; the order_number repeats across the lines that belong to the same order. There is no dim_order — the number is simply a column — and yet it does all the work of an order dimension.

Question. Model fact_sales_line so that order_number is a degenerate dimension, then show how to recover the order-level total from the line grain.

Input.

order_number	product_key	quantity	extended_amount
SO-1007	55	2	40.00
SO-1007	81	1	12.50
SO-1008	55	1	20.00

Code.

CREATE TABLE fact_sales_line (
  order_number    VARCHAR(20) NOT NULL,   -- DEGENERATE DIMENSION: no dim_order table
  date_key        INT     NOT NULL REFERENCES dim_date(date_key),
  product_key     BIGINT  NOT NULL REFERENCES dim_product(product_key),
  customer_key    BIGINT  NOT NULL REFERENCES dim_customer(customer_key),
  quantity        INT     NOT NULL,
  extended_amount NUMERIC(12,2) NOT NULL
);

-- Recover the order header total from the line grain by grouping on the DD
SELECT order_number,
       COUNT(*)            AS line_count,
       SUM(extended_amount) AS order_total
FROM   fact_sales_line
GROUP BY order_number;

Step-by-step explanation. order_number is declared as a plain column with no foreign key — there is nothing to reference because no dim_order exists. The two SO-1007 rows share the identifier, so GROUP BY order_number folds them into one header row. SUM(extended_amount) gives the order total (40.00 + 12.50 = 52.50) and COUNT(*) gives the line count, both derived purely from the degenerate dimension.

Output.

order_number	line_count	order_total
SO-1007	2	52.50
SO-1008	1	20.00

Rule of thumb. If an identifier is needed for grouping or distinct-counting but has no describing attributes, leave it in the fact as a degenerate dimension — building a one-column dimension table for it is a modelling smell, not good hygiene.

SQL interview question on degenerate dimensions

Question. A reviewer sees order_number sitting in your line-grain fact table and says "that natural key belongs in a dimension — build dim_order and put a surrogate on the fact." Defend the degenerate-dimension design and prove the order-count metric still works.

Solution Using the degenerate key directly for order counts

Code.

-- Order-level KPIs computed straight from the degenerate dimension,
-- no dim_order table, no surrogate key.
SELECT d.month,
       COUNT(DISTINCT f.order_number)                       AS orders,
       SUM(f.extended_amount)                               AS revenue,
       SUM(f.extended_amount) / COUNT(DISTINCT f.order_number) AS avg_order_value
FROM   fact_sales_line f
JOIN   dim_date d ON d.date_key = f.date_key
GROUP BY d.month
ORDER BY d.month;

Step-by-step trace.

line rows (order_number @ month, amount)	distinct orders	revenue
SO-1007 @ Mar, 40.00	—	—
SO-1007 @ Mar, 12.50	1 (same order)	52.50
SO-1008 @ Mar, 20.00	2	72.50

COUNT(DISTINCT order_number) collapses the multiple lines of an order into one, giving a correct order count from a line-grain fact — this is exactly the metric a dim_order would have been built to serve, and the degenerate dimension serves it for free.
SUM(extended_amount) aggregates the additive measure across lines for revenue.
Average order value divides revenue by distinct orders — three KPIs, no order dimension table.
Building dim_order with a surrogate would add a join, a load step, and a table that stores nothing but the same number; it buys you nothing because there are no attributes to describe. The degenerate design is simpler and faster.

Output:

month	orders	revenue	avg_order_value
Mar	2	72.50	36.25

Why this works — concept by concept:

Degenerate dimension — the operational key stays in the fact because it has no attributes; there is no dimension table and no surrogate to maintain.
Header from line grain — GROUP BY order_number reconstructs the order header from line rows, so you keep the finer grain without losing order-level rollups.
Distinct-count metric — COUNT(DISTINCT order_number) is the order-count KPI a dim_order would exist to provide, delivered directly from the fact.
Avoided join — no surrogate lookup means one fewer table to load and one fewer join at query time, which matters on billion-row facts.
Cost — the distinct-count is O(rows) with a sort/hash on order_number; a dim_order would add O(rows) load work and a join for zero descriptive benefit.

SQL
Topic — degenerate-dimension
Degenerate-dimension and operational-key problems

Practice →

Facts Topic — fact-table Fact-table grain and measure problems

Practice →

4. Role-playing dimensions — one dim, many roles via views

A role-playing dimension is one physical dimension the fact references several times, each time under a different alias

The whole idea in one line: when several foreign keys in a fact point at the same dimension for different reasons, you keep one physical dimension and expose it under one view per role — you never copy the table. The textbook case is dates: an order has an order_date, a ship_date, and a due_date, and all three are just calendar dates. One dim_date holds every date once; the fact carries three separate date_key columns; and three views (dim_order_date, dim_ship_date, dim_due_date) let queries and BI tools join to "the calendar" three times without ambiguity.

Why views, not copies.

A copy duplicates gigabytes and drifts. dim_date can be large (every day for decades, plus fiscal, holiday, and weekday attributes). Copying it three times triples storage and guarantees the copies fall out of sync when you add an attribute.
A view is free and always current. CREATE VIEW dim_ship_date AS SELECT * FROM dim_date is metadata only. Add a column to dim_date and all three roles inherit it instantly.
BI tools need distinct names to disambiguate. Most semantic layers cannot join the same physical table to a fact three times and label the joins differently. Distinct view names (dim_ship_date.day_name) give each role its own unambiguous column set in the model.

How the fact is wired.

One foreign key per role. The fact has order_date_key, ship_date_key, due_date_key — three integer columns, each a normal FK into dim_date (or its view alias).
Each role answers different questions. Join on order_date_key to trend bookings, on ship_date_key for fulfilment, on due_date_key for SLA breaches — same calendar, three lenses.
Cross-role arithmetic is easy. Because all three resolve to real dates, datediff(ship_date, order_date) gives days-to-ship straight from the keys.

Beyond dates.

Role-played dim_employee. A sale can have a sold_by_employee_key and an approved_by_employee_key — one dim_employee, two roles, two views.
Role-played dim_location. Shipments have an origin_location_key and a destination_location_key — one dim_location, two roles.
The rule generalises. Any time two FKs from one fact would point at the same dimension, that dimension is role-playing and each role gets a view.

Worked example — three date roles over one dim_date

Detailed explanation. The standard build is one dim_date, three FK columns on the fact, and three views. Below, an orders fact records when each order was placed, shipped, and due, all against a single calendar, and the views let you name each role explicitly in a query.

Question. Model fact_orders with order_date, ship_date, and due_date as role-playing dimensions over one dim_date, then show days-to-ship per order.

Input.

order_number	order_date	ship_date	due_date
SO-1007	2026-03-01	2026-03-03	2026-03-05
SO-1008	2026-03-02	2026-03-06	2026-03-04

Code.

-- One physical calendar
CREATE TABLE dim_date (
  date_key  INT PRIMARY KEY,       -- e.g. 20260301
  full_date DATE NOT NULL,
  day_name  VARCHAR(9),
  is_weekend BOOLEAN
);

-- Three role views over the SAME table (metadata only, no data copied)
CREATE VIEW dim_order_date AS SELECT * FROM dim_date;
CREATE VIEW dim_ship_date  AS SELECT * FROM dim_date;
CREATE VIEW dim_due_date   AS SELECT * FROM dim_date;

CREATE TABLE fact_orders (
  order_number   VARCHAR(20),
  order_date_key INT REFERENCES dim_date(date_key),
  ship_date_key  INT REFERENCES dim_date(date_key),
  due_date_key   INT REFERENCES dim_date(date_key),
  amount NUMERIC(12,2)
);

-- Query each role by its own view alias
SELECT f.order_number,
       od.full_date AS ordered_on,
       sd.full_date AS shipped_on,
       (sd.full_date - od.full_date) AS days_to_ship
FROM   fact_orders f
JOIN   dim_order_date od ON od.date_key = f.order_date_key
JOIN   dim_ship_date  sd ON sd.date_key = f.ship_date_key;

Step-by-step explanation. dim_date is created once. The three CREATE VIEW statements are pure metadata — no rows are copied — and give each role a distinct name so the join is unambiguous. The fact holds three integer FK columns. In the query, joining dim_order_date and dim_ship_date (both really dim_date) resolves the two roles independently, and subtracting the two dates yields days-to-ship per order.

Output.

order_number	ordered_on	shipped_on	days_to_ship
SO-1007	2026-03-01	2026-03-03	2
SO-1008	2026-03-02	2026-03-06	4

Rule of thumb. One physical dimension, one view per role, one FK per role — if you find yourself copying dim_date, stop and make a view instead.

SQL interview question on role-playing dimensions

Question. Marketing wants to trend orders by the month they were placed, while operations wants the same fact trended by the month each order was shipped. Using one dim_date, show both trends and explain why aliasing the dimension is mandatory here.

Solution Using two aliased joins to the same dim_date

Code.

-- Orders placed, by placement month
SELECT od.year, od.month, COUNT(*) AS orders_placed
FROM   fact_orders f
JOIN   dim_date od ON od.date_key = f.order_date_key   -- role 1
GROUP  BY od.year, od.month;

-- Orders shipped, by shipment month, from the SAME fact and SAME dim_date
SELECT sd.year, sd.month, COUNT(*) AS orders_shipped
FROM   fact_orders f
JOIN   dim_date sd ON sd.date_key = f.ship_date_key    -- role 2
GROUP  BY sd.year, sd.month;

Step-by-step trace.

order	order_date_key (role 1)	ship_date_key (role 2)	counted in placed	counted in shipped
SO-1007	20260301 (Mar)	20260303 (Mar)	Mar	Mar
SO-1008	20260302 (Mar)	20260406 (Apr)	Mar	Apr

The first query joins dim_date on order_date_key, so each order is bucketed by the month it was placed — both orders fall in March.
The second query joins the same dim_date on ship_date_key, so orders are bucketed by ship month — SO-1008 shifts to April because it shipped in April.
Aliasing (od vs sd, or the named views) is mandatory: without a distinct alias the optimizer cannot tell which role you mean, and joining one table twice in one query without aliases is a syntax error. The alias is the role.
Both trends read from one fact and one physical calendar, so bookings and fulfilment reconcile against the same date attributes.

Output:

trend	Mar	Apr
orders_placed	2	0
orders_shipped	1	1

Why this works — concept by concept:

Role-playing dimension — one physical dim_date answers to several roles, so every date attribute (fiscal period, holiday flag) is defined once and reused everywhere.
Alias per role — the table alias or view name is what disambiguates the join; it lets the same dimension attach to the fact multiple times with different meaning.
Views over copies — the role views are metadata, so storage stays flat and adding a dim_date column propagates to every role instantly.
Independent grouping — grouping by the placement key versus the ship key produces genuinely different trends from the same rows, which is the business value of the pattern.
Cost — each aliased join is O(fact rows) with a key lookup into dim_date; there is zero extra storage cost because no dimension data is duplicated.

SQL
Topic — role-playing-dimension
Role-playing-dimension and multi-date problems

Practice →

Schema Topic — star-schema Star-schema join and date-dimension problems

Practice →

5. Conformed dimensions — shared across facts for drill-across

A conformed dimension has identical keys and meaning across facts, which is what makes drill-across correct

The invariant to say out loud: a dimension is conformed when it means exactly the same thing — same keys, same attribute values, same definitions — everywhere it is used, so measures from different fact tables can be combined by that dimension. Conformed dimensions are the backbone of Kimball's enterprise bus architecture: instead of each data mart building its own customer and date tables (which then never agree), every fact across the organisation shares one dim_customer and one dim_date. That shared agreement is precisely what lets you put sales revenue and return volume side by side by customer and trust the result.

What "conformed" actually requires.

Identical surrogate keys. customer_key = 42 must be the same customer in fact_sales and fact_returns. If the keys differ, the dimension is not conformed and any cross-fact join is meaningless.
Identical attribute meaning. customer.segment = 'Enterprise' must mean the same thing to both processes. Conforming is a governance act, not just a shared table name.
Same or a strict subset (shrunken). A conformed dimension can be the full dimension or a shrunken rollup of it. A monthly fact_forecast can conform to dim_date at the month grain — a "shrunken conformed dimension" whose attributes are a strict subset of the daily one, with keys that roll up cleanly.

The bus matrix — how you plan conformance.

Rows are business processes (facts); columns are dimensions. Sales, returns, shipments down the side; customer, date, product, store across the top.
A tick means "this fact uses this conformed dimension." The matrix is the enterprise data-warehouse plan on one page; shared columns are the conformed dimensions every mart must agree on.
Conform first, build incrementally. You agree the conformed dimensions once, then each fact table can be built independently and still integrate, because they all bolt onto the same dimensions.

Drill-across — the payoff.

Never join two facts directly. Joining fact_sales to fact_returns on customer_key multiplies rows (a fan trap) and doubles measures. That is the classic mistake.
Split-and-combine. Query each fact separately, each aggregated to the same conformed grain (e.g. revenue by customer, returns by customer), then full-outer-join the two result sets on the conformed key. Each measure is computed in isolation, so nothing is double-counted.
Common grain is mandatory. Drill-across only works where the facts share conformed dimensions at a compatible grain; that shared grain is what the two aggregates line up on.

Worked example — one dim_customer shared by sales and returns

Detailed explanation. The clearest demonstration is two facts and one shared dimension. fact_sales and fact_returns both reference the same dim_customer, so a question like "net revenue = sales minus returns, by customer segment" becomes answerable and trustworthy. The key move is that both facts use the identical customer_key.

Question. Model fact_sales and fact_returns so they both conform to one dim_customer, then compute sales and returns per customer segment.

Input.

dim_customer	customer_key	segment
—	42	Enterprise
—	43	SMB

Sales rows: (42, 100.00), (43, 30.00). Return rows: (42, 15.00).

Code.

CREATE TABLE dim_customer (
  customer_key BIGINT PRIMARY KEY,     -- conformed surrogate: same in every fact
  customer_id  VARCHAR(20) UNIQUE,
  segment      VARCHAR(20)
);

CREATE TABLE fact_sales (
  customer_key BIGINT REFERENCES dim_customer(customer_key),
  amount NUMERIC(12,2)
);

CREATE TABLE fact_returns (
  customer_key BIGINT REFERENCES dim_customer(customer_key),   -- SAME conformed dim
  amount NUMERIC(12,2)
);

-- Aggregate each fact by the conformed segment attribute
SELECT c.segment, SUM(s.amount) AS sales
FROM   fact_sales s JOIN dim_customer c ON c.customer_key = s.customer_key
GROUP  BY c.segment;

Step-by-step explanation. dim_customer is created once with a single customer_key; both fact tables declare a foreign key to it, so customer_key = 42 is unambiguously the Enterprise customer in both. Aggregating fact_sales by segment gives sales per segment; the identical join against fact_returns gives returns per segment. Because the dimension is shared, the two segment breakdowns line up exactly.

Output.

segment	sales
Enterprise	100.00
SMB	30.00

Rule of thumb. If two facts should ever be compared by a dimension, that dimension must be conformed — build it once, key it once, and point every fact at it.

SQL interview question on conformed dimensions

Question. You must report net revenue (sales − returns) by customer segment. A colleague proposes fact_sales JOIN fact_returns ON customer_key. Explain why that is wrong and write the correct drill-across.

Solution Using drill-across with a full outer join on the conformed key

Code.

-- Aggregate EACH fact separately to the conformed segment grain,
-- then FULL OUTER JOIN the two results on the conformed key.
WITH sales AS (
  SELECT c.segment, SUM(s.amount) AS sales_amt
  FROM   fact_sales s
  JOIN   dim_customer c ON c.customer_key = s.customer_key
  GROUP  BY c.segment
),
returns AS (
  SELECT c.segment, SUM(r.amount) AS returns_amt
  FROM   fact_returns r
  JOIN   dim_customer c ON c.customer_key = r.customer_key
  GROUP  BY c.segment
)
SELECT COALESCE(sa.segment, re.segment)          AS segment,
       COALESCE(sa.sales_amt, 0)                 AS sales,
       COALESCE(re.returns_amt, 0)               AS returns,
       COALESCE(sa.sales_amt, 0) - COALESCE(re.returns_amt, 0) AS net_revenue
FROM   sales sa
FULL OUTER JOIN returns re ON re.segment = sa.segment;

Step-by-step trace.

segment	sales_amt (from sales CTE)	returns_amt (from returns CTE)	net
Enterprise	100.00	15.00	85.00
SMB	30.00	(none) → 0	30.00

Directly joining the two facts on customer_key would pair every sale of a customer with every return of that customer — a fan-out that multiplies rows and inflates both SUM(sales) and SUM(returns). That is the fan trap.
Drill-across avoids it: each fact is aggregated on its own to the conformed segment grain first, so each measure is summed exactly once.
The FULL OUTER JOIN on the conformed attribute stitches the two independent aggregates together and keeps segments that appear in only one fact (SMB has sales but no returns).
COALESCE(..., 0) turns the missing side into zero so net_revenue = sales − returns is correct even when one fact has no rows for a segment.

Output:

segment	sales	returns	net_revenue
Enterprise	100.00	15.00	85.00
SMB	30.00	0.00	30.00

Why this works — concept by concept:

Conformed dimension — because dim_customer and its segment attribute mean the same thing in both facts, the two aggregates share a common grain that can be lined up.
Drill-across — aggregating each fact separately and joining the results (not the facts) is the only combination method that keeps every measure summed exactly once.
Fan-trap avoidance — joining facts directly multiplies rows through the shared key; the split-and-combine shape sidesteps it entirely.
Full outer join + COALESCE — preserves segments present in only one process and defaults the missing measure to zero, so net revenue is defined for every segment.
Cost — two independent O(rows) aggregations plus a join on the small segment result set; far cheaper and correct versus the O(sales × returns) blow-up of a direct fact join.

SQL
Topic — conformed-dimension
Conformed-dimension and drill-across problems

Practice →

Modelling Topic — dimensional-modeling Bus-matrix and enterprise-modelling problems

Practice →

Cheat sheet — special-dimension recipes

Junk dimension (observed combinations).

CREATE TABLE dim_order_junk (
  junk_key BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  is_gift BOOLEAN, is_priority BOOLEAN, channel VARCHAR(10),
  UNIQUE (is_gift, is_priority, channel)
);
INSERT INTO dim_order_junk (is_gift, is_priority, channel)
SELECT DISTINCT is_gift, is_priority, channel FROM stg_orders
ON CONFLICT DO NOTHING;

Degenerate dimension (key on the fact, no dim table).

-- order_number is a plain column; NO dim_order, NO surrogate
SELECT order_number, SUM(extended_amount) AS order_total
FROM   fact_sales_line
GROUP  BY order_number;

Role-playing dimension (views over one dim_date).

CREATE VIEW dim_order_date AS SELECT * FROM dim_date;
CREATE VIEW dim_ship_date  AS SELECT * FROM dim_date;
-- fact has order_date_key, ship_date_key, due_date_key -> all -> dim_date

Conformed dimension (one dim, many facts).

-- same customer_key in every fact that references dim_customer
CREATE TABLE fact_returns (
  customer_key BIGINT REFERENCES dim_customer(customer_key),
  amount NUMERIC(12,2)
);

Drill-across skeleton (never join two facts directly).

WITH a AS (SELECT dim, SUM(m1) m1 FROM fact_a JOIN d USING(dim_key) GROUP BY dim),
     b AS (SELECT dim, SUM(m2) m2 FROM fact_b JOIN d USING(dim_key) GROUP BY dim)
SELECT COALESCE(a.dim,b.dim) dim, COALESCE(a.m1,0), COALESCE(b.m2,0)
FROM a FULL OUTER JOIN b ON a.dim = b.dim;

Pattern picker.

Symptom	Special dimension
Several unrelated low-cardinality flags	Junk dimension
Operational key (order_number) with no attributes	Degenerate dimension
Several FKs to the same dimension (order/ship/due date)	Role-playing dimension
One dimension shared across facts / marts	Conformed dimension

Frequently asked questions

What are special dimensions in dimensional modeling?

Special dimensions are four named patterns in Kimball dimensional modeling that handle attributes which do not fit the plain "one dimension per business entity" star schema: the junk dimension (folds unrelated low-cardinality flags into one table), the degenerate dimension (an operational key stored on the fact with no dimension table), the role-playing dimension (one physical dimension referenced multiple times under different aliases), and the conformed dimension (one dimension shared identically across many fact tables). They are refinements of the star schema, not exceptions to it — grain and surrogate keys still govern the model.

When should I use a junk dimension?

Use a junk dimension when you have several low-cardinality, unrelated indicators — booleans like is_gift or short enumerations like channel and payment_type — that each would make a trivial one-column dimension, and that would bloat the fact if left as text columns. You fold them into a single small table keyed by a surrogate junk_key, populated with the combinations that actually occur. Do not junk high-cardinality attributes (free text, ids) or attributes the business analyses independently at large scale.

What is a degenerate dimension and where is it stored?

A degenerate dimension is a dimension key that has no attributes and therefore no dimension table — it is stored directly as a column in the fact table. The classic examples are transaction identifiers like order_number, invoice_number, or ticket_id. You still group by it (to reconstruct an order header from line rows) and count distinct values of it (to count orders from a line-grain fact), but building a one-column dimension for it would be pure overhead, so the natural key stays in the fact.

How do role-playing dimensions work without duplicating data?

You keep one physical dimension table and create a database view per role — for example, one dim_date with three views dim_order_date, dim_ship_date, and dim_due_date. The fact carries a separate foreign key for each role (order_date_key, ship_date_key, due_date_key), and each query or BI join uses the appropriately named view. Views are metadata only, so no data is copied, and adding a column to dim_date instantly appears in every role.

What makes a dimension conformed?

A dimension is conformed when it means exactly the same thing everywhere it is used: identical surrogate keys, identical attribute values, and identical definitions across every fact table that references it. A shrunken (rollup) version — for example dim_date at the month grain for a forecast fact — is still conformed if its keys and attributes are a strict subset that rolls up cleanly. Conforming is a governance decision that lets measures from different facts be combined by that dimension.

What is drill-across and why does it need conformed dimensions?

Drill-across is the technique for combining measures from two different fact tables by a shared dimension — for example net revenue = sales minus returns by customer segment. You must not join the two facts directly (that fans out rows and double-counts measures); instead you aggregate each fact separately to the same conformed grain, then full-outer-join the two result sets on the conformed key. It only works when the dimension is conformed, because the two aggregates must line up on identical keys and attribute meaning.

Practice on PipeCode

Pipecode.ai is Leetcode for Data Engineering — every special-dimension idea above, from folding flags into a junk dimension to storing a degenerate order key, aliasing one date dimension across roles, and drilling across conformed dimensions, maps to a hands-on practice room where you write the DDL and the query against real graded inputs. PipeCode pairs each reading with 450+ DE-focused problems and a real-time scoring engine, so your answer to "how would you model these three dates against one calendar?" holds up under a senior interviewer's depth probes.

Practice junk-dimension problems now →
Conformed-dimension drills →

DEV Community: Gowtham Potureddi

Prophecy: Visual, Git-Backed Low-Code Spark & SQL Pipelines for the Enterprise

1. Why Prophecy changes enterprise pipeline building in 2026

Prophecy is a code generator wearing a visual UI — that one fact decides where it fits

Worked example — three boxes that become a real Spark job

2. Gems, pipelines & the visual canvas

A gem is one transform node, a pipeline is a DAG of gems, and each gem compiles to a function

Worked example — a Filter gem and a Reformat gem in one pipeline

Prophecy interview question on the gem/pipeline model

Solution Using a single projection with column expressions

3. The visual ↔ code round-trip on Git

Every visual edit regenerates readable code, and the whole project lives in your Git repo

Worked example — an Aggregate gem and its Git diff

Prophecy interview question on aggregation transforms

Solution Using groupBy with a filtered aggregate

4. Running on Spark & Databricks with Fabrics

A Fabric binds a pipeline to where it runs — Spark on a cluster, or SQL on a warehouse

Worked example — the same join as Spark and as SQL

Prophecy interview question on deduplicating with Spark SQL

Solution Using ROW_NUMBER over a partition

5. Reusable subgraphs, tests & column-level lineage

Package gems into reusable subgraphs, unit-test them, and trace every column

Worked example — a reusable "standardize customer" subgraph

Prophecy interview question on testing a transform

Solution Using a gem unit test with expected output

Cheat sheet — Prophecy recipes

Frequently asked questions

What is Prophecy?

Does Prophecy lock me into its runtime?

What are gems in Prophecy?

How does the visual-to-code round-trip work?

Does Prophecy support SQL as well as Spark?

Can Prophecy migrate legacy Informatica or Alteryx pipelines?

Practice on PipeCode

Bruin: SQL + Python Pipelines in One Framework With Built-In Quality Checks

1. Why Bruin puts the whole pipeline in one framework

Bruin is one tool for SQL, Python, ingestion, and quality — that single fact decides where it fits

Worked example — one asset that materializes a table

2. Assets: SQL, Python & YAML-in-comment metadata

The asset — code plus its YAML metadata in one file — is Bruin's entire mental model

Worked example — a Python asset that depends on a SQL asset

Bruin interview question on the asset model

Solution Using inline YAML metadata and path-based name inference

3. Materialization: how a SELECT becomes a table

Bruin wraps your SELECT in the load strategy you choose — from full refresh to merge and SCD2

Worked example — an incremental merge on a primary key

Bruin interview question on tracking history

Solution Using the scd2_by_column strategy

4. Built-in & custom quality checks

Quality checks live in the asset file and run right after it — a blocking failure stops the DAG

Worked example — column checks on a keyed table

Bruin interview question on business-rule validation

Solution Using a blocking custom check

5. Ingestion with ingestr & multi-destination DAGs

An ingestr asset lands raw data, SQL and Python transform it, and bruin run executes the whole DAG

Worked example — an ingestr asset feeding a SQL transform

Bruin interview question on orchestrating a mixed pipeline

Solution Using depends-driven topological execution and bruin validate

Cheat sheet — Bruin recipes

Frequently asked questions

What is Bruin?

How is Bruin different from dbt?

How do Bruin quality checks work?

What materialization strategies does Bruin support?

What is ingestr and how does Bruin use it?

Do I need an orchestrator to run Bruin?

Practice on PipeCode

A Local Lakehouse on Your Laptop: DuckDB + Iceberg + Trino for Zero-Cloud Dev

1. Why a local lakehouse beats cloud dev loops in 2026

A lakehouse is three swappable layers — table format, storage, engine — and every one of them now runs on a laptop

Worked example — the whole loop in two DuckDB statements

2. DuckDB: reading and writing Parquet on the local filesystem

read_parquet, COPY ... PARTITION_BY, and the iceberg extension make DuckDB the fast local reader and Parquet producer

Worked example — write a partitioned Parquet dataset, then read one partition

DuckDB interview question on Parquet pushdown

Solution Using hive partitioning with projection and predicate pushdown

3. A local Iceberg catalog on the filesystem or MinIO

The catalog is what turns a folder of Parquet into a table — SQLite for pure-filesystem, a REST catalog for sharing

Worked example — create an Iceberg table with pyiceberg on a SQLite catalog

Iceberg interview question on catalog choice

`read_parquet`, `COPY ... PARTITION_BY`, and the `iceberg` extension make DuckDB the fast local reader and Parquet producer

Trino's `iceberg` connector reads the catalogued table directly — same metadata, distributed SQL, time travel

Hidden partitioning plus `EXECUTE optimize` fixes the small-files problem locally — the same maintenance you run in production