Bauplan: Git-Native, Function-as-a-Pipeline Lakehouse Compute in Python

Bauplan is what you get when someone looks at the modern data stack — a Spark cluster for compute, dbt for transformations, a separate catalog like Nessie or lakeFS for data version control, an orchestrator to wire it all together — and decides to collapse the whole assembly into a single programmable runtime. You write your pipeline as ordinary Python functions, each function a node in a directed acyclic graph, and the platform runs them serverless over Iceberg tables in your own object storage. There is no cluster to size, no separate transformation framework to learn, and no bolt-on versioning system to keep in sync — the function-as-a-pipeline programming model and the compute engine are the same thing.

The second idea Bauplan folds in is the one data teams have wanted for a decade: git-native data. Every table lives under version control, so you can create a zero-copy branch, run a pipeline against it, audit the result with a query, and only then merge it into main — the write-audit-publish discipline, built into the runtime instead of scripted around it. That means data versioning, time travel to any previous commit, and rollback are first-class, not an afterthought bolted onto a lakehouse that never expected them. This guide is the senior-engineering walkthrough of how those two ideas — functions-as-a-pipeline and git-for-data — fit together over an open Iceberg lakehouse, framed the way a sharp interviewer probes a new platform: what problem does it actually collapse, how does the execution model work, and when does the assembled stack still win. Each section pairs a teaching block with a worked interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the ETL pipeline practice library →, rehearse the transformation logic on the data transformation practice library →, and sharpen the platform-design axis with the system design practice library →.

---

On this page

• Why Bauplan and the gap it fills
• Function-as-a-pipeline — Python functions as DAG nodes
• Git-native data — branch, commit, merge on tables
• Lakehouse execution — Iceberg, serverless, caching
• Bauplan vs dbt, Spark, and lakeFS — and CI
• Cheat sheet — Bauplan pipeline recipes
• Frequently asked questions
• Practice on PipeCode

---

1. Why Bauplan and the gap it fills

One runtime for code-first compute and data version control, instead of four tools glued together

The one-sentence invariant: Bauplan is a serverless lakehouse platform whose single runtime does three jobs the modern stack normally splits across four tools — it runs Python-function pipelines as its compute engine, it version-controls the underlying Iceberg tables with git semantics (branch, commit, merge, time travel), and it manages the environments and the DAG for you — so the reason it exists is to collapse "Spark for compute + dbt for transforms + Nessie/lakeFS for data versioning + an orchestrator to bind them" into one code-first surface where a pipeline is just decorated functions and safe data changes are just branches. Get the mental model right and a whole category of glue code, cluster ops, and cross-tool drift disappears; miss it and you will keep reaching for the assembled stack out of habit.

A quick honesty note before the code: the SDK and CLI signatures throughout this guide follow Bauplan's public documentation pattern, but treat the exact argument names as illustrative — the concepts (functions-as-nodes, bauplan.Model dependencies, branch/run/audit/merge) are stable; a specific keyword may have moved by the release you install. Check the current SDK when you build.

The four axes to probe any new lakehouse platform.

• Programming model. Do you express transformations in SQL models or in code? dbt made SQL-plus-Jinja the default; Bauplan bets on Python functions, so a node can do anything Python can — call a model, parse a weird format, use pandas or Arrow — while still being a declarative DAG node. The senior question is not "which is better" but "which fits the transformation and the team."
• Execution model. Is there a cluster you size and babysit, or is compute serverless? Spark makes you own a cluster's memory, autoscaling, and failure modes; Bauplan runs each node in an isolated serverless container with nothing to provision. The axis is operational ownership.
• Data version control. Is versioning native or bolt-on? A lakehouse without branching forces you to write straight to production tables and hope; git-native branching (write-audit-publish) makes bad data a discarded branch instead of an incident. The axis is whether the platform treats data changes like code changes.
• Table format and lock-in. Is the storage an open format you can read with other engines, or proprietary? Bauplan stores Iceberg in your own object storage, so the tables outlive the vendor. The axis is exit cost.

The 2026 reality — the pieces Bauplan assumes.

• Iceberg is the table format. Open table formats won; a serving-grade lakehouse is expected to store Iceberg (or an equivalent) in object storage you control, readable by Spark, Trino, DuckDB, and friends. Bauplan does not invent a format — it operates on the open one.
• FaaS removes cluster ops. Function-as-a-service runtimes made "no cluster to size" a realistic default; a per-node serverless container that spins up, runs a Python function, and disappears is the compute unit.
• Git-for-data is table stakes. After lakeFS and Nessie proved branch/merge on data is possible, teams now expect write-audit-publish as a first-class workflow, not a shell script wrapping S3 prefixes.
• Python is the transformation language. The gravity of pandas, Arrow, and the ML ecosystem means a lot of transformation logic wants to be code, not SQL — and a platform that makes a Python function a DAG node meets that where it lives.

What interviewers listen for.

• Do you name what Bauplan collapses — compute, transforms, and data versioning into one runtime — rather than describing it as "another Spark"? — senior signal.
• Do you explain why functions instead of SQL models, and when SQL is still the better fit? — required answer.
• Do you locate data version control as native branching, not a separate catalog? — senior signal.
• Do you name the open Iceberg format as the lock-in escape hatch? — required answer.
• Do you volunteer when NOT to pick it (petabyte shuffles, heavy dbt investment, streaming)? — senior signal.

Worked example — the pick-Bauplan decision table

Detailed explanation. The most useful artifact for a "should we adopt Bauplan" discussion is a mapping from a team's situation to a verdict. Every serious evaluation converges on it: given your transformation language, your ops appetite, and your versioning needs, does Bauplan collapse real complexity or add a dependency you do not need? Walk the axes for a mid-size analytics team standing up a new lakehouse.

• The situation. A team on raw Iceberg-in-S3, writing transforms in a mix of Python and SQL, currently scripting "backfill safely" by hand and running a small Spark cluster nobody enjoys operating.
• The tension. More tools means more integration surface; one runtime means one vendor dependency. The question is which trade the team is better off making.
• The rule. Adopt the consolidation when the pain is cluster ops plus unsafe data changes plus glue; keep the assembled stack when a single axis (e.g. petabyte Spark shuffles) dominates.

Question. For each axis, state what the assembled stack costs and what Bauplan replaces it with, then give the verdict.

Input.

Axis	Assembled stack	Bauplan
Compute	Spark cluster you size/operate	serverless, no cluster
Transforms	dbt SQL models (+ Python islands)	Python functions as DAG nodes
Data versioning	Nessie / lakeFS bolt-on	git-native branches, built in
Orchestration	Airflow wiring the pieces	DAG inferred from function args
Table format	Iceberg (open)	Iceberg (open)

Code.

Pick-Bauplan decision (read top to bottom; first strong match wins)
==================================================================

Cluster ops is a real cost AND you want code-first transforms
  -> Bauplan collapses compute + transforms into one serverless runtime.

You need safe data changes (backfills, corrections) without incidents
  -> Bauplan's git-native branching gives write-audit-publish for free.

Your team writes transforms in Python (pandas / Arrow / ML), not just SQL
  -> function-as-a-pipeline fits the language the logic already lives in.

Your data is already open Iceberg in your own object storage
  -> Bauplan operates in place; low lock-in, easy exit.

BUT: petabyte shuffles / massive joins are the dominant workload
  -> keep Spark; a serverless per-node runtime is not a distributed-shuffle engine.

BUT: you have a large, healthy dbt+SQL investment and no ops pain
  -> the migration cost outweighs the consolidation benefit. Stay.

Step-by-step explanation.

1. The first three rows of the input table are the cost side: a Spark cluster is a standing operational liability, dbt-plus-Python-islands is two transformation paradigms to maintain, and a bolt-on versioning catalog is a second system to keep synchronized with the tables. Bauplan's pitch is that one runtime removes all three at once.
2. The decision text reads as a priority list: the strongest adoption signal is cluster ops pain plus a desire for code-first transforms, because that is exactly the pair Bauplan folds together — you stop operating Spark and you stop bolting Python onto SQL.
3. The second signal is unsafe data changes. If your team backfills by writing to production tables and praying, git-native branching is the feature that converts a scary operation into a discardable branch — a qualitative safety upgrade, not a marginal one.
4. The BUT clauses are the senior part of the answer: a serverless per-node runtime is not a distributed-shuffle engine, so a petabyte join workload stays on Spark; and a healthy dbt estate with no ops pain does not justify a migration. Naming these unprompted is what separates evaluation from enthusiasm.
5. The verdict is situational, not universal: Bauplan wins when it collapses several real costs simultaneously, and loses when a single axis it does not specialize in dominates the workload.

Output.

Team situation	Verdict	Why
Cluster ops pain + Python transforms + unsafe backfills	adopt Bauplan	collapses three costs into one runtime
Petabyte shuffles dominate	keep Spark	not a distributed-shuffle engine
Large healthy dbt/SQL estate, no ops pain	stay	migration cost > benefit
New team on open Iceberg, wants git-for-data	adopt Bauplan	native branching + low lock-in

Rule of thumb. Adopt Bauplan when it collapses several real costs at once — cluster ops, split transform paradigms, and unsafe data changes — and keep the assembled stack when a single axis it does not specialize in (petabyte shuffles) or a sunk investment (a healthy dbt estate) dominates. The consolidation, not any one feature, is the pitch.

Worked example — what interviewers actually probe about Bauplan

Detailed explanation. A "tell me about Bauplan" interview has a predictable escalation: an open framing question, then progressive narrowing to test whether you understand the consolidation, the execution model, and the honest limits. Candidates who name what it collapses, why functions, and when not to use it score highest.

• Open framing. "What is Bauplan, in one sentence?"
• Follow-up 1. "Why functions instead of dbt SQL models?" — probes the programming model.
• Follow-up 2. "What does 'git-native data' actually give me?" — probes write-audit-publish.
• Follow-up 3. "Isn't this just serverless Spark?" — probes the execution model and limits.
• Follow-up 4. "When would you not use it?" — probes judgment.

Question. Draft a senior answer that pre-empts all four follow-ups without waiting to be asked.

Input.

Interview signal	Weak answer	Senior answer
What it is	"a lakehouse tool"	"one runtime for code compute + git-for-data over Iceberg"
Why functions	"Python is nicer"	"the logic already lives in Python; a function is a DAG node"
Git-native	"you can branch"	"write-audit-publish: bad data is a discarded branch"
Vs Spark	"it's faster"	"serverless per-node, not a distributed-shuffle engine"
When not	"always use it"	"petabyte shuffles, heavy dbt estate, streaming"

Code.

Senior Bauplan answer template (~4 minutes)
===========================================

Minute 1 — what it collapses
  "Bauplan is a serverless lakehouse runtime: your pipeline is Python
   functions that form a DAG, run with no cluster, over Iceberg tables
   you own — and those tables are git-versioned, so branch/commit/merge
   works on data. It folds compute, transforms, and data versioning
   into one thing instead of Spark + dbt + Nessie/lakeFS."

Minute 2 — why functions
  "A function is a DAG node; its arguments declare its upstreams. The
   transform logic already wants to be Python (pandas/Arrow/ML), so the
   node IS the code, not SQL wrapped in Jinja. SQL still fits pure-SQL
   marts — this fits code-first pipelines."

Minute 3 — git-native = write-audit-publish
  "I create a zero-copy branch, run the pipeline onto it, audit it with
   a query, and merge into main only if it passes. Bad data never lands
   in production — it's a branch I delete. Plus time travel and rollback."

Minute 4 — limits (say them before you're asked)
  "It's serverless per-node, not a distributed-shuffle engine, so
   petabyte joins stay on Spark. A big healthy dbt estate isn't worth
   migrating. And it's batch — streaming is a different tool."

Step-by-step explanation.

1. Minute 1 frames the entire answer around consolidation. Weak candidates call it "a lakehouse tool," which describes nothing; naming the four tools it collapses (Spark, dbt, Nessie/lakeFS, orchestrator) shows you understand the category, not just the product.
2. Minute 2 answers the programming-model question by locating the transform logic where it already lives — in Python — and reframing "a function" as "a DAG node whose args are its dependencies," which is the single most important mechanical idea in the platform.
3. Minute 3 makes "git-native" concrete as write-audit-publish, the workflow that turns a scary backfill into a discardable branch. Saying "bad data never lands in production" is the sentence that signals you have run data changes under pressure.
4. Minute 4 volunteers the limits before the interviewer digs for them — serverless-per-node is not distributed shuffle, a healthy dbt estate is not worth migrating, and it is batch not streaming. Naming your own tool's boundaries is the strongest credibility move you can make.
5. The whole monologue takes four minutes and pre-empts every follow-up, which is exactly what a senior candidate does: answer the question and the three questions behind it.

Output.

Grading criterion	Weak score	Senior score
Names what it collapses	rare	mandatory
Functions-as-DAG-nodes	occasional	mandatory
Write-audit-publish framing	rare	senior signal
Serverless ≠ distributed shuffle	rare	senior signal
Volunteers when-not-to-use	rare	senior signal

Rule of thumb. The senior Bauplan answer is a four-minute monologue: what it collapses, why functions are DAG nodes, how git-native branching gives write-audit-publish, why serverless-per-node is not Spark, and when not to use it. Rehearse it once; it pre-empts every follow-up.

Worked example — functions vs SQL models for the same transform

Detailed explanation. A common probe is "if dbt already models this in SQL, what does a Bauplan function buy me?" The weak answer is "Python is nicer." The senior answer contrasts the two on where the logic lives, how dependencies are declared, and what each can express — using the same transform in both idioms.

• The dbt idiom. A SQL model file; dependencies declared with ref('upstream'); the transform is whatever SQL can express.
• The Bauplan idiom. A Python function; dependencies declared as bauplan.Model('upstream') arguments; the transform is whatever Python can express.
• The decision. SQL for pure set-based marts a team already maintains in SQL; functions when the logic wants pandas/Arrow/ML or arbitrary Python.

Question. Contrast a dbt SQL model and a Bauplan function computing the same "clean, paid orders" step on dependency declaration and expressive power.

Input.

Dimension	dbt SQL model	Bauplan function
Unit	a .sql file	a decorated Python function
Dependency	ref('raw_orders')	bauplan.Model('raw.orders') arg
Transform language	SQL (+ Jinja)	any Python (pandas, Arrow, libs)
Output	a table/view materialization	a returned frame → Iceberg table
Best fit	pure set-based marts	code-first / ML-adjacent logic

Code.

-- dbt: models/clean_orders.sql — dependency via ref(), logic is SQL.
SELECT order_id, customer_id, amount_cents, ts
FROM {{ ref('raw_orders') }}
WHERE status = 'paid'

# Bauplan: the SAME step as a function. The arg IS the dependency;
# the body can be any Python, not only SQL. (Signatures illustrative.)
import bauplan

@bauplan.model()
@bauplan.python('3.11', pip={'pandas': '2.2.0'})
def clean_orders(
    orders=bauplan.Model(
        'raw.orders',
        columns=['order_id', 'customer_id', 'amount_cents', 'status', 'ts'],
    )
):
    df = orders.to_pandas()          # Arrow -> pandas
    df = df[df['status'] == 'paid']  # could just as easily call a model, parse, enrich
    return df[['order_id', 'customer_id', 'amount_cents', 'ts']]

Step-by-step explanation.

1. In dbt, the dependency is the {{ ref('raw_orders') }} call: dbt parses it to build the DAG, and the model is the SQL. Bauplan's parallel is bauplan.Model('raw.orders') as a function argument — the platform reads the arguments to build the DAG, and the model is the function.
2. The expressive difference is the point: the dbt model can only do what SQL (plus Jinja) can express, while the Bauplan function body is arbitrary Python — the same filter here, but tomorrow a call to a scoring model, a parse of a nested format, or an Arrow-compute join, all inside a node.
3. Both declare dependencies declaratively — you never hand-wire the DAG — so you keep dbt's best property (the graph comes from the references) while gaining a general-purpose language for the node body.
4. The columns=[...] argument on bauplan.Model is projection pushdown: the source scan reads only those columns, the code-first equivalent of a lean SELECT list, so "functions" does not mean "read everything into Python."
5. The senior framing is that this is not SQL-versus-Python tribalism: pure set-based marts a team already runs in SQL are perfectly happy in dbt, and the function idiom earns its place precisely when the logic wants to be code — which is a large and growing share of real pipelines.

Output.

Question	dbt SQL model	Bauplan function
"How is the DAG built?"	from ref() calls	from bauplan.Model args
"Can it call an ML model mid-step?"	awkward (Python island)	yes, it's just Python
"Can it stay pure SQL?"	yes, natively	yes, via a SQL/Arrow step
"Who materializes the output?"	dbt	Bauplan (frame → Iceberg)

Rule of thumb. Read a Bauplan function as "a dbt model whose body is arbitrary Python and whose ref() is a bauplan.Model argument." Keep SQL for pure set-based marts you already maintain; reach for functions when the transform wants pandas, Arrow, or a library call — the DAG stays declarative either way.

Senior interview question on the Bauplan value proposition

A senior interviewer often opens with: "A team runs a small Spark cluster nobody wants to operate, transforms in a mix of dbt SQL and ad-hoc Python, and backfills by writing straight to production Iceberg tables and hoping. They ask whether Bauplan would help. Make the case: what it collapses, how the programming and execution models change, what git-native data buys them operationally, and — honestly — when you would tell them to keep what they have."

Solution Using one runtime for compute, transforms, and versioning over open Iceberg

# 1. The consolidation: four tools -> one runtime (what actually changes).
BEFORE                                  AFTER (Bauplan)
Spark cluster (compute + ops)     ->    serverless runtime, no cluster
dbt SQL models  + Python islands  ->    Python functions as DAG nodes
Nessie/lakeFS (data versioning)   ->    git-native branches, built in
Airflow (wire the pieces)         ->    DAG inferred from function args
Iceberg in your S3 (open)         ->    unchanged — still open Iceberg

# 2. The pipeline is just functions; args declare the DAG. (Illustrative.)
import bauplan

@bauplan.model()
@bauplan.python('3.11', pip={'pandas': '2.2.0'})
def clean_orders(orders=bauplan.Model('raw.orders', columns=['order_id','amount_cents','status'])):
    df = orders.to_pandas()
    return df[df['status'] == 'paid']

@bauplan.model()
@bauplan.python('3.11', pip={'pyarrow': '16.0.0'})
def revenue_by_day(clean=bauplan.Model('clean_orders')):
    import pyarrow.compute as pc
    t = clean.to_arrow()
    # ... group + sum in Arrow, return a table -> becomes an Iceberg table
    return t

# 3. Safe change = a branch, not a prayer. Write-audit-publish.
bauplan branch create ingest.2026-06-01 --from main   # zero-copy branch
bauplan checkout ingest.2026-06-01
bauplan run                                            # pipeline writes onto the BRANCH
bauplan query "SELECT count(*) FROM revenue_by_day WHERE revenue_cents < 0"  # audit
# ...only if the audit returns 0:
bauplan branch merge ingest.2026-06-01 --into main     # publish; else delete the branch

Step-by-step trace.

Decision	Before (assembled stack)	After (Bauplan)
Compute	operate a Spark cluster	serverless, nothing to size
Transforms	dbt SQL + Python islands	one idiom: Python functions
DAG	Airflow / manual wiring	inferred from function args
Safe backfill	write to prod + hope	run on a branch, audit, merge
Versioning	Nessie/lakeFS to maintain	native branch/commit/merge
Lock-in	open Iceberg	open Iceberg (unchanged)

After adoption, the pipeline is a handful of decorated Python functions whose arguments form the DAG; it runs serverless with no cluster to operate; and every risky data change becomes a zero-copy branch that is run, audited with a plain query, and merged into main only when it passes — so a bad backfill is a deleted branch instead of a production incident. The one thing that does not change is the storage: the tables are still open Iceberg in the team's own object storage, so the exit door stays wide open.

Output:

Metric	Assembled stack	Bauplan
Tools to operate	Spark + dbt + lakeFS + Airflow	one runtime
Cluster ops	continuous	none (serverless)
Transform paradigms	SQL + Python islands	Python functions
Backfill safety	write-and-hope	branch, audit, merge
Storage lock-in	open Iceberg	open Iceberg

Why this works — concept by concept:

• One runtime, four jobs — folding compute, transforms, DAG, and data versioning into a single surface removes the integration seams (and the drift between them) that make the assembled stack expensive to operate, not just to build.
• Functions as DAG nodes — declaring dependencies as bauplan.Model arguments keeps the graph declarative like dbt's ref(), while letting each node be arbitrary Python, so the transform lives in the language the logic already wants.
• Serverless execution — per-node isolated containers with nothing to provision remove the standing operational liability of a cluster, which for many teams is the single biggest cost the consolidation eliminates.
• Git-native write-audit-publish — a zero-copy branch that you run, audit, and merge only on green turns unsafe production writes into a discardable branch, making data changes as reviewable as code changes.
• Cost — you trade four systems and their glue for one runtime and one vendor dependency, over storage that stays open Iceberg. The eliminated cost is cluster ops plus cross-tool synchronization plus write-and-hope risk — O(1) tool to operate instead of O(4), with the exit cost bounded by the open format.

Design
Topic — design
Design problems on lakehouse and pipeline platforms

Practice →

ETL Topic — etl ETL problems on building and wiring pipelines

Practice →

---

2. Function-as-a-pipeline — Python functions as DAG nodes

Decorated functions are DAG nodes; their arguments declare the upstreams; Arrow flows between them

The mental model in one line: in Bauplan's function-as-a-pipeline model each transformation is a decorated Python function that is a node in the DAG — the platform reads the function's arguments (each a bauplan.Model('upstream') reference) to infer the edges, runs the function in an isolated serverless container with its own declared pip environment, passes data between nodes as Apache **Arrow with no serialization tax, and materializes whatever frame the function returns as an Iceberg table — so you never hand-wire a DAG, never manage a cluster-wide environment, and never pay a copy to move data downstream; the code is the graph.** Write the functions and declare the arguments; the graph, the environments, and the data hand-off are the platform's job.

How a function becomes a node.

• The decorators. @bauplan.model() marks the function as a materialized DAG node (its return becomes a table), and @bauplan.python(version, pip={...}) pins the interpreter and dependencies for that node — so declaration of what it is and what it needs sits right on the function.
• Arguments declare dependencies. Each parameter defaults to a bauplan.Model('name') reference; the platform reads these to build edges, exactly the way dbt reads ref(). revenue(clean=bauplan.Model('clean_orders')) means "revenue depends on clean_orders."
• Source models with pushdown. bauplan.Model('raw.orders', columns=[...], filter="...") pushes projection and predicate into the Iceberg scan, so a node reads only the columns and rows it needs — lean input, not a full-table load.
• Return = materialization. Return a pandas DataFrame or a pyarrow Table and the platform writes it as an Iceberg table under the model's name; downstream nodes reference that name.

Arrow-native data passing.

• Why Arrow. Apache Arrow is a columnar in-memory format shared across pandas, Polars, DuckDB, and the Iceberg readers, so handing a table from one node to the next needs no re-serialization — the bytes are already in the lingua franca.
• to_pandas() / to_arrow(). A source model hands you an Arrow table; convert to pandas when you want pandas semantics, or stay in Arrow (pyarrow.compute) for zero-copy, vectorized work on large frames.
• The cost you avoid. In a stitched stack, moving data between a SQL step and a Python step often means writing to storage and re-reading — Arrow between nodes removes that round-trip on intermediate data.

Per-function environments.

• Isolation. Each node declares its own pip={...}; node A can use pandas 2.2 and node B can use a specific pyarrow, without a cluster-wide dependency truce — the difference between reproducible nodes and dependency hell.
• Reproducibility. Pinned versions per node mean a run is deterministic in its dependencies; a node's environment is part of its definition, not a shared mutable cluster state.
• Small blast radius. A dependency change in one node cannot break another, because they never share an interpreter — the same isolation property that makes serverless functions safe.

The failure modes senior engineers pre-empt.

• Hidden side effects. A node that reads or writes outside the DAG (a stray S3 write, an API call with state) breaks reproducibility and caching. Mitigation: keep nodes pure functions of their declared inputs; push side effects to explicit boundary steps.
• Monolith functions. One 400-line node doing five transforms is unobservable and uncacheable at the right granularity. Mitigation: one responsibility per node, so the DAG is legible and caching is effective.
• Environment drift. Unpinned pip ranges make a node's behavior depend on when it ran. Mitigation: pin exact versions per node so the environment is part of the contract.

Common interview probes on the programming model.

• "How does Bauplan know the DAG?" — from the bauplan.Model arguments of each function; the code is the graph.
• "How does data move between nodes?" — as Apache Arrow, no re-serialization; return a frame and it becomes an Iceberg table.
• "How do you avoid dependency conflicts?" — per-function pinned environments; nodes never share an interpreter.
• "How do you keep a node cacheable?" — make it a pure function of declared inputs; no hidden I/O.

Worked example — a two-model pipeline with an inferred DAG

Detailed explanation. The canonical Bauplan pipeline: a source model cleaned by one function, then aggregated by a second whose argument names the first — so the DAG edge clean_orders → revenue_by_region is inferred, never declared. Build it and read off the graph.

• Node 1. clean_orders depends on raw.orders (a source, with pushdown).
• Node 2. revenue_by_region depends on clean_orders and raw.regions.
• The DAG. raw.orders → clean_orders → revenue_by_region ← raw.regions, inferred from arguments.

Question. Write a two-node pipeline where the second node depends on the first, and show that the DAG comes entirely from the function arguments.

Input.

Node	Arguments (dependencies)	Output
clean_orders	raw.orders	cleaned paid orders
revenue_by_region	clean_orders, raw.regions	revenue per region
DAG edges	inferred from the args	3 edges, 0 hand-wired

Code.

import bauplan

@bauplan.model()
@bauplan.python('3.11', pip={'pandas': '2.2.0'})
def clean_orders(
    orders=bauplan.Model(
        'raw.orders',
        columns=['order_id', 'region_id', 'amount_cents', 'status'],
        filter="status = 'paid'",          # predicate pushed into the Iceberg scan
    )
):
    df = orders.to_pandas()
    return df[['order_id', 'region_id', 'amount_cents']]   # this frame -> an Iceberg table

@bauplan.model()
@bauplan.python('3.11', pip={'pandas': '2.2.0'})
def revenue_by_region(
    clean=bauplan.Model('clean_orders'),    # EDGE: depends on clean_orders
    regions=bauplan.Model('raw.regions', columns=['region_id', 'region_name']),
):
    orders = clean.to_pandas()
    regs = regions.to_pandas()
    j = orders.merge(regs, on='region_id', how='left')
    out = (j.groupby('region_name', as_index=False)['amount_cents']
             .sum().rename(columns={'amount_cents': 'revenue_cents'}))
    return out

Step-by-step explanation.

1. clean_orders takes one argument, orders=bauplan.Model('raw.orders', ...). That single reference does two things: it declares the edge raw.orders → clean_orders, and it pushes the columns projection and the status = 'paid' predicate into the Iceberg scan so only paid orders' four columns are read.
2. The function returns a pandas DataFrame, and because it is decorated @bauplan.model(), that frame is materialized as an Iceberg table named clean_orders — no explicit write, no CREATE TABLE.
3. revenue_by_region names clean=bauplan.Model('clean_orders') and regions=bauplan.Model('raw.regions'). Those two arguments declare two more edges, so the platform now knows the full graph without any orchestration file.
4. Inside the node, the two Arrow tables become pandas frames, a left join maps region_id to region_name, and a group-by sums revenue — arbitrary Python, exactly the point of the function idiom.
5. The DAG — raw.orders → clean_orders → revenue_by_region ← raw.regions — was never written down anywhere; it is read entirely off the function signatures, which is what "the code is the graph" means in practice.

Output.

Inferred edge	Source	From which argument
raw.orders → clean_orders	source scan	orders=Model('raw.orders')
clean_orders → revenue_by_region	intermediate	clean=Model('clean_orders')
raw.regions → revenue_by_region	source scan	regions=Model('raw.regions')
hand-wired edges	—	0

Rule of thumb. Declare every dependency as a bauplan.Model argument and the DAG builds itself — the same declarative graph dbt gets from ref(), but with an arbitrary-Python body. Return a frame to materialize a node; never write an orchestration file to wire edges that the arguments already imply.

Worked example — Arrow-native, zero-copy hand-off between nodes

Detailed explanation. The property that makes function-as-a-pipeline cheap on intermediate data is Arrow between nodes: a node hands the next node a columnar Arrow table with no serialize/deserialize round-trip. Show a node staying in Arrow for a vectorized aggregation instead of round-tripping through pandas or storage.

• The hand-off. clean_orders output arrives at the next node as an Arrow table.
• The work. A group-and-sum done with pyarrow.compute, vectorized, no Python row loop.
• The saving. No re-serialization moving data downstream; no full copy into pandas.

Question. Aggregate an upstream model's Arrow output using Arrow compute, avoiding both a pandas copy and a storage round-trip.

Input.

Aspect	Round-trip via storage	Arrow-native hand-off
Move data downstream	write + re-read	in-memory Arrow, no copy
Aggregate	load to pandas, loop/groupby	pyarrow.compute, vectorized
Serialization tax	pay it twice	none on intermediates
Memory	duplicated	shared columnar buffers

Code.

import bauplan

@bauplan.model()
@bauplan.python('3.11', pip={'pyarrow': '16.0.0'})
def revenue_by_region_arrow(clean=bauplan.Model('clean_orders')):
    import pyarrow as pa
    import pyarrow.compute as pc

    t = clean.to_arrow()                 # Arrow table handed over — no re-serialization
    # Vectorized group-by-sum entirely in Arrow (no pandas copy, no Python loop):
    grouped = t.group_by('region_id').aggregate([('amount_cents', 'sum')])
    grouped = grouped.rename_columns(['region_id', 'revenue_cents'])
    return grouped                       # returned Arrow table -> Iceberg table

Step-by-step explanation.

1. clean.to_arrow() hands this node the upstream output as an Arrow table. Because Arrow is the shared in-memory format, there is no deserialization step — the columnar buffers are already in the format both nodes speak.
2. t.group_by('region_id').aggregate([('amount_cents', 'sum')]) performs the aggregation with Arrow's vectorized compute kernels — column-at-a-time, no Python-level row iteration — which is both faster and lower-memory than a pandas group-by on a large frame.
3. Nothing is written to storage between the upstream node and this one: the intermediate data lives as Arrow in the runtime, so you avoid the "materialize to S3, then re-read" round-trip a stitched SQL-then-Python stack usually pays on intermediates.
4. The returned Arrow table is materialized as the node's Iceberg table — so the final output is persisted (durably, in the open format), while the intermediate hand-off stayed in memory. You persist boundaries, not every hop.
5. The senior nuance: staying in Arrow is most valuable on wide or large intermediate frames where a pandas copy or a storage round-trip would dominate cost; for tiny frames the convenience of to_pandas() is fine — the point is that the option to avoid the copy exists.

Output.

Step	Cost avoided
upstream → this node	serialization round-trip
aggregation	pandas full-frame copy
intermediate persist	storage write + re-read
final output	(persisted once, as Iceberg)

Rule of thumb. Move data between nodes as Arrow and do heavy aggregations with pyarrow.compute to skip both the pandas copy and the storage round-trip on intermediates; persist only the boundaries you actually need durable. Reach for to_pandas() when you want pandas semantics on a small frame — the zero-copy option is there when the frame is big.

Worked example — per-function environments and isolation

Detailed explanation. The feature that keeps a multi-node pipeline reproducible is per-function environments: each node pins its own interpreter and dependencies, so two nodes can need conflicting library versions and both are satisfied. Show two nodes with different, pinned environments.

• Node A. Needs pandas 2.2 for a reshape.
• Node B. Needs a specific pyarrow plus a small ML library for scoring.
• The isolation. They never share an interpreter, so their dependencies never collide.

Question. Configure two nodes with conflicting dependencies so both run reproducibly without a shared-environment truce.

Input.

Node	Pinned env	Why isolated matters
reshape	Python 3.11 + pandas 2.2.0	pandas-heavy transform
score	Python 3.11 + pyarrow 16 + a model lib	conflicting deps, isolated
Shared cluster env?	no	per-node pins remove the conflict

Code.

import bauplan

@bauplan.model()
@bauplan.python('3.11', pip={'pandas': '2.2.0'})       # this node's ENTIRE env
def reshape(orders=bauplan.Model('clean_orders')):
    df = orders.to_pandas()
    return df.pivot_table(index='region_id', values='amount_cents', aggfunc='sum').reset_index()

@bauplan.model()
@bauplan.python('3.11', pip={'pyarrow': '16.0.0', 'scikit-learn': '1.5.0'})  # different, isolated
def score(features=bauplan.Model('reshape')):
    import pyarrow as pa
    # ... load a model, score rows; deps here can't clash with `reshape`'s pandas pin
    t = features.to_arrow()
    return t

Step-by-step explanation.

1. Each @bauplan.python(...) declaration is the complete environment for that one node: reshape gets Python 3.11 with pandas 2.2.0 and nothing else it did not ask for; score gets Python 3.11 with pyarrow 16 and scikit-learn 1.5.
2. The two environments are built and run in isolation — the nodes never share an interpreter — so even if pandas 2.2 and some scikit-learn dependency wanted incompatible versions of a transitive library, there is no conflict to resolve, because they are never loaded together.
3. This is the opposite of a Spark cluster's model, where a single cluster-wide Python environment forces every job to agree on one dependency set — the "dependency truce" that turns a new library into a cross-team negotiation.
4. Pinning exact versions (not ranges) makes each node's behavior a function of its definition, not of when the environment was last resolved — so a run today and a run next month use byte-identical dependencies, which is a precondition for the content-hash caching covered in section 4.
5. The senior discipline: treat a node's pip set as part of its contract, keep it minimal (only what the body imports), and pin exactly — small, pinned, isolated environments are what make a large DAG reproducible instead of a dependency minefield.

Output.

Property	Shared cluster env	Per-function env
Conflicting deps	must be reconciled	both satisfied, isolated
New library	cross-team negotiation	one node's declaration
Reproducibility	depends on cluster state	pinned per node
Blast radius of a change	whole cluster	one node

Rule of thumb. Pin each node's exact interpreter and dependencies in its own @bauplan.python declaration so conflicting requirements coexist and every run is reproducible — a node's environment is part of its contract, not shared mutable cluster state. Keep the pip set minimal and exact; that discipline is also what makes caching trustworthy.

Senior interview question on the function-as-a-pipeline model

A senior interviewer might ask: "Build a Bauplan pipeline that cleans raw orders, joins reference regions, and produces a revenue-by-region table — as decorated Python functions. Explain how the DAG is derived, how data moves between nodes without a serialization tax, how you keep each node reproducible with its own environment, and what discipline keeps every node cacheable."

Solution Using decorated functions, argument-inferred edges, Arrow hand-off, and pinned envs

import bauplan

# 1. Source-scanning node with projection + predicate pushdown.
@bauplan.model()
@bauplan.python('3.11', pip={'pandas': '2.2.0'})
def clean_orders(
    orders=bauplan.Model('raw.orders',
                         columns=['order_id', 'region_id', 'amount_cents', 'status'],
                         filter="status = 'paid'")     # pushed into the Iceberg scan
):
    df = orders.to_pandas()
    return df[['order_id', 'region_id', 'amount_cents']]   # pure fn of its input

# 2. Downstream node: two args = two edges; Arrow in, Arrow out.
@bauplan.model()
@bauplan.python('3.11', pip={'pyarrow': '16.0.0'})
def revenue_by_region(
    clean=bauplan.Model('clean_orders'),               # EDGE
    regions=bauplan.Model('raw.regions', columns=['region_id', 'region_name']),  # EDGE
):
    import pyarrow as pa
    o = clean.to_arrow()
    r = regions.to_arrow()
    joined = o.join(r, keys='region_id')               # Arrow join, no pandas copy
    out = joined.group_by('region_name').aggregate([('amount_cents', 'sum')])
    return out.rename_columns(['region_name', 'revenue_cents'])

# 3. Run the whole DAG serverless — no cluster, edges from the args.
bauplan run
# DAG:  raw.orders -> clean_orders -> revenue_by_region <- raw.regions
bauplan query "SELECT * FROM revenue_by_region ORDER BY revenue_cents DESC LIMIT 5"

Step-by-step trace.

Node	Args → edges	Env	Data hand-off
clean_orders	raw.orders	pandas 2.2.0	Arrow in (pushdown), frame out
revenue_by_region	clean_orders, raw.regions	pyarrow 16	Arrow in/out, no copy
DAG	inferred from args	per-node pinned	Arrow between nodes
Output	revenue_by_region	—	materialized as Iceberg

After bauplan run, the platform reads the two functions' arguments to build the DAG raw.orders → clean_orders → revenue_by_region ← raw.regions, runs each node in its own isolated container with its pinned environment, hands the cleaned orders to the downstream node as an Arrow table (no serialization, no pandas copy for the join), and materializes the final revenue_by_region frame as an Iceberg table — all with no cluster to provision and no orchestration file to maintain.

Output:

Metric	Hand-wired stack	Bauplan functions
DAG definition	orchestration file	inferred from args
Node language	SQL + Python islands	uniform Python
Intermediate data	write + re-read	Arrow, no round-trip
Environments	cluster-wide truce	per-node pinned
Cluster	sized and operated	none (serverless)

Why this works — concept by concept:

• Argument-inferred DAG — declaring each dependency as a bauplan.Model argument means the graph is read off the code, so there is no separate orchestration definition to drift from the actual pipeline.
• Arrow hand-off — passing columnar Arrow tables between nodes removes the serialize/deserialize round-trip and the pandas copy on intermediates, so moving data downstream is nearly free until you deliberately materialize a boundary.
• Per-node environments — each function pins its own interpreter and dependencies, so conflicting requirements coexist and a run is reproducible in its dependencies, not hostage to shared cluster state.
• Pure functions of declared inputs — keeping side effects out of nodes makes the DAG legible and every node a candidate for content-hash caching, because its output depends only on its declared inputs.
• Cost — you trade a hand-wired orchestration file, a cluster-wide environment, and storage round-trips on intermediates for argument-inferred edges, per-node pins, and in-memory Arrow. The eliminated cost is orchestration glue plus serialization plus dependency negotiation — the graph, the hand-off, and the environments all come from the function definitions themselves.

ETL
Topic — etl
ETL problems on multi-step pipelines and DAGs

Practice →

Data transformation Topic — data-transformation Data transformation problems on cleaning, joining, and aggregating

Practice →

---

3. Git-native data — branch, commit, merge on tables

Write on a branch, audit it, merge only on green — main is never half-written

The mental model in one line: Bauplan makes your Iceberg tables **git-native: creating a branch is a zero-copy operation that gives you an isolated writable view of every table, every pipeline run or merge is a commit with a hash, and you publish by merging a branch into main — which turns the write-audit-publish (WAP) discipline into the default workflow: you run a pipeline onto a fresh branch, audit the result with an ordinary query, and merge into production only if it passes, so bad data is a branch you delete rather than an incident you clean up — and because every commit is retained, data versioning gives you time travel to read any table as of a past commit and rollback by pointing main back at a known-good one.** Branches are cheap, audits are queries, and main only ever advances through a merge you chose to make.

Branches and commits over tables.

• Zero-copy branches. A branch is a new named reference over the same underlying Iceberg data files, not a physical copy — so branching a multi-terabyte lakehouse is instant and cheap, and writes on the branch create new files that main cannot see.
• Commits with hashes. Every run or merge produces a commit hash identifying an immutable snapshot of the catalog; a table's state is always "as of a commit," which is what makes time travel and reproducibility possible.
• Refs everywhere. The CLI and SDK take a --ref (a branch name or a commit hash) so any query, run, or export targets an exact version of the data — the same way git commands take a ref.
• Merge to publish. Merging a branch into main advances production to the branch's state atomically; consumers of main see the change all at once, never a half-written intermediate.

Write-audit-publish (WAP).

• Write. Run the pipeline (or an import) onto a branch, not main. All new data lands on the branch, isolated from every consumer of production.
• Audit. Run quality checks as plain queries against the branch — row counts, null rates, referential checks, business invariants — using the branch ref. The audit reads the candidate data, before anyone else can.
• Publish. If the audit passes, merge the branch into main; if it fails, delete the branch. Production either gets fully-validated data or is untouched — there is no partial-bad state.

Time travel and rollback.

• Read as of a commit. Query any table --ref <commit_hash> to see exactly what it contained at that point — for debugging, reproducing a report, or comparing before/after a change.
• Rollback. If a bad merge slipped through, reset main to a previous good commit; because commits are immutable snapshots, the rollback is exact and instant, not a reconstructive restore.
• Reproducibility. Pinning a pipeline run to an input commit makes the run reproducible: same code, same input snapshot, same output — the data analogue of pinning a dependency version.

The failure modes senior engineers pre-empt.

• Writing straight to main. Skipping the branch means bad data lands in production before anyone checks it. Mitigation: make "run onto a branch" the only way data is written; main advances only through audited merges.
• Branch sprawl. Thousands of abandoned ingestion branches clutter the catalog. Mitigation: name branches by purpose/date and expire or delete them after merge or failure.
• Auditing after publish. Running checks on main after the merge is checking the patient after surgery. Mitigation: the audit runs on the branch, before the merge — that is the entire point of WAP.

Common interview probes on git-native data.

• "What is write-audit-publish?" — write to a branch, audit with a query, merge only on green; bad data is a deleted branch.
• "Is a branch a copy?" — no; zero-copy over shared Iceberg files, so branching is instant.
• "How do you roll back a bad change?" — reset main to a prior commit; commits are immutable snapshots.
• "How do you reproduce yesterday's report?" — query the tables as of yesterday's commit via --ref.

Worked example — create a zero-copy branch and run a pipeline on it

Detailed explanation. The foundational move: branch off main, run the pipeline onto the branch, and confirm that main is untouched until you decide otherwise. Show the branch/checkout/run sequence and where the data lives.

• The branch. ingest.2026-06-01, created from main, zero-copy.
• The run. bauplan run writes the pipeline's outputs onto the branch.
• The isolation. main still shows the old data; only the branch has the new.

Question. Create an isolated branch, run the pipeline onto it, and demonstrate that production main does not see the new data yet.

Input.

Step	Command	Effect
branch	branch create ingest.2026-06-01 --from main	zero-copy ref
checkout	checkout ingest.2026-06-01	work on the branch
run	run	writes onto the branch
verify	query --ref main vs --ref branch	main unchanged

Code.

# 1. Branch off main — instant, zero-copy over the shared Iceberg files.
bauplan branch create ingest.2026-06-01 --from main
bauplan checkout ingest.2026-06-01

# 2. Run the pipeline; every output table is written ONTO the branch.
bauplan run

# 3. The new data exists ONLY on the branch — main is untouched.
bauplan query "SELECT count(*) FROM revenue_by_region" --ref ingest.2026-06-01   # new rows
bauplan query "SELECT count(*) FROM revenue_by_region" --ref main                # old count

Step-by-step explanation.

1. branch create ... --from main creates a new named reference over the same underlying Iceberg data files as main. Nothing is copied, so the operation is instant even on a huge lakehouse — the branch and main share history up to the split point.
2. checkout makes subsequent commands target the branch by default, so the run in the next step writes there without you passing --ref on every call.
3. bauplan run executes the DAG and materializes each node's output as an Iceberg table on the branch. Crucially, these writes create new data files that are visible only through the branch ref — main has no pointer to them.
4. The two query calls prove the isolation: reading --ref ingest.2026-06-01 shows the freshly-written rows, while reading --ref main still returns the pre-run count. The same table name resolves to different data depending on the ref.
5. This isolation is the precondition for everything else in this section: because the run landed on a branch, you can now audit the candidate data at leisure, and main's consumers keep seeing stable production data the entire time.

Output.

Query ref	What it sees
--ref ingest.2026-06-01	new pipeline output (candidate)
--ref main	pre-run production data (stable)
copy made?	none (zero-copy branch)
main consumers affected?	no

Rule of thumb. Always land a pipeline run on a purpose-named branch off main; branching is zero-copy and instant, and it keeps every production consumer on stable data while your candidate output waits to be audited. The same table name resolves per ref — main never sees the branch's writes until you merge.

Worked example — the write-audit-publish flow end to end

Detailed explanation. WAP is the reason git-native data matters operationally: it converts "write to prod and hope" into "write to a branch, prove it's good, then publish." Script the full flow — write, audit with real checks, and merge only on green — in the SDK.

• Write. Run onto ingest.<date>.
• Audit. Query the branch for invariant violations (negative revenue, null keys, row-count sanity).
• Publish. Merge on pass; delete the branch on fail.

Question. Implement write-audit-publish so production only ever receives data that passed explicit quality checks, and a failed batch leaves main untouched.

Input.

Phase	Action	Gate
Write	run pipeline onto a branch	isolated from main
Audit	run check queries on the branch	must all pass
Publish	merge into main	only if audit passed
Fail	delete the branch	main untouched

Code.

import bauplan

client = bauplan.Client()             # SDK; signatures illustrative
branch = "ingest.2026-06-01"

# --- WRITE: isolated, zero-copy branch, run the pipeline onto it ---
client.create_branch(branch, from_ref="main")
client.run(project_dir=".", ref=branch)

# --- AUDIT: run quality checks against the BRANCH, before anyone sees it ---
def scalar(sql):
    return list(client.query(sql, ref=branch))[0][0]

neg_revenue = scalar("SELECT count(*) FROM revenue_by_region WHERE revenue_cents < 0")
null_keys   = scalar("SELECT count(*) FROM revenue_by_region WHERE region_name IS NULL")
row_count   = scalar("SELECT count(*) FROM revenue_by_region")

audit_passes = (neg_revenue == 0) and (null_keys == 0) and (row_count > 0)

# --- PUBLISH: merge only on green; otherwise discard the branch ---
if audit_passes:
    client.merge_branch(source_ref=branch, into_branch="main")   # production advances
else:
    client.delete_branch(branch)                                 # main never touched
    raise ValueError(f"Audit failed: neg={neg_revenue} nulls={null_keys} rows={row_count}")

Step-by-step explanation.

1. The write phase creates the branch and runs the pipeline onto it. Because ref=branch, every table the pipeline produces lands on ingest.2026-06-01, fully isolated from main and its consumers.
2. The audit phase runs ordinary queries with ref=branch — negative-revenue count, null-key count, and a non-empty row count. These read the candidate data before publication, which is the property WAP exists to provide: you inspect the exact bytes that would become production.
3. audit_passes combines the checks into a single gate. In real pipelines this is where you encode business invariants (referential integrity, freshness, distribution sanity) as SQL — the audit is only as good as the checks you write.
4. The publish phase branches on the gate: on pass, merge_branch advances main to the branch's state atomically, so consumers see all the new data at once; on fail, delete_branch discards the candidate and main is provably untouched, then the code raises so the batch is visibly failed.
5. The invariant this buys you: production never contains data that did not pass the checks, and a bad batch costs you a deleted branch and an alert — not an incident, a cleanup, and a post-mortem. The scary operation became a reviewable one.

Output.

Outcome	Audit result	main state
good batch	all checks pass	advanced (merged)
negative revenue found	fail	untouched (branch deleted)
null keys found	fail	untouched (branch deleted)
empty output	fail	untouched (branch deleted)

Rule of thumb. Make write-audit-publish the only path to production: run onto a branch, encode your invariants as audit queries against that branch, and merge only when they all pass — deleting the branch on failure. Production then structurally cannot hold data that failed a check, and a bad batch is a deleted branch, not an incident.

Worked example — time travel and rollback to a good commit

Detailed explanation. Because every commit is an immutable snapshot, you can read a table as of any past commit and, if a bad change slipped through, reset main back to a known-good one. Show reading historical state and performing a rollback.

• Time travel. Query a table --ref <old_commit> to see its past contents.
• Diagnosis. Compare a suspect commit against the previous one.
• Rollback. Reset main to the last good commit — instant, exact.

Question. Reproduce a table's earlier state for debugging, then roll main back to a known-good commit after a bad merge.

Input.

Need	Mechanism	Cost
see past state	query ... --ref <commit>	none (snapshot exists)
find the bad merge	compare commits	two reads
undo it	reset main to good commit	instant, exact
reproduce a report	run/query pinned to a commit	deterministic

Code.

# 1. Time travel: read a table exactly as it was at an earlier commit.
bauplan query "SELECT count(*) FROM revenue_by_region" --ref main          # current (suspect)
bauplan query "SELECT count(*) FROM revenue_by_region" --ref 7a1c9e2       # a known-good commit

# 2. Diagnose: the counts differ -> the merge at commit b4f0aa1 introduced the regression.
bauplan log main            # lists commit hashes on main, newest first

# 3. Rollback: point main back at the last good commit. Immutable snapshots make this exact.
bauplan branch reset main --to 7a1c9e2
# main now serves the known-good data again; the bad commit still exists in history.

Step-by-step explanation.

1. The two query calls use different refs against the same table name: --ref main shows the current (suspect) state, and --ref 7a1c9e2 shows the table exactly as it existed at that earlier commit. No restore or snapshot job runs — the historical snapshot already exists because commits are immutable.
2. Comparing the counts (or any aggregate) between commits localizes the problem: if 7a1c9e2 was fine and main is wrong, the regression entered in a commit between them, which bauplan log lets you identify by hash.
3. branch reset main --to 7a1c9e2 moves the main ref back to the known-good commit. Because that commit is an immutable, complete snapshot of the catalog, the rollback is exact and instant — you are not reconstructing state, you are re-pointing a ref.
4. Notice the bad commit is not erased; it remains in history, so you can still inspect it to understand what went wrong. Rolling back changes what main points at, not what happened — the audit trail is intact.
5. The same --ref mechanism gives reproducibility: pin a report or a downstream run to an input commit and it will always read the identical snapshot, so "the numbers changed" becomes debuggable by comparing commits rather than a mystery.

Output.

Action	Result
query --ref <old_commit>	exact past contents
compare commits	regression localized
reset main --to <good>	production restored, instant
history after rollback	bad commit retained for audit

Rule of thumb. Treat every commit as an immutable snapshot: read any table as of a past --ref for time travel and reproducibility, and recover from a bad merge by resetting main to the last good commit — exact and instant, with the bad commit retained in history for the post-mortem. Rolling back re-points a ref; it never reconstructs state.

Senior interview question on git-native data and safe changes

A senior interviewer might ask: "You need to run a daily ingestion into a multi-terabyte Iceberg lakehouse such that bad data never reaches production, you can prove every batch passed quality checks before it published, and you can roll back or reproduce any day's state. Design it with Bauplan's git-native model: how you isolate the write, what you audit and when, how you publish atomically, and how you recover from a bad batch."

Solution Using zero-copy branches, write-audit-publish, atomic merge, and time-travel rollback

import bauplan
client = bauplan.Client()             # signatures illustrative

run_date = "2026-06-01"
branch = f"ingest.{run_date}"

# 1. WRITE — isolate the batch on a zero-copy branch (main untouched, instant).
client.create_branch(branch, from_ref="main")
client.run(project_dir=".", ref=branch)

# 2. AUDIT — encode invariants as queries against the BRANCH, before publish.
def scalar(sql): return list(client.query(sql, ref=branch))[0][0]

checks = {
    "negative_revenue": scalar("SELECT count(*) FROM revenue_by_region WHERE revenue_cents < 0"),
    "null_region":      scalar("SELECT count(*) FROM revenue_by_region WHERE region_name IS NULL"),
    "row_count":        scalar("SELECT count(*) FROM revenue_by_region"),
    "freshness_ok":     scalar(f"SELECT count(*) FROM clean_orders")  # non-empty for the day
}
passed = checks["negative_revenue"] == 0 and checks["null_region"] == 0 \
         and checks["row_count"] > 0 and checks["freshness_ok"] > 0

# 3. PUBLISH — atomic merge on green; discard + alert on red.
if passed:
    client.merge_branch(source_ref=branch, into_branch="main")   # consumers see it all at once
else:
    client.delete_branch(branch)                                 # main provably untouched
    raise ValueError(f"WAP audit failed: {checks}")

# 4. RECOVER — if a bad batch ever slips through, roll main back to a good commit.
# bauplan branch reset main --to <known_good_commit>   # exact, instant, history retained

Step-by-step trace.

Phase	Mechanism	Guarantee
Write	zero-copy branch + run	main untouched during the batch
Audit	invariant queries on the branch	checks run on candidate data
Publish	atomic merge_branch	consumers see all-or-nothing
Fail	delete_branch + raise	no partial-bad production state
Recover	reset main --to <good>	exact rollback, audit trail kept
Reproduce	--ref <commit>	any day's state re-readable

After deployment, each day's ingestion runs onto ingest.<date>, a zero-copy branch that leaves main and its consumers on stable data; the batch is audited by invariant queries that read the candidate output before publication; a passing batch is merged into main atomically so consumers see the full day at once, while a failing batch is discarded and alerted with main provably untouched; and any past day is re-readable by commit for reproduction, with a bad merge recoverable by resetting main to the last good commit.

Output:

Metric	Write-to-prod-and-hope	Git-native WAP
Bad data reaching prod	possible	structurally prevented
Proof a batch passed	ad-hoc/none	the merge gate itself
Partial-bad state	possible mid-write	impossible (atomic merge)
Rollback	reconstructive restore	re-point a ref, instant
Reproduce a past day	often impossible	--ref <commit>

Why this works — concept by concept:

• Zero-copy branch isolation — a branch is a cheap named ref over shared Iceberg files, so isolating a batch is instant and every production consumer keeps reading stable main data while the candidate is prepared.
• Audit before publish — running invariant checks as queries against the branch inspects the exact data that would become production, so the gate is on the candidate, not on the patient after surgery.
• Atomic merge — merging advances main in one step, so consumers never observe a half-written state; they see the previous version or the full new one, nothing in between.
• Immutable commits for recovery — because every commit is a complete snapshot, rollback is re-pointing main at a known-good hash and time travel is reading any past ref — both exact and instant, with history retained for audits.
• Cost — you trade a scary in-place write for a cheap branch, a set of audit queries, and an atomic merge. The eliminated cost is the production incident a bad batch would cause — O(1) branch-and-merge with all-or-nothing publication, instead of O(cleanup) after corrupting a shared table.

Data validation
Topic — data-validation
Data validation problems on audits and quality gates

Practice →

Data processing Topic — data-processing Data processing problems on batch ingestion and snapshots

Practice →

---

4. Lakehouse execution — Iceberg, serverless, caching

Open Iceberg tables, a serverless runtime with no cluster, and a cache that skips unchanged nodes

The mental model in one line: Bauplan's execution rests on three properties — the storage is open **Iceberg in your object storage (so tables are readable by other engines and the vendor is not a lock-in), the compute is serverless with each DAG node running in an isolated container that autoscales per node and needs no cluster to size, and the runtime keeps a content-hash cache of node outputs so a re-run skips any node whose code and inputs did not change — which together mean the lakehouse gives you an open format, zero cluster ops, and incremental re-runs where only what actually changed is recomputed.** The tables outlive the tool, the compute appears on demand, and the second run of an unchanged pipeline is nearly free.

Iceberg as the open substrate.

• Your object storage. Tables are Iceberg data + metadata files in a bucket you own; Bauplan operates on them in place rather than importing them into a proprietary store — so the data never leaves your control.
• Open readability. Because the format is standard Iceberg, the same tables are readable by Spark, Trino, DuckDB, and other Iceberg clients — the exit door and the "use another engine for this one job" door are both open.
• Projection + predicate pushdown. bauplan.Model('t', columns=[...], filter="...") prunes columns and rows at the scan, using Iceberg's metadata (and file/partition statistics) to skip data files that cannot match — the difference between reading a partition and reading a table.
• Snapshots underpin versioning. Iceberg's snapshot model is what the git-native layer builds on: a commit is a catalog snapshot, which is why branching and time travel are cheap.

Serverless execution.

• No cluster. There is no standing compute to size, patch, or autoscale; each node's container is provisioned for its run and released, so idle cost is zero and you never own a cluster's failure modes.
• Per-node isolation and scale. Nodes run independently, so the runtime can scale them out and give each the resources it declares — a heavy node and a light node do not share (or fight over) a fixed cluster.
• Fast cold behavior. Because the unit of compute is a function container rather than a JVM cluster warm-up, spinning up work is quick — well-suited to the many small-to-medium transformations a typical DAG contains.
• The limit. A per-node serverless model is not a distributed-shuffle engine: a single massive cross-partition join/shuffle is Spark's home turf, not this runtime's — a boundary to name, not hide.

Caching for incremental re-runs.

• Content-hash keys. The runtime hashes a node's code, its declared environment, and its inputs; if the hash matches a previous successful run, the cached output is reused and the node is skipped.
• Only-what-changed recompute. Change one node and re-run: that node and its descendants recompute, while unchanged upstream nodes are served from cache — so iterating on a late-stage transform does not re-run the whole pipeline.
• Determinism is the contract. Caching is only correct if nodes are pure functions of their declared inputs; a hidden clock read or random seed makes a node's output unpredictable and its cache untrustworthy.

The failure modes senior engineers pre-empt.

• Fat scans with no pushdown. Loading a whole table into a node when you needed two columns and one partition wastes I/O and memory. Mitigation: always pass columns and filter on source models so Iceberg prunes at the scan.
• Non-deterministic nodes. now(), unseeded randomness, or hidden external reads defeat caching and reproducibility. Mitigation: pass time/seed as explicit inputs; keep nodes pure.
• Tiny-file explosions. Many small writes create metadata bloat and slow scans over time. Mitigation: compact/partition sensibly and avoid pathologically small per-run outputs.

Common interview probes on execution.

• "Where does the data live?" — open Iceberg in your own object storage, readable by other engines.
• "What runs the compute?" — a serverless runtime; each node an isolated container, no cluster.
• "How are re-runs fast?" — content-hash caching skips nodes whose code and inputs are unchanged.
• "What is it bad at?" — massive distributed shuffles; that stays on Spark.

Worked example — an Iceberg scan with projection and predicate pushdown

Detailed explanation. The cheapest scan is the one that reads the least. Iceberg pushdown lets a source model read only the needed columns and only the data files that can match a predicate. Contrast a naive full load with a pushed-down scan on the same node.

• Naive. Read the whole table into the node, then filter in pandas.
• Pushdown. Declare columns and filter on bauplan.Model so Iceberg prunes at the scan.
• The win. Fewer bytes read, less memory, files skipped via partition/statistics.

Question. Rewrite a node so the source scan reads only two columns and only the matching partition, instead of loading the table and filtering in Python.

Input.

Aspect	Naive load	Pushdown
Columns read	all	columns=[...] only
Rows/files read	all files	files matching filter
Where filtering happens	in pandas	at the Iceberg scan
I/O and memory	full table	pruned subset

Code.

import bauplan

# NAIVE: reads the whole table, then filters in Python (wasteful I/O + memory).
@bauplan.model()
@bauplan.python('3.11', pip={'pandas': '2.2.0'})
def eu_paid_naive(orders=bauplan.Model('raw.orders')):        # no pushdown
    df = orders.to_pandas()                                   # ALL columns, ALL rows loaded
    df = df[(df['region'] == 'EU') & (df['status'] == 'paid')]
    return df[['order_id', 'amount_cents']]

# PUSHDOWN: Iceberg prunes columns and files at the scan; the node loads a subset.
@bauplan.model()
@bauplan.python('3.11', pip={'pandas': '2.2.0'})
def eu_paid_pushdown(
    orders=bauplan.Model(
        'raw.orders',
        columns=['order_id', 'amount_cents', 'region', 'status'],   # projection pushdown
        filter="region = 'EU' AND status = 'paid'",                 # predicate pushdown
    )
):
    return orders.to_pandas()[['order_id', 'amount_cents']]   # already filtered at the scan

Step-by-step explanation.

1. eu_paid_naive passes bauplan.Model('raw.orders') with no columns or filter, so the node receives every column of every row and only narrows the data after it is in memory — the I/O and memory cost is the whole table regardless of how little survives the filter.
2. eu_paid_pushdown declares columns=[...], which is projection pushdown: the Iceberg scan reads only those four columns' data, skipping the rest of each row group entirely.
3. The filter="region = 'EU' AND status = 'paid'" is predicate pushdown: Iceberg uses partition values and per-file column statistics to skip data files that cannot contain matching rows, so if the table is partitioned by region, non-EU files are never opened.
4. The result is that the pushed-down node loads a pruned subset — fewer bytes, less memory, fewer files touched — while producing the identical output, because the filtering moved from pandas to the scan.
5. The senior habit is to treat columns and filter on every source model as mandatory, not optional: "read everything into Python and filter there" is the single most common way a function-based pipeline becomes needlessly expensive.

Output.

Node	Columns read	Files read	Filter location
eu_paid_naive	all	all	pandas (after load)
eu_paid_pushdown	4	EU partition only	Iceberg scan
bytes read	full table	pruned subset	—
output rows	identical	identical	—

Rule of thumb. Put columns and filter on every bauplan.Model source so Iceberg prunes columns and skips non-matching data files at the scan — never load a table into a node to filter it in pandas. Pushdown turns "read the table" into "read the partition," which is usually the biggest single cost win in a function pipeline.

Worked example — the serverless run model versus a cluster

Detailed explanation. The operational difference from Spark is that there is no cluster: each node's container is provisioned for its run and released. Contrast running a DAG on Bauplan's serverless runtime with running the same DAG on a Spark cluster you operate.

• Spark. A sized, running cluster; you own autoscaling, memory tuning, and idle cost.
• Bauplan. Per-node containers on demand; zero idle cost; nothing to size.
• The trade. No shuffle engine for petabyte joins, but no ops for the common case.

Question. Describe how the same three-node DAG executes on a serverless per-node runtime versus a managed cluster, on provisioning, isolation, idle cost, and the workload each suits.

Input.

Dimension	Spark cluster	Bauplan serverless
Provisioning	size + run a cluster	none; per-node containers
Isolation	shared cluster env	isolated per node
Idle cost	pay while up	zero
Best workload	huge distributed shuffles	many small/medium transforms

Code.

Same DAG:  raw.orders -> clean_orders -> revenue_by_region <- raw.regions

ON A SPARK CLUSTER (you operate it)
  - Provision: choose node types, count, autoscaling; keep it warm or pay cold-starts.
  - Run: executors share one cluster-wide Python env (dependency truce).
  - Idle: the cluster costs money whether or not a pipeline is running.
  - Strength: a massive cross-partition shuffle/join distributes across executors.

ON BAUPLAN (serverless)
  - Provision: nothing. Each node gets an isolated container for its run, then it's gone.
  - Run: `bauplan run` — clean_orders and revenue_by_region each run in their own env.
  - Idle: zero standing compute; you pay for the work, not for a cluster sitting there.
  - Limit: not a distributed-shuffle engine — a petabyte join is still Spark's job.

# The entire "operate the cluster" step, on Bauplan:
bauplan run        # provisions per-node containers, runs the DAG, releases them

Step-by-step explanation.

1. On Spark, the DAG cannot run until a cluster exists: you choose instance types and counts, configure autoscaling, and either keep the cluster warm (paying for idle) or accept cold-start latency. That provisioning-and-tuning loop is standing operational work.
2. On Bauplan, bauplan run provisions an isolated container per node for the duration of that node's execution and releases it afterward — there is no cluster object to create, size, or keep alive, so the "operate the cluster" task simply does not exist.
3. Isolation flips accordingly: Spark executors share one cluster-wide Python environment (the dependency truce from section 2), while Bauplan nodes each carry their own pinned environment, so provisioning and reproducibility improve together.
4. Idle cost is the sharp economic contrast: a Spark cluster costs money whenever it is up, so teams over-provision and forget to tear down; a serverless runtime has zero standing compute, so you pay for pipeline work, not for a cluster idling overnight.
5. The honest limit keeps the comparison credible: a per-node container model is excellent for the many small-to-medium transforms a typical DAG contains, but a single petabyte cross-partition shuffle is what Spark's distributed executors exist for — so the right answer is "serverless for the common case, Spark for the genuine shuffle monster," not "serverless always."

Output.

Concern	Spark cluster	Bauplan serverless
"Operate a cluster?"	yes	no
Idle cost	continuous	zero
Per-node environments	no (shared)	yes (isolated)
Petabyte shuffle	strong	not its job

Rule of thumb. Reach for Bauplan's serverless runtime to erase cluster ops and idle cost for the many small-to-medium transforms that dominate real DAGs, and keep Spark for genuine petabyte distributed shuffles. "No cluster to operate" is the win; "not a distributed-shuffle engine" is the boundary — say both.

Worked example — cache-aware re-runs that skip unchanged nodes

Detailed explanation. Content-hash caching makes the second run of a pipeline nearly free: nodes whose code, environment, and inputs are unchanged are served from cache, and only changed nodes and their descendants recompute. Show what re-runs after different edits.

• First run. Every node computes and its output is cached by content hash.
• Edit a late node. Only it and its descendants recompute; upstream is cached.
• The requirement. Nodes must be deterministic for the cache to be correct.

Question. Given a three-node DAG, determine which nodes recompute after (a) no change, (b) editing the last node, and (c) changing a source filter — and state the determinism requirement.

Input.

Change	Recomputes	Served from cache
nothing	none	all
edit last node	last node only	upstream nodes
change source filter	source + all descendants	nothing downstream of change
add a nondeterministic now()	unpredictable	cache untrustworthy

Code.

DAG:  clean_orders --> revenue_by_region --> region_report

(a) Re-run with NO changes
    hash(clean_orders)   == cached  -> SKIP
    hash(revenue_by_...) == cached  -> SKIP
    hash(region_report)  == cached  -> SKIP
    => whole pipeline served from cache; near-zero compute.

(b) Edit ONLY region_report's code
    clean_orders         cached  -> SKIP
    revenue_by_region    cached  -> SKIP
    region_report        hash changed -> RECOMPUTE (it's a leaf; nothing downstream)

(c) Change clean_orders' source filter (an INPUT change)
    clean_orders         hash changed -> RECOMPUTE
    revenue_by_region    input changed -> RECOMPUTE
    region_report        input changed -> RECOMPUTE
    => a change ripples to descendants only.

# Determinism is the cache contract. This node is NOT cacheable-correct:
@bauplan.model()
@bauplan.python('3.11', pip={'pandas': '2.2.0'})
def bad(orders=bauplan.Model('clean_orders')):
    import pandas as pd, datetime as dt
    df = orders.to_pandas()
    df['loaded_at'] = dt.datetime.now()   # hidden nondeterminism -> output differs each run
    return df
# Fix: pass the timestamp as an explicit input so the node is a pure function of its inputs.

Step-by-step explanation.

1. On the first run, every node computes and the runtime stores its output keyed by a content hash of the node's code, its declared environment, and its inputs. That hash is what later runs compare against.
2. Case (a) — re-running with no changes — recomputes nothing: each node's hash matches a cached entry, so all three are skipped and the pipeline returns from cache in near-zero compute, which is what makes rapid iteration cheap.
3. Case (b) — editing only the leaf region_report — changes just that node's code hash, so only region_report recomputes; the two upstream nodes are unchanged and served from cache. Iterating on a final report does not re-run the expensive upstream.
4. Case (c) — changing clean_orders' source filter — is an input change at the top of the DAG, so clean_orders recomputes and the change ripples to every descendant. Caching skips unchanged nodes, but a real change correctly invalidates everything downstream of it.
5. The bad node shows why determinism is the contract: datetime.now() makes the output differ every run, so its content hash can never stably match — the cache becomes untrustworthy and reproducibility is lost. The fix is to pass time (or a seed) as an explicit input, keeping the node a pure function of its declared inputs.

Output.

Scenario	Nodes recomputed	Cache benefit
no change	0 of 3	full pipeline cached
edit leaf node	1 of 3	upstream reused
change source filter	3 of 3	correct invalidation
nondeterministic node	unpredictable	cache defeated

Rule of thumb. Lean on content-hash caching to make re-runs incremental — only changed nodes and their descendants recompute — but earn it by keeping every node deterministic: pass timestamps and seeds as explicit inputs and do no hidden I/O. A pure function of its declared inputs is cacheable and reproducible; a node that reads the clock is neither.

Senior interview question on the lakehouse execution model

A senior interviewer might ask: "Explain how a Bauplan pipeline actually executes: where the data lives and why that matters for lock-in, how the compute runs without a cluster, how scans stay cheap, how re-runs avoid recomputing everything, and what workload this execution model is the wrong choice for. Tie each property to a concrete engineering consequence."

Solution Using open Iceberg storage, serverless per-node compute, pushdown, and content-hash caching

# 1. Open Iceberg + pushdown: read only the columns/files you need from YOUR bucket.
import bauplan

@bauplan.model()
@bauplan.python('3.11', pip={'pandas': '2.2.0'})
def clean_orders(
    orders=bauplan.Model('raw.orders',
                         columns=['order_id', 'region', 'amount_cents', 'status'],
                         filter="status = 'paid'")       # Iceberg prunes at the scan
):
    return orders.to_pandas()[['order_id', 'region', 'amount_cents']]

# 2. Serverless run: no cluster; each node an isolated container; incremental via cache.
bauplan run
#   first run   -> every node computes, outputs cached by content hash
#   second run  -> unchanged nodes SKIPPED; only edited nodes + descendants recompute
#   storage     -> Iceberg files in your object storage, readable by Spark/Trino/DuckDB

# 3. Property -> engineering consequence (say this out loud in the interview):
Open Iceberg in your bucket    -> no lock-in; other engines read the same tables; easy exit.
Serverless per-node compute    -> no cluster to size/patch; zero idle cost; per-node isolation.
Projection + predicate pushdown-> read a partition, not a table; big I/O + memory win.
Content-hash caching           -> incremental re-runs; iterate on a leaf without re-running upstream.
NOT a distributed-shuffle engine-> petabyte cross-partition joins stay on Spark. (the boundary)

Step-by-step trace.

Property	Mechanism	Engineering consequence
Open storage	Iceberg in your bucket	no lock-in; multi-engine reads
Serverless compute	per-node containers	no cluster ops; zero idle cost
Cheap scans	column + predicate pushdown	read a partition, not a table
Incremental re-runs	content-hash cache	skip unchanged nodes
The limit	no distributed shuffle	petabyte joins stay on Spark

After deployment, the pipeline reads open Iceberg tables from the team's own object storage with column and predicate pushdown pruning each scan; bauplan run executes each node in an isolated serverless container with no cluster to operate and zero idle cost; a content-hash cache makes every re-run incremental so only edited nodes and their descendants recompute; and the one workload this model does not fit — a petabyte cross-partition shuffle — is named up front as Spark's job, keeping the recommendation honest.

Output:

Metric	Cluster + proprietary store	Bauplan lakehouse
Storage lock-in	high	none (open Iceberg)
Cluster ops	continuous	none (serverless)
Idle cost	paid	zero
Scan cost	often full table	pruned by pushdown
Re-run cost	full pipeline	only what changed

Why this works — concept by concept:

• Open Iceberg storage — keeping tables as standard Iceberg in your own bucket means other engines can read them and the vendor is never a lock-in, so the execution model's benefits never come at the price of your data's portability.
• Serverless per-node compute — provisioning an isolated container per node and releasing it removes both cluster operations and idle cost, and gives each node its own environment for free.
• Pushdown scans — declaring columns and predicates lets Iceberg prune columns and skip non-matching files, turning "read the table" into "read the partition," usually the largest single I/O win.
• Content-hash caching — hashing code, environment, and inputs lets the runtime reuse unchanged nodes' outputs, so iterating on a late-stage transform is cheap and full re-runs are rare — provided nodes stay deterministic.
• Cost — you trade a distributed-shuffle engine's raw join muscle for zero cluster ops, zero idle cost, pruned scans, and incremental re-runs on open storage. The eliminated cost is cluster operation plus full-table scans plus full re-runs — O(changed) recompute over O(pruned) I/O, with petabyte shuffles explicitly left to Spark.

Data processing
Topic — data-processing
Data processing problems on scans, pruning, and compute

Practice →

Optimization Topic — optimization Optimization problems on pushdown, caching, and incremental runs

Practice →

---

5. Bauplan vs dbt, Spark, and lakeFS — and CI

One runtime replaces an assembled stack — and data finally gets code-style CI

The mental model in one line: Bauplan sits where an assembled stack normally needs three or four tools — Spark for distributed compute, dbt for SQL transforms, and Nessie or **lakeFS for git-native data versioning, wired by an orchestrator — and folds compute, transforms, and versioning into one runtime, which unlocks the pattern the assembled stack makes awkward: branch-per-PR CI for data, where every change opens a branch, runs the pipeline onto it, audits the output, and merges into main only when the checks pass — exactly like code CI — so the decision to adopt is really "do I want fewer moving parts and data CI, or do I have a workload (petabyte shuffles) or an investment (a healthy dbt estate) that keeps the assembled stack ahead?"** It is a consolidation-and-CI story, and the honest version names both when it wins and when it does not.

What each tool in the assembled stack does — and where Bauplan absorbs it.

• Spark → serverless compute. Spark is a distributed compute engine you operate; Bauplan replaces it for the common case with per-node serverless containers, while conceding the petabyte-shuffle case back to Spark.
• dbt → Python functions. dbt models transforms as SQL with ref() dependencies; Bauplan models them as Python functions with bauplan.Model dependencies — the same declarative DAG, a more general node body.
• Nessie / lakeFS → native branches. Nessie and lakeFS add git-for-data as a separate catalog/proxy over your lake; Bauplan builds branch/commit/merge into the runtime, so versioning is not a second system to operate.
• Orchestrator → inferred DAG. Airflow/Dagster wire the pieces and schedule them; Bauplan infers the DAG from function arguments, so intra-pipeline wiring disappears (you may still schedule the trigger externally).

The branch-per-PR CI pattern.

• Branch per change. Every proposed change (new model, fixed logic, backfill) opens a data branch, so the change is isolated exactly like a code feature branch.
• Run + audit in CI. The CI job runs the pipeline onto the branch and executes audit queries; a failing audit fails the check, blocking the merge — data quality becomes a required status check.
• Merge on green. Only a passing branch merges into main, so production data advances through reviewed, audited changes — the write-audit-publish flow, now driven by your PR process.
• Reproducible reviews. Because a branch is a real, queryable version of the data, a reviewer can inspect the actual proposed output, not just the code diff — review the data change, not only the code change.

When the assembled stack still wins.

• Petabyte distributed shuffles. A workload dominated by massive cross-partition joins wants Spark's shuffle engine; a per-node serverless runtime is the wrong tool. Keep Spark for that job.
• A large healthy dbt/SQL estate. If transforms are pure SQL, the team is fluent, and there is no ops pain, the migration cost outweighs the consolidation benefit. Stay.
• Streaming. Bauplan is batch-oriented; sub-second/continuous pipelines belong to a streaming engine (Flink, Kafka Streams, Spark Structured Streaming).
• Non-Iceberg lakes / hard platform constraints. If your data is not (and will not be) Iceberg, or a managed dependency is disallowed, the fit is poor.

The failure modes senior engineers pre-empt.

• Adopting for a single feature. Choosing Bauplan only for branching (when lakeFS over your existing stack would do) buys a whole runtime to get one capability. Mitigation: adopt for the consolidation, not one axis.
• Ignoring the shuffle limit. Betting a petabyte-join workload on a per-node runtime. Mitigation: benchmark the heavy job; keep Spark where it wins.
• Skipping the audit in CI. A branch-per-PR flow with no audit query is just a branch — the value is the gate. Mitigation: require audit checks as blocking CI status.

Common interview probes on positioning.

• "How is it different from dbt?" — Python-function nodes vs SQL models; and it also does compute and versioning, which dbt does not.
• "How is it different from lakeFS/Nessie?" — those add versioning to your stack; Bauplan builds it in and also runs the compute.
• "Isn't it just serverless Spark?" — no; it is not a distributed-shuffle engine, and it bundles transforms + versioning.
• "What does data CI look like?" — branch per PR, run + audit in CI, merge on green.

Worked example — the Bauplan-vs-assembled-stack comparison

Detailed explanation. The interview-ready artifact is a side-by-side of Bauplan against the stack it replaces, axis by axis, ending in a one-line verdict per axis. Build it so you can recite the positioning without hand-waving.

• The axes. Compute, transforms, versioning, orchestration, storage, ops.
• The stack. Spark + dbt + Nessie/lakeFS + Airflow over Iceberg.
• The verdict. Consolidation and data CI, with two honest exceptions.

Question. Contrast Bauplan with the assembled stack on six axes and give the net positioning.

Input.

Axis	Assembled stack	Bauplan
Compute	Spark (operate a cluster)	serverless, per-node
Transforms	dbt SQL + ref()	Python functions + bauplan.Model
Versioning	Nessie / lakeFS (bolt-on)	native branch/commit/merge
Orchestration	Airflow/Dagster	DAG from function args
Storage	Iceberg (open)	Iceberg (open)
Ops surface	four systems + glue	one runtime

Code.

Bauplan vs the assembled stack — axis by axis
==============================================

Compute        Spark cluster you size/operate      | serverless, no cluster
               strong at petabyte shuffles         | strong at many small/med transforms

Transforms     dbt: SQL models, ref() edges        | Python functions, bauplan.Model() edges
               best for pure set-based marts        | best for code-first / ML-adjacent logic

Versioning     Nessie/lakeFS: a separate catalog    | built into the runtime
               another system to run/sync           | branch/commit/merge, no extra system

Orchestration  Airflow/Dagster wire + schedule      | DAG inferred from args (schedule trigger only)

Storage        open Iceberg in your bucket          | open Iceberg in your bucket (same!)

Ops            4 tools + integration glue           | 1 runtime

VERDICT: Bauplan trades four tools for one runtime and unlocks data CI.
         Keep Spark for petabyte shuffles; keep a healthy dbt estate as-is.

Step-by-step explanation.

1. On compute, the honest framing is complementary, not "better": Spark owns petabyte distributed shuffles, Bauplan owns the many small-to-medium transforms that dominate real DAGs by count — so the axis is workload shape, not raw superiority.
2. On transforms, both are declarative DAGs (dbt's ref(), Bauplan's bauplan.Model), and the difference is the node body — SQL versus arbitrary Python — which makes dbt ideal for pure set-based marts and Bauplan ideal for code-first and ML-adjacent logic.
3. On versioning, the contrast is systemic: Nessie/lakeFS deliver git-for-data as a separate catalog or proxy you run alongside the lake, whereas Bauplan builds branch/commit/merge into the runtime, so there is no second versioning system to operate or keep in sync.
4. On orchestration and ops, the assembled stack needs an orchestrator plus integration glue across four tools, while Bauplan infers the intra-pipeline DAG from function arguments and presents one runtime — you may still trigger a scheduled run externally, but the wiring inside the pipeline is gone.
5. Storage is identical — open Iceberg in your bucket — which is the key de-risking point: adopting Bauplan does not change where or how your data is stored, so the exit cost stays bounded and the verdict ("one runtime plus data CI, minus two honest exceptions") is a low-lock-in bet.

Output.

Axis	Net verdict
Compute	Bauplan for common case; Spark for petabyte shuffle
Transforms	Bauplan for code-first; dbt for pure SQL marts
Versioning	Bauplan (native) beats a bolt-on catalog
Ops	one runtime beats four tools + glue
Storage	tie — open Iceberg either way

Rule of thumb. Position Bauplan as trading four tools (Spark + dbt + Nessie/lakeFS + orchestrator) for one runtime plus native data CI, on the same open Iceberg storage — then name the two honest exceptions (petabyte shuffles stay on Spark, a healthy dbt estate stays put). Same storage is what keeps the bet low-lock-in.

Worked example — a branch-per-PR CI pipeline for data

Detailed explanation. The capability the consolidation unlocks is treating a data change like a code change: open a PR, and CI branches the data, runs the pipeline, audits it, and blocks the merge unless the audit passes. Sketch the CI job.

• On PR open. Create a data branch named for the PR.
• In CI. Run the pipeline onto the branch and execute audit queries.
• On green. Allow the merge (which merges the data branch into main).

Question. Write a CI job that runs the pipeline on a per-PR data branch and blocks the merge unless audit checks pass.

Input.

CI stage	Action	Blocks merge?
branch	create pr-<n> from main	—
run	bauplan run --ref pr-<n>	on run failure
audit	invariant queries on the branch	on any failure
publish	merge pr-<n> into main on green	—

Code.

# .ci/data-pipeline.yml — branch-per-PR CI for data (illustrative).
on: pull_request
jobs:
  data-ci:
    steps:
      - name: Create a data branch for this PR
        run: bauplan branch create "pr-${PR_NUMBER}" --from main

      - name: Run the pipeline onto the PR branch (no cluster to spin up)
        run: bauplan run --ref "pr-${PR_NUMBER}"

      - name: Audit the candidate data — FAIL the check on any violation
        run: |
          bad=$(bauplan query --ref "pr-${PR_NUMBER}" \
            "SELECT count(*) FROM revenue_by_region WHERE revenue_cents < 0")
          nulls=$(bauplan query --ref "pr-${PR_NUMBER}" \
            "SELECT count(*) FROM revenue_by_region WHERE region_name IS NULL")
          test "$bad" -eq 0 && test "$nulls" -eq 0   # non-zero exit blocks the merge

# On merge of the PR (green only): publish by merging the data branch into main.
bauplan branch merge "pr-${PR_NUMBER}" --into main
# Reviewers could inspect the ACTUAL candidate data before approving:
bauplan query --ref "pr-${PR_NUMBER}" "SELECT * FROM revenue_by_region ORDER BY revenue_cents DESC LIMIT 20"

Step-by-step explanation.

1. On pull_request, the job creates a data branch pr-<n> from main — the data analogue of the code feature branch — so the change is isolated and main stays stable while CI runs.
2. bauplan run --ref pr-<n> executes the pipeline onto that branch with no cluster to provision, materializing the candidate output where only this PR can see it. A run failure fails the CI step and blocks the merge.
3. The audit stage runs invariant queries against the branch and uses shell exit status as the gate: if bad or nulls is non-zero, test returns non-zero, the CI check goes red, and the PR cannot merge — data quality is now a required status check, not a hope.
4. Because the branch is a real, queryable version of the data, a reviewer can run the final query to inspect the actual proposed rows — they review the data change, not just the code diff, which is a strictly stronger review than reading a SQL diff.
5. On merge, branch merge --into main publishes the audited branch atomically, so production advances only through changes that passed CI — write-audit-publish, now driven entirely by the team's existing PR workflow.

Output.

PR state	CI result	main advances?
audit passes	green	yes (on merge)
negative revenue found	red	no (blocked)
null keys found	red	no (blocked)
run fails	red	no (blocked)

Rule of thumb. Wire data changes into your PR flow: branch per PR, bauplan run onto the branch, audit queries as blocking CI checks, and an atomic merge into main only on green — with reviewers inspecting the real candidate data by ref. That is write-audit-publish expressed as ordinary code CI, and the audit gate is where the value lives.

Worked example — the "when the assembled stack wins" decision

Detailed explanation. A credible recommendation names its own exceptions. Walk three teams whose situation makes the assembled stack the right call despite Bauplan's consolidation appeal, and state the deciding factor for each.

• Team X. A petabyte cross-partition join is the core nightly job.
• Team Y. A mature, well-loved dbt estate with no ops pain.
• Team Z. Sub-second streaming enrichment is the requirement.

Question. For each team, decide Bauplan or the assembled stack and name the single deciding factor.

Input.

Team	Situation	Deciding factor
X	petabyte shuffle nightly	needs a distributed-shuffle engine
Y	healthy dbt/SQL, no ops pain	migration cost > benefit
Z	sub-second streaming	Bauplan is batch-oriented
(general)	mixed batch, ops pain, Iceberg	Bauplan fits

Code.

When the assembled stack wins (name the deciding factor, not a vibe)
===================================================================

Team X — petabyte cross-partition join is the core job
  Deciding factor: needs a DISTRIBUTED-SHUFFLE engine.
  Verdict: Spark. A serverless per-node runtime is the wrong tool for that shuffle.

Team Y — mature dbt/SQL estate, fluent team, NO ops pain
  Deciding factor: migration COST outweighs the consolidation benefit.
  Verdict: stay on dbt. Bauplan shines when there IS pain to remove.

Team Z — sub-second streaming enrichment
  Deciding factor: Bauplan is BATCH-oriented.
  Verdict: a streaming engine (Flink / Kafka Streams / Spark Structured Streaming).

General fit — mixed batch transforms, cluster+glue ops pain, data already Iceberg,
              Python-heavy logic, wants safe branch-based changes
  Verdict: Bauplan. This is the situation the consolidation was built for.

Step-by-step explanation.

1. Team X's deciding factor is workload shape: a petabyte cross-partition join needs a distributed-shuffle engine, and a per-node serverless runtime does not provide that, so Spark wins regardless of how appealing consolidation is elsewhere.
2. Team Y's deciding factor is sunk investment and absence of pain: Bauplan's value is removing cluster ops, split paradigms, and unsafe changes, but if a team has a healthy dbt/SQL estate and none of those pains, the migration cost is unjustified — the tool shines where there is pain to remove.
3. Team Z's deciding factor is execution paradigm: Bauplan is batch-oriented, so a sub-second continuous requirement belongs to a streaming engine — this is a category boundary, not a tuning question.
4. The general-fit row is the mirror image: mixed batch transforms, real cluster-and-glue ops pain, data already in Iceberg, Python-heavy logic, and a desire for safe branch-based changes is exactly the situation the consolidation targets — every deciding factor points the same way.
5. The senior habit is to lead with the deciding factor for each case rather than a general preference, because a recommendation that names when it is wrong is far more trustworthy than one that claims to fit everything.

Output.

Team	Verdict	Deciding factor
X (petabyte shuffle)	Spark	needs distributed shuffle
Y (healthy dbt)	stay on dbt	migration cost > benefit
Z (streaming)	streaming engine	Bauplan is batch
general fit	Bauplan	consolidation removes real pain

Rule of thumb. Recommend Bauplan by naming the deciding factor, not a vibe: petabyte shuffles keep Spark, a healthy dbt estate stays put, streaming needs a streaming engine — and mixed batch with real ops pain over Iceberg is the consolidation's home. A recommendation that states its own exceptions is the one worth trusting.

Senior interview question on positioning and data CI

A senior interviewer might ask: "Compare Bauplan with the assembled stack of Spark, dbt, and lakeFS/Nessie, explain the branch-per-PR CI pattern it unlocks for data, and — critically — tell me when you would advise a team to keep the assembled stack instead. Make the case as an engineer who has to own the decision, not sell the tool."

Solution Using consolidation framing, branch-per-PR CI, and named exceptions

# 1. Consolidation: what Bauplan absorbs, and what it deliberately doesn't.
Spark (compute)          -> serverless per-node   [EXCEPT petabyte shuffles -> keep Spark]
dbt (SQL transforms)     -> Python functions      [EXCEPT a healthy pure-SQL estate -> keep dbt]
Nessie/lakeFS (versioning) -> native branch/commit/merge   [strict win: no extra system]
Airflow (orchestration)  -> DAG inferred from args [external scheduler still triggers runs]
Iceberg storage          -> unchanged (open, your bucket) [de-risks the whole bet]

# 2. The CI pattern the consolidation unlocks: data changes reviewed like code.
on: pull_request
jobs:
  data-ci:
    steps:
      - run: bauplan branch create "pr-${PR_NUMBER}" --from main   # branch the DATA
      - run: bauplan run --ref "pr-${PR_NUMBER}"                    # run onto the branch
      - run: |                                                      # audit = blocking check
          bad=$(bauplan query --ref "pr-${PR_NUMBER}" \
            "SELECT count(*) FROM revenue_by_region WHERE revenue_cents < 0")
          test "$bad" -eq 0        # red audit blocks the merge
      # on green merge:  bauplan branch merge "pr-${PR_NUMBER}" --into main

# 3. When to KEEP the assembled stack (own the decision honestly):
Petabyte cross-partition shuffle dominates -> Spark. Not a shuffle engine.
Mature dbt/SQL estate, fluent team, no pain -> stay. Migration cost > benefit.
Sub-second streaming requirement           -> streaming engine. Bauplan is batch.
Data not (going to be) Iceberg / managed-svc banned -> poor fit.
Otherwise (mixed batch, ops pain, Iceberg, Python) -> Bauplan: real pain removed.

Step-by-step trace.

Question	Assembled stack	Bauplan
Who runs compute?	Spark cluster (operated)	serverless runtime
Where do transforms live?	dbt SQL models	Python functions
Where is versioning?	Nessie/lakeFS (separate)	native to the runtime
How are changes reviewed?	code diff only	code + real data by ref
What blocks a bad change?	ad-hoc	audit as CI status check
Storage	open Iceberg	open Iceberg (same)

After laying it out, the positioning is that Bauplan folds Spark's common-case compute, dbt's transforms, and lakeFS/Nessie's versioning into one runtime over the same open Iceberg storage, and in doing so unlocks branch-per-PR CI where a data change is branched, run, audited, and merged on green just like code — while the assembled stack remains the right call for petabyte shuffles, a healthy dbt estate, streaming, or non-Iceberg lakes. The recommendation is a judgment with named exceptions, not a pitch.

Output:

Metric	Assembled stack	Bauplan + data CI
Tools to operate	4 + glue	1 runtime
Data change review	code diff	code + candidate data
Bad-change gate	manual/ad-hoc	blocking audit in CI
Storage lock-in	open Iceberg	open Iceberg
Wrong for	—	petabyte shuffle, streaming, big dbt estate

Why this works — concept by concept:

• Consolidation over four tools — replacing Spark's common case, dbt, a versioning catalog, and an orchestrator with one runtime removes integration seams and the drift between systems, which is the durable operational win.
• Native versioning unlocks data CI — because branch/commit/merge is built in, a PR can branch the data, run the pipeline, and gate on an audit, making data changes reviewable and blockable exactly like code.
• Same open Iceberg storage — keeping storage as open Iceberg in your bucket means the whole bet is low-lock-in: you can leave, or run another engine over the same tables, without a migration.
• Named exceptions — conceding petabyte shuffles to Spark, a healthy dbt estate to dbt, and streaming to a streaming engine makes the recommendation a trustworthy engineering judgment rather than a sales claim.
• Cost — you trade four systems plus glue for one runtime and gain a blocking data-quality gate in CI, on unchanged open storage. The eliminated cost is multi-tool operation plus unreviewed data changes — O(1) runtime with code-style review, minus the specific workloads explicitly left to specialized engines.

Design
Topic — design
Design problems on platform trade-offs and CI/CD for data

Practice →

Optimization
Topic — optimization
Optimization problems on pipeline cost and tool selection

Practice →

---

Cheat sheet — Bauplan pipeline recipes

• The consolidation thesis. Bauplan folds four tools into one runtime — Spark's common-case compute, dbt's transforms, Nessie/lakeFS's data versioning, and the orchestrator's wiring — over open Iceberg in your own bucket. Adopt for the consolidation (fewer moving parts + data CI), not for a single feature. Concede petabyte shuffles to Spark and a healthy dbt estate to dbt.
• Function-as-a-pipeline template. A node is a decorated function; its arguments are its dependencies. @bauplan.model() + @bauplan.python('3.11', pip={...}) on top; parameters default to bauplan.Model('upstream', columns=[...], filter="..."); return a pandas/Arrow frame → it materializes as an Iceberg table. The DAG is read off the arguments — no orchestration file.
• Arrow between nodes. Data passes as Apache Arrow with no serialization tax; use to_arrow() + pyarrow.compute for zero-copy vectorized work on big intermediates, to_pandas() for pandas semantics on small ones. Persist boundaries, not every hop.
• Per-function environments. Each node pins its own interpreter + exact deps, so conflicting requirements coexist and runs are reproducible. Keep the pip set minimal and pinned — it is part of the node's contract and a precondition for trustworthy caching.
• Git-native / write-audit-publish. Never write to main. branch create <name> --from main (zero-copy) → run onto the branch → audit with plain queries (--ref <branch>) → branch merge <name> --into main only on green, else delete_branch. Production structurally cannot hold data that failed a check.
• Time travel + rollback. Every commit is an immutable snapshot. Read any past state with query ... --ref <commit>; recover a bad merge with branch reset main --to <good_commit> — exact, instant, history retained. Pin runs to input commits for reproducibility.
• Iceberg + pushdown. Storage is open Iceberg readable by Spark/Trino/DuckDB. Always pass columns and filter on source models so Iceberg prunes columns and skips non-matching files — "read a partition, not a table" is usually the biggest cost win.
• Serverless execution. No cluster to size or patch; each node an isolated container; zero idle cost; per-node scale. The boundary: not a distributed-shuffle engine — petabyte cross-partition joins stay on Spark.
• Content-hash caching. Re-runs skip nodes whose code, environment, and inputs are unchanged; only changed nodes + descendants recompute. Determinism is the contract — pass timestamps/seeds as explicit inputs and do no hidden I/O, or the cache is untrustworthy.
• Branch-per-PR CI for data. On a PR: branch the data, bauplan run onto the branch, run audit queries as blocking status checks, merge into main on green. Reviewers inspect the real candidate data by ref, not just the code diff — write-audit-publish driven by your PR flow.
• When NOT to use it. Petabyte distributed shuffles (Spark), a mature healthy dbt/SQL estate with no ops pain (stay), sub-second streaming (a streaming engine), non-Iceberg lakes or banned managed dependencies (poor fit). Name the deciding factor, not a vibe.

Frequently asked questions

What is Bauplan?

Bauplan is a serverless lakehouse platform whose single runtime does the jobs the modern data stack normally splits across several tools: it runs your pipeline as Python functions (the compute engine), it version-controls the underlying Iceberg tables with git semantics — branch, commit, merge, time travel — and it manages the DAG and per-node environments for you. Instead of assembling Spark for compute, dbt for transforms, and Nessie or lakeFS for data versioning and wiring them with an orchestrator, you write decorated Python functions whose arguments declare their dependencies, and the platform runs them with no cluster over open Iceberg tables in your own object storage. The pitch is consolidation: fewer moving parts, code-first pipelines, and safe branch-based data changes in one place.

How does function-as-a-pipeline work in Bauplan?

Each transformation is an ordinary Python function decorated to mark it as a DAG node, and its arguments declare its upstreams: a parameter defaulting to bauplan.Model('clean_orders') tells the platform this node depends on clean_orders, exactly the way dbt reads a ref(). The platform builds the whole DAG from those arguments — you never hand-wire an orchestration file — runs each node in an isolated serverless container with its own pinned pip environment, and passes data between nodes as Apache Arrow so there is no serialization tax on intermediates. Whatever frame a function returns (pandas or Arrow) is materialized as an Iceberg table under the node's name, which downstream nodes then reference. The node body can be any Python — a pandas reshape, an Arrow-compute join, a call to an ML model — which is the main difference from dbt's SQL-only models.

What does "git-native data" mean in Bauplan?

It means your Iceberg tables are under version control with the same operations you use on code: you create a branch (a zero-copy reference over the shared data files, so it is instant even on a huge lakehouse), every pipeline run or merge is a commit with a hash, and you publish changes by merging a branch into main. The workflow this enables is write-audit-publish: run a pipeline onto a fresh branch, audit the result with ordinary queries against that branch, and merge into production only if the checks pass — so bad data is a branch you delete rather than an incident in production. Because every commit is an immutable snapshot, you also get time travel (read any table as of a past commit) and rollback (reset main to a known-good commit), which together make data changes as reviewable and recoverable as code changes.

Bauplan vs dbt + Spark — what's the difference?

dbt models transforms as SQL with ref() dependencies but does not run the compute or version the data; Spark is a distributed compute engine you operate but is not a transformation framework or a versioning system — so the two are pieces you assemble. Bauplan collapses both roles: transforms are Python functions (a more general node body than SQL) whose arguments form the DAG, and they run on a serverless per-node runtime with no cluster to size, all with git-native versioning built in. The trade-offs are honest ones: dbt remains ideal for a mature pure-SQL mart estate, and Spark remains the right engine for petabyte cross-partition shuffles that a per-node serverless runtime is not built for. For mixed batch pipelines where the logic wants to be Python and cluster ops are a pain, Bauplan's consolidation is the win.

Bauplan vs lakeFS / Nessie for data branching?

lakeFS and Nessie add git-for-data as a separate layer — a catalog or a versioning proxy — that you run alongside your existing compute and storage, so branching becomes another system to operate and keep in sync. Bauplan builds branch/commit/merge/time-travel directly into its runtime, so data versioning is not a bolt-on but a native property of the same platform that runs your pipelines — which is what makes branch-per-PR CI (branch the data, run, audit, merge on green) feel seamless rather than stitched together. The trade-off is scope: lakeFS/Nessie are versioning-only and pair with whatever compute you already have (Spark, Trino, dbt), so if all you want is data versioning over an existing stack you are happy with, a dedicated layer may fit better. If you want versioning and the compute and the transform model unified, Bauplan bundles them.

When should I NOT use Bauplan?

Skip it when a single axis it does not specialize in dominates your workload or when there is no pain to remove. A workload built around petabyte cross-partition shuffles belongs on Spark, whose distributed executors exist for exactly that, not on a per-node serverless runtime. A mature, well-loved dbt/SQL estate with a fluent team and no operational pain is not worth a migration — Bauplan's value is removing cluster ops, split paradigms, and unsafe changes, so if you have none of those pains the consolidation buys little. Sub-second or continuous streaming is a category mismatch because Bauplan is batch-oriented; reach for Flink, Kafka Streams, or Spark Structured Streaming. And if your data is not (and will not be) Iceberg, or a managed dependency is disallowed by your platform constraints, the fit is poor.

Practice on PipeCode

• Drill the ETL pipeline practice library → for the multi-step, DAG-shaped pipeline problems that function-as-a-pipeline makes concrete — cleaning, joining, and materializing tables the way Bauplan nodes do.
• Rehearse the transformation logic on the data transformation practice library → for the pandas/Arrow reshapes, joins, and aggregations that live inside each Bauplan function body.
• Sharpen the platform-design axis with the system design practice library → for the consolidation, write-audit-publish, and branch-per-PR CI trade-offs a lakehouse platform decision turns on.
• Stack the prerequisites against PipeCode's broader 450+ data-engineering catalogue to anchor the git-native branching, Iceberg pushdown, serverless-execution, and data-validation patterns against real graded inputs — pipelines, transforms, audits, and design.

Lock in Bauplan and lakehouse muscle memory

Docs explain Bauplan's decorators and CLI. PipeCode drills explain the decision — when a Python function beats a SQL model, when `git-native` branching turns a scary backfill into a discardable branch, when `write-audit-publish` must gate every merge, and when the assembled stack still wins. Pipecode.ai is Leetcode for Data Engineering — pipeline and lakehouse practice tuned for the production trade-offs senior data engineers actually face.

Practice ETL pipeline problems →
Practice system design problems →