Data Version Control: lakeFS, Nessie & Dolt for Git-Like Data Branching

data version control is the discipline that lets you treat a petabyte of data the way you already treat a repository of code — branch it, commit it, diff two versions, merge a validated change, and roll back a bad one — and it is the capability senior data engineers reach for the moment "we overwrote the production table and can't get it back" stops being a hypothetical. A code repository has an undo button; most data platforms do not. When a backfill corrupts a fact table, when a schema migration lands half-applied, when an ML model trains on data that quietly changed underneath it, the team without version control is reduced to restoring from a nightly backup and hoping. The engineering problem is not "should data have versions" — every serious lakehouse now assumes it does — but at which layer you version it: the raw files in the object store, the table metadata in the catalog, or the individual rows inside a database.

This guide is the senior-DE walkthrough for the three tools that answer that question differently — lakeFS puts Git-like branches over object storage, Nessie puts them over an Iceberg/Delta catalog, and Dolt puts them inside a SQL database at the row and cell level. It works through the four axes an interviewer probes — versioning granularity, isolation model, commit/merge semantics, and time travel / rollback — then the canonical workflow for each engine, and the operational patterns that tie them together: the write-audit-publish gate, data branching for CI/CD, and the garbage collection that keeps versioned storage from growing without bound. Each section pairs a teaching block with a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the ETL practice library →, rehearse on the SQL practice library →, and sharpen the modelling axis with the design practice library →.

---

On this page

• Why the data version control choice determines everything downstream
• lakeFS — Git-like branching over object storage
• Nessie — a versioned catalog for Iceberg / Delta tables
• Dolt — the versioned SQL database
• Choosing and operating data version control
• Cheat sheet — data version control recipes
• Frequently asked questions
• Practice on PipeCode

---

1. Why data version control determines everything downstream

Three tools, three versioning layers — the choice binds your isolation, rollback, and reproducibility story

The one-sentence invariant: data version control is a picking exercise between versioning the raw files in an object store, versioning the table metadata in a catalog, or versioning the rows inside a SQL database — and each layer trades the granularity of what you can branch and diff against the engines you can use, the isolation you get for pipelines, and the shape of your time-travel and rollback story. The layer you pick decides whether a "branch" is a cheap pointer over S3 objects (lakeFS), a set of Iceberg snapshot references in a catalog (Nessie), or a full row-level fork of a MySQL-compatible database (Dolt). Every downstream workflow — the CI job that tests a pipeline on isolated data, the ML run that pins a training set, the audit that asks "what did this table look like last Tuesday" — hard-codes an assumption about which layer holds the version, and moving between layers later is a migration, not a config flag.

The four axes interviewers actually probe.

• Granularity. What is the unit of a commit? lakeFS commits a set of object changes across a whole repository of files (Parquet, CSV, images, anything). Nessie commits changes to Iceberg/Delta table metadata — a new snapshot pointer per table, many tables in one commit. Dolt commits individual row and cell changes inside tables. Coarser granularity (files) versions anything but diffs bluntly; finer granularity (rows) diffs precisely but only inside a SQL engine.
• Isolation. Can a pipeline get a private, writable copy of the data to work on without touching production? All three give you a branch, but the branch means different things: a lakeFS branch is a zero-copy view over the same objects; a Nessie branch is a private line of catalog history; a Dolt branch is a copy-on-write fork of the working set. Isolation is the headline reason teams adopt data version control at all.
• Commit / merge semantics. Is a merge a metadata pointer swap or a real three-way row merge? lakeFS and Nessie merges are fast metadata operations (with conflict detection at the object / table level). Dolt performs a genuine three-way merge of rows and can raise cell-level conflicts you resolve like a Git merge. The finer the merge, the more it behaves like code version control.
• Time travel / rollback. How do you read the past and undo a mistake? lakeFS: read @ a commit or branch, revert a commit on main. Nessie: query a table AT a branch/tag/timestamp, reset a branch to a prior hash. Dolt: AS OF a commit, dolt_reset / dolt_revert. Getting this axis wrong is the difference between a five-minute rollback and a restore-from-backup outage.

The 2026 reality — data version control is the write-audit-publish backbone.

• lakeFS is the default when your data lives as files in an object store and you want to version everything in the bucket regardless of format — Parquet lakes, ML feature files, images, models. It sits transparently in front of S3/GCS/Azure and speaks the S3 API, so Spark, Trino, and most tools work unchanged. If your unit of work is "a prefix full of objects," you pick lakeFS.
• Nessie is the default when your data is an Iceberg (or Delta) lakehouse and you want Git-like branches over the catalog so many tables commit and roll back together. Nessie is engine-agnostic — Spark, Flink, Trino, and Dremio all point at the same Nessie catalog and share its branches. If your unit of work is "a consistent set of tables," you pick Nessie.
• Dolt is the default when your data is relational and modest-to-mid scale and you want true row-and-cell versioning inside the database itself — reference data, config, curated dimensions, data you edit and want to diff. Dolt is a MySQL-compatible database with Git built in. If your unit of work is "rows I want to branch, diff, and merge like source code," you pick Dolt.
• The common thread is the write-audit-publish (WAP) pattern: write to an isolated branch, audit / validate it, and only publish (merge to main) when it passes. All three engines exist to make WAP cheap, and senior interviews probe WAP because it is the load-bearing correctness pattern for lakehouse ETL.

What interviewers listen for.

• Do you distinguish the three versioning layers — files vs table metadata vs rows — without prompting? — senior signal.
• Do you say "zero-copy branch" and can you explain why a branch does not duplicate the data? — required answer.
• Do you name the write-audit-publish pattern as the reason data version control exists, not as "an alternative to backups"? — senior signal.
• Do you describe a rollback as "revert / reset to a prior commit" rather than "restore from last night's snapshot"? — required answer.
• Do you raise garbage collection / retention unprompted — that versioned storage grows and must be reaped? — senior signal.

Worked example — the three-layer comparison table

Detailed explanation. The single most useful artifact for a data-version-control interview is a memorised comparison of the three layers. Every discussion converges on it within the first ten minutes; having it in your head is what separates a fluent answer from a hand-wave. Walk through building the table for a hypothetical lakehouse that has raw ingest files, curated Iceberg tables, and a small hand-maintained reference dataset.

• Raw zone. Thousands of Parquet and JSON files landing in s3://lake/raw/ from ingestion.
• Curated zone. Iceberg tables (orders, customers, line_items) queried by Spark and Trino.
• Reference data. A country_codes / fx_rates dataset a human edits and must be able to diff and roll back.

Question. Build the three-layer comparison and pick the version-control engine each zone should use.

Input.

Engine	Versions	Branch is	Merge is	Time travel
lakeFS	objects/files in a bucket	zero-copy view over objects	metadata pointer swap + object-path conflict check	read @ commit / branch
Nessie	Iceberg/Delta table metadata	private catalog history line	metadata merge across tables	query AT branch / tag / timestamp
Dolt	rows and cells in SQL tables	copy-on-write DB fork	three-way row merge, cell conflicts	AS OF commit / timestamp

Code.

Which layer holds the version?
==============================

  s3://lake/raw/*.parquet   ── files, any format ──▶  lakeFS
       (ingest, ML feature files, images, models)

  catalog.db.orders (Iceberg) ── table metadata ──▶  Nessie
       (many curated tables that must commit together)

  reference.country_codes   ── rows & cells ──────▶  Dolt
       (small, human-edited, diff-and-blame-worthy)

  Rule: version at the coarsest layer that still lets you
  diff and roll back the thing you actually care about.

Step-by-step explanation.

1. The raw zone is heterogeneous files with no shared table format, so the only layer that can version all of it is the object layer — lakeFS. You branch the whole raw/ prefix, run ingestion into the branch, and merge when the batch is complete. Granularity is "a set of objects," which is exactly right for files.
2. The curated zone is Iceberg tables that are queried together and must stay mutually consistent (an orders load and its line_items load should appear atomically). That is the catalog layer — Nessie — where one commit can advance several tables' metadata pointers at once. Versioning files here would be too coarse; versioning rows would lose the table-format benefits.
3. The reference dataset is small, relational, and human-edited — you want to see which cell changed and who changed it. That is the row layer — Dolt — where dolt_diff shows a before/after per cell and dolt_blame attributes each one. Versioning this as files would make a one-cell fix an opaque whole-file rewrite.
4. The choice is granularity-driven, not popularity-driven. Ask "what is the smallest thing I need to diff and roll back?" Files → lakeFS. A consistent set of tables → Nessie. Individual rows → Dolt.
5. In a real platform you often run more than one: lakeFS over the raw zone, Nessie over the curated lakehouse, Dolt for the reference data — each versioning the layer it fits, all feeding the same write-audit-publish discipline.

Output.

Zone	Recommended engine	Why
Raw ingest (mixed files)	lakeFS	version any object; branch the whole prefix
Curated Iceberg lakehouse	Nessie	multi-table atomic commits; engine-agnostic
Small reference data	Dolt	cell-level diff, blame, and three-way merge
ML training snapshots	lakeFS or Nessie	pin a commit/tag as the immutable training set

Rule of thumb. Never pick a data-version-control engine by brand familiarity. Pick it by the granularity of the thing you must diff and roll back — files (lakeFS), a set of tables (Nessie), or rows (Dolt). Write the three-layer table on a whiteboard first; the engine falls out of the layer.

Worked example — what interviewers actually probe

Detailed explanation. The senior data-version-control interview has a predictable arc: an ambiguous opener ("how would you stop a bad backfill from corrupting the warehouse?"), then progressive narrowing toward isolation, atomicity, and rollback. Candidates who name the write-audit-publish pattern and a specific engine score highest; candidates who say "we'd take a backup first" score lowest. Walk through the grading rubric.

• Ambiguous opener. "How do you protect production data from a bad pipeline run?" — invites the WAP pattern.
• Follow-up 1. "How do you give the pipeline a safe place to work?" — probes isolation / branching.
• Follow-up 2. "How do you promote the result atomically?" — probes merge semantics.
• Follow-up 3. "A load was wrong — how do you undo it?" — probes rollback / revert.
• Follow-up 4. "How do you read yesterday's data?" — probes time travel.

Question. Draft a five-minute senior answer that covers isolation, atomic publish, rollback, and time travel without waiting to be asked.

Input.

Interview signal	Weak answer	Senior answer
Protection	"take a backup before the job"	"write-audit-publish on an isolated branch"
Isolation	"run against a copy of the table"	"zero-copy branch; production is untouched"
Atomic publish	"swap the table at the end"	"merge the branch to main in one commit"
Rollback	"restore from last night"	"revert / reset to the prior commit"
Time travel	"we keep dated snapshots"	"read AT / AS OF a commit or tag"

Code.

Senior data-version-control answer template (5 minutes)
=======================================================

Minute 1 — name the pattern up front
  "I'd run write-audit-publish on a data-version-control system:
   branch, transform, validate, then merge to main only if it passes."

Minute 2 — isolation
  "The pipeline writes to a fresh branch, not to production. On lakeFS
   or Nessie that branch is zero-copy, so isolation is nearly free and
   production readers never see the in-progress data."

Minute 3 — atomic publish
  "When validation passes, I merge the branch into main in a single
   commit. Consumers flip from the old state to the new state atomically
   — no half-written table, no torn read."

Minute 4 — rollback
  "If a bad load slips through, rollback is 'revert the merge commit' or
   'reset main to the prior hash' — seconds, not a restore-from-backup
   outage. The old commit is still there, immutable."

Minute 5 — time travel + retention
  "Any consumer can read the table AS OF a commit or a tag for
   reproducibility — pin an ML training set to a tag. And I run garbage
   collection to expire unmerged branches and old commits so versioned
   storage stays bounded."

Step-by-step explanation.

1. Minute 1 frames the whole answer around write-audit-publish. Weak candidates dive into tools ("we'd use a staging table and…") before naming the pattern; naming WAP signals you have run isolated pipelines before.
2. Minute 2 addresses isolation with the zero-copy point. Saying "the branch does not copy the data, it copies the pointers" preempts the interviewer's "isn't that expensive?" and shows you understand the metadata model underneath.
3. Minute 3 is the atomicity probe. "Merge in a single commit" is the differentiator versus "swap the table at the end," which invites torn reads. The atomic publish is why WAP is safe.
4. Minute 4 is rollback. "Revert / reset to a prior commit" is the version-control answer; "restore from last night" is the backup answer, and mixing them up is the tell that you have not used a versioned system. The prior commit being immutable and still present is the key property.
5. Minute 5 covers reproducibility (tags for ML training sets) and retention (garbage collection). Raising GC unprompted is a senior signal — it shows you know versioned storage is not free and must be reaped.

Output.

Grading criterion	Weak score	Senior score
Names WAP in minute 1	rare	mandatory
Explains zero-copy isolation	rare	required
Atomic merge, not table swap	occasional	mandatory
Rollback = revert/reset commit	rare	senior signal
Raises time travel + GC	rare	senior signal

Rule of thumb. The senior answer is a five-minute monologue: write-audit-publish, zero-copy isolation, atomic merge, revert-to-commit rollback, tag-based time travel, and garbage collection. Rehearse it once; deploy it every interview.

Worked example — the "pick the layer" decision tree

Detailed explanation. Given a new dataset to version, the senior architect runs a short decision tree in their head. Codifying the tree makes the interview answer reproducible: any interviewer can hand you a dataset and you can walk the tree out loud. Walk it with three canonical datasets — a raw ML image corpus, a curated Iceberg warehouse, and a hand-edited pricing table.

• Q1. Is the data a bag of arbitrary files (any format), or is it tabular? → files = lakeFS; tabular = go to Q2.
• Q2. Is it an Iceberg/Delta lakehouse where many tables must version and roll back together? → yes = Nessie; no = go to Q3.
• Q3. Is it small-to-mid, relational, and human-edited where you need cell-level diff and blame? → yes = Dolt; no = reconsider whether you need row-level at all (Nessie/Iceberg time travel may suffice).
• Q4 (parallel). Do you need it queryable by external engines (Spark/Trino/Flink) unchanged? → lakeFS (S3 API) and Nessie (catalog) yes; Dolt speaks MySQL, so plan a connector.

Question. Walk the decision tree for the three datasets and record the engine each ends up with.

Input.

Dataset	Q1 (files?)	Q2 (lakehouse?)	Q3 (row-level edit?)
ML image corpus	yes (images)	—	—
Curated Iceberg warehouse	no (tables)	yes	—
Pricing / reference table	no (tables)	no	yes

Code.

# Decision-tree helper (illustrative)
def pick_dvc_engine(is_files: bool,
                    is_iceberg_lakehouse: bool,
                    needs_row_level_edit: bool) -> str:
    """Return the data-version-control engine for a dataset."""
    if is_files:
        return "lakeFS"                       # version any object in the bucket
    if is_iceberg_lakehouse:
        return "Nessie"                       # git-like catalog over table metadata
    if needs_row_level_edit:
        return "Dolt"                         # row & cell versioning in a SQL DB
    return "Nessie/Iceberg time travel"       # tabular but no per-row branching need


# Walk the three datasets
print(pick_dvc_engine(True,  False, False))
# -> lakeFS

print(pick_dvc_engine(False, True,  False))
# -> Nessie

print(pick_dvc_engine(False, False, True))
# -> Dolt

Step-by-step explanation.

1. Dataset 1 — an ML image corpus is arbitrary binary files, so Q1 short-circuits to lakeFS. You branch s3://lake/images/, add a labelled batch on the branch, and merge when it is reviewed; the commit is your immutable training-set reference.
2. Dataset 2 — a curated Iceberg warehouse is tabular and the tables must roll back together (a fact and its dimension), so Q1 = no, Q2 = yes → Nessie. One Nessie commit advances several tables' snapshots atomically, and every engine pointed at the catalog shares the branch.
3. Dataset 3 — a pricing/reference table is small, relational, and edited by a human who wants to see and justify each change, so Q1 = no, Q2 = no, Q3 = yes → Dolt. dolt_diff and dolt_blame give the per-cell audit trail that files and Iceberg snapshots cannot.
4. The parallel Q4 (engine access) is orthogonal but decisive in practice: lakeFS and Nessie plug into existing Spark/Trino stacks with minimal change, while Dolt is a MySQL-compatible database you connect to as such — great for serving reference data, a bridge to build for a Spark lakehouse.
5. If none of Q1–Q3 clearly fire, default to the format you already run: an Iceberg shop gets Nessie-style time travel almost for free, so do not add Dolt unless you genuinely need per-row branching and merge.

Output.

Dataset	Engine	Access pattern
ML image corpus	lakeFS	S3 API; Spark / ML tools unchanged
Curated Iceberg warehouse	Nessie	shared catalog; Spark / Trino / Flink
Pricing / reference table	Dolt	MySQL protocol; connector to the lakehouse

Rule of thumb. The three-question decision tree is whiteboard-friendly. Files → lakeFS, a consistent set of lakehouse tables → Nessie, human-edited rows → Dolt. Practice walking it end-to-end so an interviewer can hand you any dataset and get an engine name in under 60 seconds.

Senior interview question on data version control selection

A senior interviewer often opens with: "Your team keeps corrupting the production warehouse with bad backfills, and recovery means a four-hour restore from last night's snapshot. Walk me through how you'd introduce data version control so a bad load is isolated before it lands and rollback is a seconds-long operation — the pattern, the engine choice for a mixed file-plus-Iceberg lake, and the failure modes you'd guard against."

Solution Using write-audit-publish on branched data with atomic merge and revert-based rollback

Target architecture — WAP over a mixed lake
===========================================

  raw files  ─▶ lakeFS repo "lake"          (branch the raw prefix)
  curated    ─▶ Nessie catalog "warehouse"  (branch the Iceberg tables)

  Every batch runs on a fresh branch:
     1. branch        (zero-copy; production untouched)
     2. write         (ingest / transform into the branch)
     3. audit         (row counts, null checks, referential checks)
     4. publish       (merge branch -> main in ONE commit)  or  discard

# Orchestrator step — WAP for one curated batch on Nessie via Spark
from datetime import date

BRANCH = f"etl_orders_{date.today():%Y%m%d}"

def run_wap_batch(spark):
    # 1. BRANCH — private, zero-copy line of catalog history off main
    spark.sql(f"CREATE BRANCH IF NOT EXISTS {BRANCH} IN nessie FROM main")
    spark.sql(f"USE REFERENCE {BRANCH} IN nessie")

    # 2. WRITE — transform into the branch; production readers on main see nothing
    spark.sql("""
        INSERT INTO warehouse.db.orders
        SELECT * FROM staging.orders_delta
    """)

    # 3. AUDIT — validate the branch before anyone can see it
    checks = spark.sql("""
        SELECT
          (SELECT count(*) FROM warehouse.db.orders)                       AS row_count,
          (SELECT count(*) FROM warehouse.db.orders WHERE order_id IS NULL) AS null_ids,
          (SELECT count(*) FROM warehouse.db.orders o
             LEFT JOIN warehouse.db.customers c ON c.id = o.customer_id
             WHERE c.id IS NULL)                                           AS orphan_fk
    """).first()

    if checks.null_ids > 0 or checks.orphan_fk > 0 or checks.row_count == 0:
        # discard: drop the branch; main was never touched
        spark.sql(f"DROP BRANCH {BRANCH} IN nessie")
        raise ValueError(f"audit failed: {checks}")

    # 4. PUBLISH — atomic merge into main; consumers flip in one commit
    spark.sql(f"MERGE BRANCH {BRANCH} INTO main IN nessie")
    spark.sql(f"DROP BRANCH {BRANCH} IN nessie")

-- Rollback path — a bad load merged anyway; undo it on main in seconds
-- Find the offending commit, then reset main back one commit.
SELECT * FROM warehouse.db.`orders@main` ORDER BY committed_at DESC;  -- inspect refs
-- Reset main to the parent of the bad merge (Nessie assign/reset by hash):
CALL nessie.assign_branch('main', 'main~1');    -- illustrative; old data reappears

Step-by-step trace.

Step	Before (backup-only)	After (WAP + version control)
Bad load lands	overwrites production	lands on an isolated branch
Detection	after consumers see it	at the audit gate, before publish
Blast radius	whole warehouse	one branch, discarded
Rollback mechanism	restore from snapshot	revert/reset to prior commit
Rollback time	hours	seconds
Reproducibility	dated backup files	immutable commit / tag

After the change, every batch runs on a throwaway branch; a failed audit drops the branch and production never saw the data; a successful audit merges in one commit so consumers flip atomically; and the rare bad load that slips through is undone by resetting main to the prior hash. The four-hour restore disappears from the runbook.

Output:

Metric	Before	After
Corruption reaches production	often	only past the audit gate
Mean time to rollback	~4 h (restore)	seconds (reset commit)
Isolation cost per batch	full table copy	zero-copy branch
Reproducible training sets	manual snapshots	tag a commit
Storage overhead	backup copies	versioned deltas + GC

Why this works — concept by concept:

• Write-audit-publish — the batch writes to a branch, is validated in isolation, and only merges to main on success. The audit gate is the correctness invariant: bad data is caught before any consumer can read it, not after.
• Zero-copy branch — a lakeFS/Nessie branch copies pointers, not bytes, so isolating a pipeline costs almost nothing. This is why WAP is affordable per-batch rather than a special-occasion event.
• Atomic merge — publishing is a single commit that advances main from the old state to the new state. Consumers never observe a half-applied load; there is no torn-read window the way a multi-statement table swap has.
• Revert / reset rollback — because every prior state is an immutable commit still present in history, undo is a pointer move (reset main to the prior hash), not a restore. Recovery time collapses from hours to seconds.
• Cost — versioned metadata plus deltas, one branch per batch, and a garbage-collection job to expire unmerged branches and aged commits. The eliminated cost is the recurring restore-from-backup outage and the un-reproducible ML run. Net O(changes) storage per commit versus O(full-table) per backup.

Design
Topic — design
Design problems on versioned data platforms

Practice →

ETL Topic — etl ETL problems on write-audit-publish pipelines

Practice →

---

2. lakeFS — Git-like branching over object storage

lakeFS puts branch, commit, and merge over S3/GCS/Azure — version any object, isolate any pipeline, roll back any commit

The mental model in one line: lakeFS is a versioning layer that sits transparently in front of an object store (S3, GCS, Azure Blob), exposes the same S3 API, and adds Git-like branch / commit / merge / revert over everything in the bucket — branches are zero-copy because they share the underlying immutable objects and only diverge as new writes create new objects, so isolating a pipeline or pinning a dataset costs metadata, not a data copy. Any file format works because lakeFS versions objects, not tables; Spark, Trino, pandas, and ML tooling read and write through the S3 gateway unchanged.

The four axes for lakeFS.

• Granularity. A commit is a set of object changes across the repository — new, changed, and deleted objects under any prefix. You branch and version whole prefixes (raw/, curated/, models/), so lakeFS versions data of any format, not just tables.
• Isolation. A branch is a zero-copy, writable view of the repository at a point in time. A pipeline gets its own branch, writes freely, and production readers on main see nothing until merge. This is the cheapest strong isolation available for a file lake.
• Commit / merge semantics. Merge is a fast metadata operation that reconciles the branch's object changes into main; conflicts are detected at the object-path level (both branches changed the same object). There is no row-level merge — the unit is the object.
• Time travel / rollback. Read the repository @ any commit or branch (lakefs://repo/<ref>/path); revert a commit to undo it on a branch; every commit is immutable and addressable, so rollback is a pointer move.

The anatomy — repository, branch, commit, ref.

• Repository. A versioned namespace backed by one storage prefix (e.g. s3://my-bucket/lakefs/). Objects are content-addressed and immutable underneath.
• Branch. A named, movable pointer to a commit, plus a staging area for uncommitted changes. main is the default. Creating a branch is O(1) — it copies a pointer.
• Commit. An immutable snapshot of the whole repository with a parent, message, and metadata. Addressable by hash forever (until GC).
• Ref. Anything you can resolve to a commit: a branch name, a tag, a commit hash, or an expression like main~1 (one commit before main).

How zero-copy actually works.

• Content-addressed objects. Each physical object is stored once, keyed by content. A branch that does not modify an object simply points at the same physical object as main.
• Copy-on-write. Writing to a file on a branch creates a new physical object and updates only the branch's metadata range — main still points at the old object. The two branches diverge only for the objects that actually changed.
• Merge = range reconciliation. Merging replays the branch's metadata changes onto main, which is why it is fast and why conflicts are per-object.

Common interview probes on lakeFS.

• "Why doesn't a lakeFS branch double your storage?" — required answer: content-addressed objects + copy-on-write; branches share unchanged objects.
• "How do you isolate an ETL job?" — branch, write to the branch, audit, merge to main.
• "How do you roll back a bad merge?" — revert the commit on main; the prior objects are still referenced.
• "How do consumers read without seeing in-progress data?" — they read main; the pipeline writes to a branch until merge.

Worked example — create a branch, write, commit, and merge

Detailed explanation. The canonical lakeFS flow: create an ingest branch off main, write new Parquet into it, commit, then merge to main so consumers see the batch atomically. Use both the lakectl CLI and the Python SDK so the pattern is clear from a shell script or an orchestrator.

• Branch. ingest-2026-08-18 off main.
• Write. Spark writes Parquet to the branch via the S3 gateway path.
• Commit + merge. Commit the branch, then merge to main.

Question. Show the branch → write → commit → merge cycle for one ingest batch.

Input.

Parameter	Value
Repository	lakefs://lake
Source branch	main
Work branch	ingest-2026-08-18
Path written	raw/orders/dt=2026-08-18/
Engine	Spark (S3A → lakeFS gateway)

Code.

# 1. Create a zero-copy branch off main
lakectl branch create lakefs://lake/ingest-2026-08-18 \
    --source lakefs://lake/main

# 2. (Spark writes Parquet to the branch — see Python below)

# 3. Commit the staged objects on the branch
lakectl commit lakefs://lake/ingest-2026-08-18 \
    -m "ingest orders dt=2026-08-18" \
    --meta batch_id=2026-08-18 --meta rows=128934

# 4. Merge the branch into main (atomic publish)
lakectl merge lakefs://lake/ingest-2026-08-18 lakefs://lake/main

# Orchestrator version — lakeFS Python SDK + Spark write
import lakefs

repo = lakefs.repository("lake")

# 1. Branch (idempotent create)
branch = repo.branch("ingest-2026-08-18").create(source_reference="main", exist_ok=True)

# 2. Write into the branch via the S3A gateway (spark configured for lakeFS endpoint)
(spark.read.parquet("s3a://staging/orders/2026-08-18/")
      .write.mode("append")
      .parquet("s3a://lake/ingest-2026-08-18/raw/orders/dt=2026-08-18/"))

# 3. Commit the staged changes
commit = branch.commit(
    message="ingest orders dt=2026-08-18",
    metadata={"batch_id": "2026-08-18", "rows": "128934"},
)
print("committed", commit.get_commit().id)

# 4. Merge to main — production readers now see the batch atomically
branch.merge_into(repo.branch("main"))

Step-by-step explanation.

1. branch create off main is O(1): lakeFS records a new pointer at main's current commit and a staging area. No objects are copied — the branch shares every object with main until something is written.
2. The Spark write targets the branch's path (s3a://lake/ingest-2026-08-18/...). Copy-on-write means these new Parquet files become new physical objects visible only on the branch; main is unchanged and consumers reading main see the pre-batch state.
3. commit freezes the branch's staged objects into an immutable snapshot with a message and metadata. The metadata (batch_id, rows) is queryable later and is how you correlate a commit to a pipeline run.
4. merge reconciles the branch's object changes into main in one operation. Because nothing else touched those object paths, the merge is conflict-free and fast; consumers flip from the old state to the new state atomically at the merge commit.
5. Had the write been wrong, you would simply not merge — drop the branch and main never saw it. That "discard the branch" escape hatch is the whole point of isolation.

Output.

Ref	Sees raw/orders/dt=2026-08-18/?	Notes
main (before merge)	no	consumers unaffected during ingest
ingest-2026-08-18	yes	isolated work branch
main (after merge)	yes	atomic flip at the merge commit
main~1	no	prior state still addressable

Rule of thumb. Treat every ingest batch as a branch: create off main, write, commit with run metadata, then merge. Never write directly to main — the branch is your undo button and your isolation boundary at once.

Worked example — write-audit-publish with a hook-enforced gate

Detailed explanation. Isolation is only half of WAP; the audit has to be enforced, not optional. lakeFS supports pre-merge hooks (Lua or webhook actions) that run automatically when someone tries to merge into main and block the merge if a validation fails. Wire a data-quality check as a merge gate so a bad branch can never reach production even if a human clicks merge.

• The gate. A pre-merge hook on main runs a validation and fails the merge on violation.
• The check. Row count > 0, no null primary keys, schema matches the contract.
• The effect. Merge is rejected; the branch stays isolated; nobody sees bad data.

Question. Configure a lakeFS pre-merge hook that blocks a merge into main when the branch fails data-quality checks.

Input.

Component	Value
Protected branch	main
Trigger	pre-merge
Validation	non-empty, no null order_id, expected columns
On failure	merge rejected; exit non-zero

Code.

# .lakefs/actions/pre_merge_orders.yaml — committed to the repo
name: audit-orders-before-merge
on:
  pre-merge:
    branches:
      - main
hooks:
  - id: validate_orders
    type: airflow          # or 'webhook' / 'lua'
    properties:
      url: "http://airflow:8080/api/v1/dags/audit_orders/dagRuns"
      # Airflow DAG runs the checks against the SOURCE branch of the merge

# audit_orders task (invoked by the hook) — fails => merge blocked
def audit_orders(source_ref: str):
    import lakefs
    from pyspark.sql import functions as F

    # Read the branch that is trying to merge (NOT main)
    df = spark.read.parquet(f"s3a://lake/{source_ref}/raw/orders/")

    n         = df.count()
    null_ids  = df.filter(F.col("order_id").isNull()).count()
    expected  = {"order_id", "customer_id", "total_cents", "status"}
    missing   = expected - set(df.columns)

    problems = []
    if n == 0:            problems.append("empty batch")
    if null_ids > 0:      problems.append(f"{null_ids} null order_id")
    if missing:           problems.append(f"missing columns {missing}")

    if problems:
        # Non-zero exit => lakeFS aborts the merge; main is untouched
        raise SystemExit("AUDIT FAILED: " + "; ".join(problems))
    print(f"audit passed: {n} rows, 0 null ids, schema ok")

Step-by-step explanation.

1. The action YAML lives in the repository under .lakefs/actions/ and binds a pre-merge hook to main. lakeFS runs it automatically whenever any branch attempts to merge into main — the gate cannot be skipped by a well-meaning human clicking "merge."
2. The hook invokes an audit task (here an Airflow DAG; a webhook or embedded Lua script works the same way) and passes the source ref of the merge — the branch being promoted — so the check validates the candidate data, not the already-published main.
3. The audit reads the branch's objects through the S3 gateway and runs concrete checks: non-empty, no null order_id, and the expected schema. These are cheap and catch the common corruption modes.
4. A non-zero exit from the hook causes lakeFS to abort the merge. The branch remains exactly as it was, isolated; main is untouched; consumers see nothing. The bad batch is contained by construction.
5. On success the hook exits zero and lakeFS proceeds with the merge. The audit is now a hard gate in the promotion path — WAP enforced by the platform, not by pipeline discipline alone.

Output.

Branch state	Hook result	Merge outcome
128,934 rows, 0 null ids, schema ok	pass	merge proceeds; main advances
0 rows (empty batch)	fail	merge aborted; branch preserved
42 null order_id	fail	merge aborted; branch preserved
missing status column	fail	merge aborted; branch preserved

Rule of thumb. Make the audit a pre-merge hook on main, not a step the pipeline chooses to run. If the only path to production is a merge, and merges are gated by hooks, then bad data is unmergeable by construction — that is the strongest form of write-audit-publish.

Worked example — revert a bad commit and time-travel a query

Detailed explanation. Even with a gate, sometimes bad data merges (a check you did not write yet). lakeFS rollback is revert: it creates a new commit that undoes a prior one, so main returns to the good state while preserving full history. And time travel lets any consumer read the repository as it was at any commit for reproducibility or forensics. Walk through both.

• Revert. Undo the bad merge commit on main with lakectl branch revert.
• Time travel. Read main at the commit before the bad merge.
• Forensics. Diff the bad commit against its parent to see exactly what changed.

Question. Roll back a bad merge on main and show how to read the pre-merge state.

Input.

Component	Value
Bad commit	c0ffee… (a merge that landed bad rows)
Rollback	revert on main
Time-travel read	main at c0ffee~1
Diff	c0ffee~1 vs c0ffee

Code.

# 1. Inspect recent history on main
lakectl log lakefs://lake/main --amount 5

# 2. See exactly what the bad commit changed vs its parent
lakectl diff lakefs://lake/main@c0ffee~1 lakefs://lake/main@c0ffee

# 3. Revert the bad commit — creates a NEW commit undoing it; history preserved
lakectl branch revert lakefs://lake/main c0ffee --parent-number 1 --yes

# 4. Verify main is back to good state
lakectl log lakefs://lake/main --amount 3

# Time travel — read main AS OF the commit before the bad merge (reproducibility)
good_ref = "c0ffee~1"   # parent of the bad commit

df_good = spark.read.parquet(f"s3a://lake/main@{good_ref}/raw/orders/dt=2026-08-18/")
print("rows in the last-good state:", df_good.count())

# Pin an ML training set to an immutable commit so a re-run is byte-identical
TRAINING_REF = "c0ffee~1"
train_df = spark.read.parquet(f"s3a://lake/main@{TRAINING_REF}/curated/features/")
# model.fit(train_df) — reproducible: the ref never moves

Step-by-step explanation.

1. lakectl log shows the immutable commit chain on main. Each commit has a hash, a message, and the run metadata you attached — enough to identify the offending merge without guessing.
2. lakectl diff <parent> <bad> shows the exact object-level changes the bad commit introduced (added/changed/deleted objects). This is the forensic step: you confirm what went wrong before undoing it.
3. branch revert main c0ffee creates a new commit whose content is main minus the bad commit's changes. Crucially it does not rewrite history — the bad commit still exists and is auditable; main simply moves forward to a state equivalent to before it. --parent-number 1 tells lakeFS which parent of the merge to revert toward.
4. Reading main@c0ffee~1 resolves main's history one commit back and reads the repository exactly as it was then. This is time travel: no separate snapshot was needed because every commit is a complete, addressable state.
5. Pinning a training set to a fixed ref (c0ffee~1) makes an ML run reproducible: the ref is immutable, so re-training months later reads byte-identical inputs. This is the reproducibility half of data version control that backups cannot provide.

Output.

Action	main head after	Consumers see
bad merge lands	c0ffee	corrupted rows
revert c0ffee	revert-of-c0ffee (new commit)	last-good state restored
read main@c0ffee~1	unchanged	historical state, read-only
history	c0ffee still present	full audit trail preserved

Rule of thumb. Roll back with revert (a forward-moving undo commit), not by deleting history — you keep the audit trail and can always diff what went wrong. Use immutable refs (~1, tags, hashes) to pin reproducible reads; a ref that never moves is the only honest way to reproduce a training set.

Senior interview question on lakeFS

A senior interviewer might ask: "You have a 50 TB Parquet lake on S3 feeding Spark and an ML feature pipeline. Product wants (a) every nightly ingest isolated so a bad batch cannot corrupt production, (b) reproducible ML training sets, and (c) seconds-long rollback. Design the lakeFS setup — repository layout, the branch-per-batch flow, the enforced audit, rollback, and how you keep versioned storage from growing without bound."

Solution Using branch-per-batch, hook-gated merge, tag-pinned training sets, and GC retention

# 1. Repository over the existing bucket (data stays in S3; lakeFS adds versioning)
lakectl repo create lakefs://lake s3://prod-data-lake/lakefs --default-branch main

# 2. Protect main: only hook-gated merges may advance it
#    (.lakefs/actions/pre_merge.yaml runs the audit DAG on the source branch)

# 3. Tag reproducible training snapshots (immutable, never move)
lakectl tag create lakefs://lake/train-2026-08-18 lakefs://lake/main

# 4. Nightly branch-per-batch flow (orchestrator)
import lakefs
from datetime import date

repo = lakefs.repository("lake")

def nightly_ingest():
    b = f"ingest-{date.today():%Y%m%d}"
    branch = repo.branch(b).create(source_reference="main", exist_ok=True)

    # write ingest into the branch (Spark, via s3a://lake/<branch>/...)
    ingest_into(branch=b)

    branch.commit(message=f"ingest {b}", metadata={"batch": b})

    # merge is gated: the pre-merge hook runs the audit and blocks on failure
    try:
        branch.merge_into(repo.branch("main"))
    except lakefs.exceptions.HookFailedException as e:
        # audit failed => branch preserved for inspection; main untouched
        alert(f"ingest {b} blocked by audit: {e}")
        return
    # optional: tag the new main as the day's reproducible snapshot
    repo.tag(f"train-{date.today():%Y%m%d}").create(source_ref="main", exist_ok=True)

# 5. Retention / garbage collection — reap unmerged branches + unreachable objects
#    Rules: keep tagged commits forever; expire merged-branch objects after 30d.
lakectl branch list lakefs://lake | awk '$1 ~ /^ingest-/ {print $1}' \
  | while read b; do lakectl branch delete "lakefs://lake/$b" --yes; done   # stale work branches

# GC config (committed as retention rules); run the GC Spark job on a schedule
#   default_retention_days: 30
#   branches: [{ branch_id: main, retention_days: 90 }]

Step-by-step trace.

Concern	Mechanism	Result
Isolation	branch per nightly batch	production main untouched during ingest
Enforced audit	pre-merge hook on main	bad batch is unmergeable
Atomic publish	merge_into(main)	consumers flip in one commit
Reproducible ML	immutable tags per day	re-train reads byte-identical inputs
Rollback	revert the merge commit	last-good state in seconds
Storage growth	GC + branch cleanup	versioned deltas bounded by retention

After deployment, each nightly ingest runs on its own branch and can only reach main through a hook-gated merge; failed audits leave the branch quarantined and main clean; daily tags give the ML team immutable, reproducible training references; a bad merge is undone with revert in seconds; and a scheduled GC job plus work-branch cleanup keeps the versioned footprint proportional to real change, not to the number of runs.

Output:

Metric	Value
Isolation cost per batch	zero-copy branch (metadata only)
Bad-batch blast radius	one branch; never reaches main
Rollback time	seconds (revert commit)
Training reproducibility	immutable tag per day
Storage overhead	changed objects only + 30–90d retention
Engine compatibility	Spark / ML tools unchanged (S3 API)

Why this works — concept by concept:

• Branch-per-batch isolation — every run gets a zero-copy branch, so a bad load lives and dies on that branch. Production main is only ever advanced by an explicit, gated merge.
• Hook-gated merge — the pre-merge action makes the audit a hard precondition of publishing. Data quality is enforced by the platform, not left to pipeline discipline that can be forgotten.
• Immutable tags for reproducibility — a tag is a ref that never moves, so an ML training set pinned to it re-reads byte-identical inputs forever. This is the property backups cannot guarantee.
• Revert-based rollback — undo is a forward commit that restores the prior state while preserving history, so recovery is seconds and the incident stays auditable.
• Cost — content-addressed storage of changed objects, one GC/retention job, and the merge-hook infrastructure. Compared with full-copy staging tables and nightly restores, lakeFS spends O(changed objects) instead of O(dataset) and turns a multi-hour restore into a seconds-long revert. GC keeps the delta footprint bounded.

ETL
Topic — etl
ETL problems on isolated, branch-based ingestion

Practice →

Design Topic — design Design problems on object-store data lakes

Practice →

---

3. Nessie — a versioned catalog for Iceberg / Delta tables

Nessie puts Git-like branches and tags over the catalog — many Iceberg tables commit, roll back, and time-travel together

The mental model in one line: Nessie is a transactional catalog that tracks the metadata pointers of your Iceberg (and Delta) tables as a Git-like history — branches and tags reference table snapshots, one commit can advance many tables at once atomically, and every engine that points at the Nessie catalog (Spark, Flink, Trino, Dremio) shares the same branches — so you get cross-table isolation and multi-table transactions without ever copying data files. Nessie versions the catalog, not the objects: a branch is a private line of catalog history, and a merge reconciles table-metadata references, not row data.

The four axes for Nessie.

• Granularity. A commit is a change to one or more tables' metadata (new snapshots, schema changes, adds/drops). Because the unit is the table snapshot, a single Nessie commit can move several tables together — the multi-table transaction that plain Iceberg lacks.
• Isolation. A branch is an isolated line of catalog history. A pipeline works on its own branch, mutating tables freely; readers on main see the pre-branch snapshots. Isolation spans all tables on the branch, so cross-table consistency is preserved.
• Commit / merge semantics. Merge reconciles the branch's table references into main. Conflicts are detected per table (both branches produced new snapshots for the same table); there is no row-level merge — Nessie versions metadata, and the data files are immutable Iceberg objects.
• Time travel / rollback. Query a table AT a branch, tag, or timestamp; reset/assign a branch to a prior hash to roll back many tables at once. Tags give immutable, named releases (release-v1) for reproducibility.

The anatomy — ref, commit, table pointer.

• Ref. A named pointer into catalog history: a branch (movable, e.g. main, etl) or a tag (immutable, e.g. release-2026-08). Also addressable by commit hash.
• Commit. An atomic change to the catalog — one or more table snapshot updates — with a parent, author, and message. Immutable and hash-addressed.
• Table pointer. For each table, the current metadata location (the Iceberg metadata JSON / snapshot). Nessie versions these pointers; the Parquet data files are ordinary immutable objects shared across refs.
• Engine-agnostic. Spark, Flink, Trino, and Dremio all resolve tables through the same Nessie catalog, so a branch created in Spark is visible in Trino instantly.

Why multi-table atomicity matters.

• Plain Iceberg commits one table at a time. Loading a fact and its dimension in "one logical batch" leaves a window where the fact is updated but the dimension is not.
• Nessie lets you write both on a branch and merge in one commit, so consumers on main flip from "both old" to "both new" with no inconsistent intermediate state.
• This is the catalog analogue of a database transaction spanning multiple tables — the reason Nessie exists.

Common interview probes on Nessie.

• "What does a Nessie branch version — files or tables?" — required answer: table metadata (snapshot pointers) in the catalog.
• "How do you commit two tables atomically?" — write both on a branch, merge the branch in one commit.
• "How do you pin a reproducible release?" — an immutable tag on a commit.
• "Which engines can share a Nessie branch?" — any engine pointed at the Nessie catalog (Spark, Flink, Trino, Dremio).

Worked example — branch the catalog, write, and merge

Detailed explanation. The canonical Nessie flow with Spark's SQL extensions: create a catalog branch, switch the session to it, write to tables, then merge the branch into main. Readers on main see nothing until the merge. Walk through the SQL.

• Branch. etl off main.
• Switch. USE REFERENCE etl.
• Write + merge. Insert into a table on the branch, then MERGE BRANCH etl INTO main.

Question. Show the branch → write → merge cycle for a single-table load on Nessie.

Input.

Parameter	Value
Catalog	nessie (Iceberg + Nessie)
Source ref	main
Work branch	etl
Table	warehouse.db.orders
Engine	Spark SQL (Nessie extensions)

Code.

-- 1. Create a branch off main (catalog history forks; no data copied)
CREATE BRANCH IF NOT EXISTS etl IN nessie FROM main;

-- 2. Point this Spark session at the branch
USE REFERENCE etl IN nessie;

-- 3. Write into the table ON THE BRANCH; main readers see nothing yet
INSERT INTO warehouse.db.orders
SELECT order_id, customer_id, total_cents, status
FROM   staging.orders_incoming;

-- 4. Inspect the branch in isolation
SELECT count(*) FROM warehouse.db.orders;   -- resolves against 'etl'

-- 5. Publish: merge the branch into main in one atomic commit
MERGE BRANCH etl INTO main IN nessie;

-- 6. Clean up the work branch
DROP BRANCH etl IN nessie;

# Spark session config that makes `nessie` a Nessie-backed Iceberg catalog
spark_conf = {
    "spark.sql.extensions":
        "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions,"
        "org.projectnessie.spark.extensions.NessieSparkSessionExtensions",
    "spark.sql.catalog.nessie": "org.apache.iceberg.spark.SparkCatalog",
    "spark.sql.catalog.nessie.catalog-impl": "org.apache.iceberg.nessie.NessieCatalog",
    "spark.sql.catalog.nessie.uri": "http://nessie:19120/api/v2",
    "spark.sql.catalog.nessie.ref": "main",              # default ref
    "spark.sql.catalog.nessie.warehouse": "s3a://lake/warehouse",
}

Step-by-step explanation.

1. CREATE BRANCH etl FROM main forks the catalog at main's current commit. No Parquet is copied — the branch simply references the same table snapshots main does until something is written.
2. USE REFERENCE etl scopes the session so every table read/write resolves against the etl branch. This is how isolation works: two sessions on two branches see two independent versions of the same tables.
3. The INSERT writes new Iceberg data files and produces a new snapshot for warehouse.db.orders on the etl branch. main still points at the old snapshot, so consumers querying main see the pre-load table.
4. The count on the branch reflects the in-progress load — you can audit here (row counts, referential checks) entirely in isolation before publishing.
5. MERGE BRANCH etl INTO main advances main's pointer for orders (and any other tables changed on the branch) in one atomic commit. Consumers flip to the new snapshot together; there is no partially-published state. Dropping the branch afterward is housekeeping.

Output.

Ref	count(orders)	Notes
main (before merge)	1,000,000	consumers see pre-load table
etl	1,128,934	isolated, in-progress load
main (after merge)	1,128,934	atomic flip at merge commit
main (hash before merge)	1,000,000	prior snapshot still addressable

Rule of thumb. Do lakehouse ETL on a Nessie branch, never on main. USE REFERENCE for isolation, audit on the branch, and MERGE BRANCH … INTO main to publish atomically. The branch is free because Nessie versions metadata, not data files.

Worked example — a multi-table atomic commit

Detailed explanation. The feature that sets Nessie apart is committing several tables together. Load a fact and its dimension on one branch and merge once, so consumers never see the fact updated while the dimension lags. Walk through a two-table batch.

• Tables. warehouse.db.orders (fact) and warehouse.db.customers (dimension).
• Branch. batch-2026-08-18 off main.
• Atomicity. Both tables change on the branch; one merge publishes both.

Question. Load two related tables on a branch and publish them as a single atomic commit.

Input.

Component	Value
Fact	warehouse.db.orders
Dimension	warehouse.db.customers
Branch	batch-2026-08-18
Guarantee	consumers see both new or both old — never mixed

Code.

-- 1. One branch for the whole batch
CREATE BRANCH batch_2026_08_18 IN nessie FROM main;
USE REFERENCE batch_2026_08_18 IN nessie;

-- 2. Update BOTH tables on the branch
MERGE INTO warehouse.db.customers t
USING staging.customers_delta s ON t.id = s.id
WHEN MATCHED     THEN UPDATE SET t.name = s.name, t.tier = s.tier
WHEN NOT MATCHED THEN INSERT (id, name, tier) VALUES (s.id, s.name, s.tier);

INSERT INTO warehouse.db.orders
SELECT order_id, customer_id, total_cents, status
FROM   staging.orders_delta;

-- 3. Audit the whole batch in isolation: no orphan orders?
SELECT count(*) AS orphan_orders
FROM   warehouse.db.orders o
LEFT   JOIN warehouse.db.customers c ON c.id = o.customer_id
WHERE  c.id IS NULL;                       -- must be 0 before publishing

-- 4. Publish BOTH tables in ONE commit
MERGE BRANCH batch_2026_08_18 INTO main IN nessie;
DROP BRANCH batch_2026_08_18 IN nessie;

# The atomicity guarantee, stated as a consumer-side invariant
# A reader on `main` at ANY instant sees a consistent pair:
#   before merge:  (customers = old, orders = old)
#   after  merge:  (customers = new, orders = new)
# There is NO instant where orders reference customers that don't exist yet,
# because the merge advances both table pointers in a single Nessie commit.

Step-by-step explanation.

1. A single branch (batch_2026_08_18) scopes the entire batch. Both the dimension MERGE and the fact INSERT land on this branch and produce new snapshots for their respective tables — all invisible to main.
2. The dimension is updated first (upsert customers), then the fact (insert orders) — but ordering within the branch does not matter to consumers, because none of it is visible on main until the branch merges.
3. The audit query runs on the branch and checks the cross-table invariant (no order references a missing customer). This is the payoff of multi-table isolation: you validate relationships, not just single tables, before anything publishes.
4. MERGE BRANCH … INTO main advances the pointers for both orders and customers in one atomic Nessie commit. A consumer reading main sees either the old pair or the new pair, never a fact that references not-yet-published dimension rows.
5. This is the multi-table transaction plain Iceberg cannot give you: Iceberg commits one table per operation, so without Nessie there is always a window where the fact is ahead of the dimension. Nessie closes that window.

Output.

Reader observation on main	Possible?	Why
old customers + old orders	yes	pre-merge state
new customers + new orders	yes	post-merge state
new orders + old customers	no	both advance in one commit
old orders + new customers	no	both advance in one commit

Rule of thumb. When two or more tables must stay mutually consistent, load them on one Nessie branch and publish with a single MERGE BRANCH. The atomic multi-table commit is the reason to choose Nessie over bare Iceberg time travel.

Worked example — tags for releases and time-travel queries

Detailed explanation. Branches move; tags do not. Use an immutable tag to mark a reproducible release of the whole catalog, then query any table AT that tag (or a timestamp) for reproducibility and audit. Walk through tagging a release and time-travelling.

• Tag. release_2026_08 on main after a good batch.
• Time travel by tag. Query orders AT TAG release_2026_08.
• Time travel by timestamp. Query orders AT TIMESTAMP '2026-08-17 00:00:00'.

Question. Tag a reproducible release and run reproducible reads by tag and by timestamp.

Input.

Component	Value
Tag	release_2026_08 (immutable)
Reproducible read	AT TAG release_2026_08
Point-in-time read	AT TIMESTAMP '2026-08-17 00:00:00'
Use case	reproducible ML training / audit

Code.

-- 1. Tag the current main as an immutable, named release
CREATE TAG release_2026_08 IN nessie FROM main;

-- 2. Reproducible read by tag — the tag never moves, so this is stable forever
SELECT count(*) FROM warehouse.db.`orders@release_2026_08`;

-- (equivalent Nessie AT-syntax via session reference)
USE REFERENCE release_2026_08 IN nessie;   -- read-only; it's a tag
SELECT * FROM warehouse.db.orders LIMIT 100;

-- 3. Point-in-time read by timestamp on a branch's history
SELECT count(*)
FROM   warehouse.db.orders
FOR SYSTEM_TIME AS OF TIMESTAMP '2026-08-17 00:00:00';

-- 4. Diff two refs to see what a release changed (table-level)
--    (via the Nessie CLI)
--    nessie content list --ref release_2026_08
--    nessie diff main release_2026_08

# Pin an ML training set to a Nessie TAG for byte-identical re-training
TRAINING_TAG = "release_2026_08"
train = spark.table(f"nessie.warehouse.db.features@{TRAINING_TAG}")
# model.fit(train)  — re-running months later reads the exact same snapshot,
# because a tag is immutable: it cannot be advanced or overwritten.

Step-by-step explanation.

1. CREATE TAG release_2026_08 FROM main records an immutable ref at main's current commit. Unlike a branch, a tag cannot be moved or written to — it is a permanent name for a catalog state.
2. Reading orders@release_2026_08 resolves the table's snapshot as of that tag. Because the tag never moves, this read returns the identical result today and in a year — the property reproducible ML and audit require.
3. FOR SYSTEM_TIME AS OF TIMESTAMP … time-travels within a ref's history to a wall-clock point, resolving to whichever commit was current then. Useful for "what did this table look like before the incident" without having created a tag in advance.
4. nessie diff main release_2026_08 lists which tables (and snapshots) differ between the two refs — a catalog-level changelog you can attach to a release note or an audit record.
5. Pinning a training set to the tag makes retraining deterministic: the tag guarantees the same input snapshot every time, which is what makes an experiment reproducible months later. A branch would be wrong here because it can move; a tag is the honest choice.

Output.

Read	Resolves to	Stability
orders@release_2026_08	tagged snapshot	immutable forever
USE REFERENCE release_2026_08	same tagged state	read-only
AS OF TIMESTAMP '2026-08-17'	commit current then	fixed once history exists
nessie diff main release_2026_08	table-level delta	audit changelog

Rule of thumb. Tag every release you might need to reproduce or audit; pin ML training sets and compliance reads to tags, never branches. A branch answers "latest on this line"; a tag answers "exactly this state, forever" — and reproducibility needs the second one.

Senior interview question on Nessie

A senior interviewer might ask: "You run an Iceberg lakehouse on S3 queried by Spark and Trino. Nightly jobs load a fact and three dimensions that must stay mutually consistent, the ML team needs reproducible training snapshots, and audit needs point-in-time reads. Design the Nessie setup — branching for ETL isolation, the multi-table atomic commit, tags for reproducibility, how Spark and Trino share refs, and how you keep catalog history and orphaned data files bounded."

Solution Using branch-per-batch, multi-table merge, release tags, and Iceberg-plus-Nessie GC

-- 1. ETL isolation: one branch per nightly batch, all four tables on it
CREATE BRANCH nightly_2026_08_18 IN nessie FROM main;
USE REFERENCE nightly_2026_08_18 IN nessie;

-- load dimensions then fact (order within the branch is invisible to main)
MERGE INTO warehouse.db.dim_customer t USING staging.customer_delta s ON t.id=s.id
  WHEN MATCHED THEN UPDATE SET t.name=s.name, t.tier=s.tier
  WHEN NOT MATCHED THEN INSERT (id,name,tier) VALUES (s.id,s.name,s.tier);
MERGE INTO warehouse.db.dim_product  t USING staging.product_delta  s ON t.id=s.id
  WHEN MATCHED THEN UPDATE SET t.name=s.name WHEN NOT MATCHED THEN INSERT (id,name) VALUES (s.id,s.name);
MERGE INTO warehouse.db.dim_date     t USING staging.date_delta     s ON t.d=s.d
  WHEN NOT MATCHED THEN INSERT (d, dow, month) VALUES (s.d, s.dow, s.month);
INSERT INTO warehouse.db.fact_orders
  SELECT order_id, customer_id, product_id, order_date, total_cents FROM staging.orders_delta;

-- 2. Audit the cross-table invariants ON THE BRANCH before publishing
SELECT
  (SELECT count(*) FROM warehouse.db.fact_orders f
     LEFT JOIN warehouse.db.dim_customer c ON c.id = f.customer_id WHERE c.id IS NULL) AS orphan_customer,
  (SELECT count(*) FROM warehouse.db.fact_orders f
     LEFT JOIN warehouse.db.dim_product  p ON p.id = f.product_id  WHERE p.id IS NULL) AS orphan_product;
-- both must be 0

-- 3. Publish all four tables atomically, then tag the release
MERGE BRANCH nightly_2026_08_18 INTO main IN nessie;
CREATE TAG   release_2026_08_18 IN nessie FROM main;   -- immutable, reproducible
DROP   BRANCH nightly_2026_08_18 IN nessie;

# 4. Trino shares the SAME refs (points at the same Nessie catalog)
#    catalog/nessie.properties:
#      connector.name=iceberg
#      iceberg.catalog.type=nessie
#      iceberg.nessie-catalog.uri=http://nessie:19120/api/v2
#      iceberg.nessie-catalog.ref=main
#    Trino: SELECT * FROM nessie."warehouse.db"."fact_orders@release_2026_08_18";

# 5. Retention / GC — two layers must both be reaped:
#    (a) Nessie: expire unmerged/old refs beyond the cutoff
nessie gc --cutoff main=P90D --cutoff-default P30D --delete
#    (b) Iceberg: expire snapshots + remove orphan data files no live ref points to
#        CALL nessie.system.remove_orphan_files(table => 'warehouse.db.fact_orders');
#        (run AFTER Nessie GC so live refs are known)

Step-by-step trace.

Concern	Mechanism	Result
ETL isolation	one branch per nightly batch	four tables staged privately
Cross-table consistency	multi-table MERGE BRANCH	fact + 3 dims flip atomically
Reproducibility	immutable release tag	ML/audit read a fixed state
Engine sharing	Trino + Spark on one Nessie catalog	same branches and tags
History growth	nessie gc on refs	old/unmerged commits expired
Orphan data files	Iceberg orphan-file removal	S3 not filled by dead snapshots

After deployment, each nightly batch loads all four tables on one branch, is audited for referential consistency in isolation, then publishes in a single atomic commit so Spark and Trino consumers never see a fact without its dimensions; a release tag makes the day's state reproducible for ML and audit; and a two-layer GC (Nessie refs plus Iceberg orphan files) keeps both catalog history and object storage bounded.

Output:

Metric	Value
Multi-table publish	atomic (one Nessie commit)
Cross-table window	none (both/neither)
Reproducible reads	by immutable tag or timestamp
Engines sharing refs	Spark + Trino (+ Flink/Dremio)
Catalog retention	90d on main, 30d default
Storage cleanup	Nessie GC + Iceberg orphan removal

Why this works — concept by concept:

• Branch-per-batch on the catalog — the whole batch is staged on one branch, so all four tables are isolated together and can be validated for relationships, not just individually, before anything publishes.
• Multi-table atomic merge — MERGE BRANCH … INTO main advances every changed table's pointer in one commit, closing the fact-ahead-of-dimension window that bare Iceberg leaves open.
• Immutable release tags — a tag pins the exact catalog state for reproducible ML training and point-in-time audit; because it cannot move, re-reads are deterministic.
• Engine-agnostic refs — Nessie is the shared catalog, so Spark and Trino resolve the same branches and tags; a branch created in one engine is instantly visible in the other.
• Cost — versioned catalog metadata plus two GC layers (Nessie ref expiry and Iceberg orphan-file removal). The data files are immutable Iceberg objects shared across refs, so branching is O(metadata); the only real storage cost is un-expired snapshots, which GC reaps. Compared with per-table Iceberg operations, Nessie buys multi-table atomicity for the price of running a catalog service.

ETL
Topic — etl
ETL problems on lakehouse ingestion

Practice →

ETL Topic — data-transformation Data-transformation problems on multi-table batches

Practice →

---

4. Dolt — the versioned SQL database

Dolt is Git plus MySQL — version rows and cells, dolt_commit, dolt_diff, branch, and three-way merge inside the database

The mental model in one line: Dolt is a MySQL-compatible SQL database with Git built into the storage engine — every table is versioned at the row and cell level, so you dolt_commit changes, dolt_diff two versions to see exactly which cells moved, branch to work in isolation, and dolt_merge with genuine three-way conflict resolution — all through SQL and a Git-style CLI, making it the right tool for relational data you edit and want to audit like source code. Where lakeFS versions files and Nessie versions table metadata, Dolt versions the data itself, down to the cell, and exposes history as queryable system tables.

The four axes for Dolt.

• Granularity. A commit captures row- and cell-level changes across tables. dolt_diff shows added, deleted, and modified rows with per-cell before/after values — the finest granularity of the three engines, and the only one with a true diff of the data.
• Isolation. A branch is a copy-on-write fork of the working set. You branch, edit rows, and the parent branch is untouched until merge. Because Dolt is a full database, the branch is queryable with ordinary SQL.
• Commit / merge semantics. Merge is a real three-way merge of rows against a common ancestor; non-overlapping changes merge cleanly, and genuine conflicts (both branches changed the same cell differently) surface in dolt_conflicts_<table> for you to resolve — exactly like a Git merge conflict.
• Time travel / rollback. Query AS OF a commit, branch, or timestamp; dolt_reset / dolt_revert to undo; dolt_history_<table> and dolt_blame_<table> expose full per-row history and attribution.

The anatomy — commit, branch, system tables.

• Commit. An immutable snapshot of all tables with a parent, author, and message — created via CALL DOLT_COMMIT(...) or dolt commit. Hash-addressed like a Git commit.
• Branch. A movable ref; CALL DOLT_BRANCH('fix') / CALL DOLT_CHECKOUT('fix'). Working changes stage like Git (dolt_status, DOLT_ADD).
• Diff system tables. dolt_diff_<table> (row changes between refs), dolt_history_<table> (every version of every row), dolt_commit_diff_<table> (diff between two arbitrary commits), dolt_blame_<table> (who last changed each row).
• Conflict system table. dolt_conflicts_<table> — populated on a merge conflict with base/ours/theirs values per cell.

Why cell-level versioning is different.

• A file diff (lakeFS) tells you an object changed but not which rows.
• An Iceberg snapshot (Nessie) tells you the table changed but not which cells.
• A Dolt diff tells you row 42's price went from 1500 to 1400 in commit abc123 by user etl — the same fidelity Git gives source code, applied to data.

Common interview probes on Dolt.

• "What granularity does Dolt version at?" — required answer: row and cell level, with a true data diff.
• "How do you see what changed between two versions?" — dolt_diff_<table> or SELECT … FROM dolt_commit_diff_<table>.
• "What happens on a merge conflict?" — conflicts land in dolt_conflicts_<table>; you resolve per cell, then commit.
• "How is Dolt queried by other tools?" — it speaks the MySQL wire protocol, so MySQL clients and connectors work.

Worked example — commit a change and diff two versions

Detailed explanation. The canonical Dolt flow: make a data change, commit it with a message, then diff the two commits to see exactly which cells moved. Everything is SQL (with CALL DOLT_* procedures) so it runs from any MySQL client. Walk through editing a price and diffing.

• Change. Update a product price and insert a new product.
• Commit. CALL DOLT_COMMIT('-am', 'message').
• Diff. SELECT … FROM dolt_diff_products.

Question. Commit a data edit and produce a cell-level diff of what changed.

Input.

Parameter	Value
Database	refdata (Dolt, MySQL protocol)
Table	products (id PK, name, price_cents)
Change	update price of id=7; insert id=99
Diff	dolt_diff_products between HEAD~1 and HEAD

Code.

-- 0. Starting point is committed as HEAD~1 (previous commit)

-- 1. Make data changes (ordinary SQL)
UPDATE products SET price_cents = 1400 WHERE id = 7;      -- was 1500
INSERT INTO products (id, name, price_cents) VALUES (99, 'Gadget', 999);

-- 2. Commit them with a message (stages + commits in one call)
CALL DOLT_COMMIT('-a', '-m', 'lower id=7 price; add Gadget id=99');

-- 3. Diff the two most recent commits at the cell level
SELECT from_id, to_id,
       from_name, to_name,
       from_price_cents, to_price_cents,
       diff_type
FROM   dolt_diff_products
WHERE  to_commit = HASHOF('HEAD')
  AND  from_commit = HASHOF('HEAD~1');

# Equivalent from the Dolt CLI (Git-style)
dolt sql -q "UPDATE products SET price_cents=1400 WHERE id=7"
dolt sql -q "INSERT INTO products VALUES (99,'Gadget',999)"
dolt commit -am "lower id=7 price; add Gadget id=99"
dolt diff HEAD~1 HEAD products      # human-readable +/- diff

Step-by-step explanation.

1. The data change is ordinary SQL — UPDATE and INSERT. Dolt records these against the current branch's working set exactly as MySQL would, but the storage engine keeps them versioned underneath.
2. CALL DOLT_COMMIT('-a', '-m', …) stages all changes (-a) and creates an immutable commit with a message. This is the Git commit -am equivalent expressed as a stored procedure so it works over the MySQL protocol.
3. dolt_diff_products is an auto-generated system table exposing row changes between any two commits. Selecting from_* / to_* columns shows the before/after of each cell; diff_type is added, removed, or modified.
4. HASHOF('HEAD') and HASHOF('HEAD~1') resolve refs to commit hashes so the diff is scoped to exactly the last commit. You see id=7's price_cents move 1500 → 1400 (modified) and id=99 appear (added) — nothing else.
5. The CLI path produces the same result as a human-readable colored diff. Either way, the point is fidelity: you know precisely which cells changed, in which commit, which is impossible with file- or snapshot-level versioning.

Output.

from_id	to_id	from_price_cents	to_price_cents	diff_type
7	7	1500	1400	modified
(null)	99	(null)	999	added

Rule of thumb. Commit data edits with a message the way you commit code, and reach for dolt_diff_<table> (or dolt diff) to review exactly which cells changed before you promote. Cell-level diff is Dolt's superpower — use it as your review tool.

Worked example — branch, fix, and three-way merge

Detailed explanation. Dolt's branches are full database forks and its merges are real three-way merges. Branch off main, fix data in isolation, and merge back; non-conflicting changes merge cleanly. Walk through a data-fix branch.

• Branch. fix-tiers off main.
• Fix. Correct several customers' tiers on the branch.
• Merge. CALL DOLT_MERGE('fix-tiers') back into main.

Question. Make an isolated data fix on a branch and merge it into main.

Input.

Component	Value
Base branch	main
Work branch	fix-tiers
Change	set tier='gold' for 3 customers
Merge	three-way against common ancestor

Code.

-- 1. Branch and switch (session now on fix-tiers; main untouched)
CALL DOLT_BRANCH('fix-tiers');
CALL DOLT_CHECKOUT('fix-tiers');

-- 2. Make the fix on the branch and commit it
UPDATE customers SET tier = 'gold' WHERE id IN (14, 22, 37);
CALL DOLT_COMMIT('-a', '-m', 'correct tier for VIP customers 14,22,37');

-- 3. Back to main; meanwhile main may have advanced independently
CALL DOLT_CHECKOUT('main');

-- 4. Three-way merge: non-overlapping row changes merge cleanly
CALL DOLT_MERGE('fix-tiers');

-- 5. Verify no conflicts remained
SELECT * FROM dolt_conflicts;            -- empty => clean merge
CALL DOLT_COMMIT('-a', '-m', 'merge fix-tiers into main');

-- If a conflict HAD occurred (both branches changed customer 14's tier):
-- dolt_conflicts_customers exposes base/ours/theirs per cell.
SELECT base_tier, our_tier, their_tier, our_id
FROM   dolt_conflicts_customers;
-- Resolve by choosing a value, then clear the conflict:
UPDATE customers SET tier = 'gold' WHERE id = 14;      -- pick the winner
CALL DOLT_CONFLICTS_RESOLVE('--ours', 'customers');    -- or --theirs / manual
CALL DOLT_COMMIT('-a', '-m', 'resolve tier conflict for id=14');

Step-by-step explanation.

1. DOLT_BRANCH + DOLT_CHECKOUT fork the database and switch the session to fix-tiers. This is copy-on-write: main is untouched and fully queryable by other sessions while you work.
2. The fix is plain SQL committed on the branch. Because it lives on fix-tiers, nothing on main sees it yet — the same isolation property as lakeFS/Nessie but at row granularity inside a real database.
3. Switching back to main models the realistic case where main advanced (other loads) while your fix was in progress. Dolt's merge is three-way, so it compares both branches against their common ancestor, not against each other blindly.
4. DOLT_MERGE('fix-tiers') merges the row changes. Rows only you touched apply cleanly; rows only main touched are untouched; the merge is conflict-free unless the same cell diverged on both sides.
5. The conflict path (shown second) is Dolt's differentiator: dolt_conflicts_customers gives base/ours/theirs per cell, you pick a resolution with SQL, and DOLT_CONFLICTS_RESOLVE clears it — exactly the Git conflict workflow, applied to data.

Output.

Scenario	dolt_conflicts	Result
fix touches rows main didn't	empty	clean three-way merge
both changed different cells	empty	clean; both changes kept
both changed customer 14's tier	1 row	resolve, then commit
main deleted a row you edited	1 row	resolve delete-vs-edit

Rule of thumb. Do data fixes on a branch and merge with DOLT_MERGE; trust the three-way merge for non-overlapping edits and use dolt_conflicts_<table> to resolve genuine cell conflicts. Dolt is the only one of the three engines where "merge the data" means a real row-level three-way merge.

Worked example — cell-level history and blame

Detailed explanation. Because Dolt versions every cell, it can answer "how did this value get here?" dolt_history_<table> returns every version of a row across commits, and dolt_blame_<table> attributes each row's current value to the commit and author that last set it. Walk through auditing one customer's tier changes.

• History. Every version of customer 14 across commits.
• Blame. Who last set each customer's tier.
• Point-in-time. The table AS OF a past commit.

Question. Trace the full change history of one row and attribute its current value.

Input.

Component	Value
Table	customers
Row	id = 14
History	dolt_history_customers
Blame	dolt_blame_customers

Code.

-- 1. Full history of one row: every version across every commit
SELECT commit_hash, committer, commit_date, tier
FROM   dolt_history_customers
WHERE  id = 14
ORDER  BY commit_date;

-- 2. Blame: who last set each row's current value
SELECT commit_hash, committer, message, tier
FROM   dolt_blame_customers
WHERE  id = 14;

-- 3. Time travel: the whole table AS OF a past commit or timestamp
SELECT * FROM customers AS OF 'HEAD~5';
SELECT * FROM customers AS OF TIMESTAMP '2026-08-10 00:00:00';

-- 4. Diff a specific pair of commits (arbitrary, not just adjacent)
SELECT from_tier, to_tier, diff_type
FROM   dolt_commit_diff_customers
WHERE  from_commit = 'abc123' AND to_commit = 'def456' AND to_id = 14;

Step-by-step explanation.

1. dolt_history_customers returns one row per (row-version, commit): every state customer 14 ever had, with the committing author and date. Ordering by commit_date reconstructs the audit timeline — bronze → silver → gold with exactly when and in which commit.
2. dolt_blame_customers is the git-blame analogue: for each current row it names the commit, author, and message that last set it. This answers "who made this the value it is now" without scanning history manually.
3. AS OF 'HEAD~5' (or a timestamp) reads the entire table as it was at that point — reproducible historical reads without pre-planned snapshots, because every commit is a full state.
4. dolt_commit_diff_customers diffs any two commits (not just adjacent ones), so you can compare a release six months apart and see the per-cell delta. This is the forensic tool for "what changed between these two known-good points."
5. Together these system tables make data auditable like code: full history, blame, point-in-time reads, and arbitrary diffs — all in SQL, which is why Dolt suits regulated reference data and human-curated datasets.

Output.

commit_hash	committer	tier	note
9a1…	seed	bronze	initial load
c4b…	crm-sync	silver	tier upgrade
def…	etl	gold	VIP correction (current)

Rule of thumb. Use dolt_history_<table> for the timeline, dolt_blame_<table> for attribution, and AS OF for point-in-time reads. When an auditor asks "how did this value get here," Dolt answers in one SQL query — the payoff of versioning at the cell.

Senior interview question on Dolt

A senior interviewer might ask: "You maintain a hand-curated reference dataset — currency codes, tax rules, tiering thresholds — edited by analysts and consumed by pipelines. You need every edit reviewable, isolated pull-request-style changes, safe merges, full audit history, and reproducible reads for compliance. Design the Dolt workflow — branching for proposed changes, review via diff, merge with conflict handling, blame/history for audit, and how pipelines consume it."

Solution Using proposal branches, diff-based review, gated merge, and MySQL-protocol serving

-- 1. Analyst proposes a change on a branch (pull-request style)
CALL DOLT_BRANCH('proposal-tax-2026q3');
CALL DOLT_CHECKOUT('proposal-tax-2026q3');

UPDATE tax_rules SET rate = 0.088 WHERE region = 'CA' AND effective_from = '2026-07-01';
INSERT INTO tax_rules (region, rate, effective_from) VALUES ('WA', 0.065, '2026-07-01');
CALL DOLT_COMMIT('-a', '-m', 'CA rate 8.8%, add WA rule (ticket TAX-482)');

-- 2. Reviewer sees EXACTLY what changed (cell-level), like a code review
SELECT region, from_rate, to_rate, from_effective_from, to_effective_from, diff_type
FROM   dolt_diff_tax_rules
WHERE  to_commit = HASHOF('proposal-tax-2026q3')
  AND  from_commit = HASHOF('main');

-- 3. Gated merge into main (only after review approves the diff)
CALL DOLT_CHECKOUT('main');
CALL DOLT_MERGE('proposal-tax-2026q3');
SELECT count(*) FROM dolt_conflicts;                 -- must be 0
CALL DOLT_COMMIT('-a', '-m', 'merge proposal-tax-2026q3 (approved by reviewer)');

-- 4. Tag the approved state for compliance reproducibility
CALL DOLT_TAG('refdata-2026-08-18', 'main');

# 5. Pipelines consume Dolt over the MySQL wire protocol (unchanged clients)
import pymysql
conn = pymysql.connect(host="dolt-sql-server", port=3306, db="refdata", user="pipeline")
with conn.cursor() as cur:
    # Reproducible compliance read: pin to the approved tag
    cur.execute("SELECT region, rate FROM `tax_rules` AS OF 'refdata-2026-08-18'")
    rates = dict(cur.fetchall())
# The tag is immutable, so a re-run months later reads identical rules.

-- 6. Audit any value on demand: who set it, when, and the full timeline
SELECT commit_hash, committer, commit_date, rate
FROM   dolt_history_tax_rules
WHERE  region = 'CA'
ORDER  BY commit_date;                                -- full change trail for CA

Step-by-step trace.

Layer	Component	Purpose
Proposal	branch + commit	isolated, reviewable change set
Review	dolt_diff_tax_rules	cell-level diff = the "pull request"
Merge	DOLT_MERGE + conflict check	safe integration into main
Reproducibility	DOLT_TAG	immutable compliance snapshot
Serving	MySQL protocol	pipelines read Dolt as a MySQL DB
Audit	dolt_history_tax_rules	full timeline + attribution per value

After deployment, every analyst edit is a branch whose diff is reviewed like a code change; approved proposals merge into main with a conflict check; an immutable tag pins each approved state so compliance reads are reproducible; pipelines consume the data over the ordinary MySQL protocol; and any value's provenance is one dolt_history query away. The reference dataset now has the same review, merge, and audit discipline as the codebase.

Output:

Metric	Value
Change granularity	row / cell (true data diff)
Review artifact	dolt_diff (per-cell before/after)
Merge safety	three-way + dolt_conflicts
Reproducible reads	immutable tag via AS OF
Consumption	MySQL wire protocol
Audit	full history + blame per value

Why this works — concept by concept:

• Proposal branches — each analyst change is an isolated branch, so main reference data is never edited in place; the branch is the reviewable, revertible unit, exactly like a pull request.
• Cell-level diff review — dolt_diff_<table> shows the per-cell before/after, so a reviewer approves a precise change, not an opaque file or snapshot. This is the review fidelity only row-level versioning provides.
• Three-way merge with conflicts — DOLT_MERGE integrates non-overlapping edits cleanly and surfaces genuine cell conflicts in dolt_conflicts_<table> for explicit resolution, so bad merges cannot silently overwrite.
• Immutable tags + MySQL serving — a tag pins an approved state for reproducible compliance reads, while the MySQL wire protocol lets existing pipelines consume Dolt with no client changes.
• Cost — Dolt stores versioned row history, so storage grows with edit volume and it targets modest-to-mid-scale relational data rather than petabyte lakes. In exchange you get true data diff, three-way merge, blame, and point-in-time reads in SQL — capabilities no file- or snapshot-level system can match. Run dolt gc to compact unreferenced history; keep tags for the states you must reproduce.

SQL
Topic — sql
SQL problems on versioned tables and diffs

Practice →

Database Topic — database Database problems on history and audit tables

Practice →

---

5. Choosing and operating data version control

Pick by granularity, operate with write-audit-publish, and reap with garbage collection — the same discipline across all three engines

The mental model in one line: choosing a data-version-control engine is a granularity decision (files → lakeFS, table metadata → Nessie, rows → Dolt) and operating one is a discipline decision — every engine implements the same write-audit-publish loop (branch → validate → merge), the same rollback primitive (revert/reset to a prior commit), and the same obligation to garbage-collect unmerged branches and unreachable data so versioned storage stays bounded. The tool differs; the operating model does not, which is why senior interviews test whether you can run WAP and retention on any of them.

The decision axes, side by side.

• Versioning layer. lakeFS = objects/files (any format); Nessie = Iceberg/Delta table metadata; Dolt = rows and cells in SQL tables. This is the primary discriminator.
• Diff fidelity. lakeFS = which objects changed; Nessie = which table snapshots changed; Dolt = which cells changed. Finer fidelity, narrower scope.
• Engine fit. lakeFS speaks the S3 API (Spark/Trino/ML unchanged); Nessie is a shared catalog (Spark/Flink/Trino/Dremio); Dolt speaks MySQL (relational clients).
• Scale. lakeFS and Nessie scale to the object store (petabytes); Dolt targets modest-to-mid relational data where per-cell history is worth its storage.

The operating loop — identical across engines.

• Write. Create a branch; run the pipeline / edit against the branch only.
• Audit. Validate the branch in isolation — row counts, null/PK checks, referential and cross-table invariants, schema contracts. Gate the merge on it.
• Publish. Merge the branch to main in one atomic commit; consumers flip together.
• Rollback. If a bad change lands, revert/reset main to the prior commit — seconds, not a restore.

Retention — the cost you must manage.

• Unmerged branches. Delete stale work branches; a failed batch's branch should not live forever.
• Old commits. Expire commit history beyond a retention window (keep main longer than feature branches).
• Unreachable data. lakeFS GC removes objects no live commit references; Iceberg+Nessie removes orphan data files after snapshot expiry; Dolt gc compacts unreferenced history. Always run the object-level cleanup after the ref-level expiry so live refs are known.
• Tags are forever (by choice). Anything you may need to reproduce or audit — release tags, training snapshots — is exempt from GC. Retention reaps the churn, not the record.

Common interview probes on choosing and operating.

• "How do you choose between the three?" — required answer: granularity — files/tables/rows.
• "What's the common operating pattern?" — write-audit-publish with a gated merge.
• "How do you stop versioned storage from growing forever?" — GC unmerged branches + expire old commits + remove unreachable data; keep tags.
• "Can you use more than one?" — yes; lakeFS for the raw file zone, Nessie for curated Iceberg, Dolt for reference data — one WAP discipline across all.

Worked example — the choose-and-operate matrix

Detailed explanation. The senior artifact for this section is a single matrix that maps each engine to its layer, diff fidelity, WAP flavor, and GC obligation, so you can defend any choice and its operations in one glance. Build it for a platform running all three.

• Raw zone. lakeFS over s3://lake/raw/.
• Curated zone. Nessie over the Iceberg warehouse.
• Reference data. Dolt for analyst-edited tables.

Question. Build the choose-and-operate matrix and state each engine's WAP and GC command surface.

Input.

Engine	Layer	Branch primitive	Merge primitive
lakeFS	objects	lakectl branch create	lakectl merge
Nessie	table metadata	CREATE BRANCH … IN nessie	MERGE BRANCH … INTO main
Dolt	rows/cells	CALL DOLT_BRANCH(...)	CALL DOLT_MERGE(...)

Code.

Choose-and-operate matrix
=========================

               lakeFS              Nessie                 Dolt
 layer         objects/files       Iceberg table meta     rows & cells
 diff          which objects       which snapshots        which cells
 branch        lakectl branch      CREATE BRANCH          DOLT_BRANCH
 audit on      branch (S3 read)    USE REFERENCE branch   branch (SQL)
 publish       lakectl merge       MERGE BRANCH           DOLT_MERGE
 rollback      branch revert       assign/reset branch    DOLT_RESET/REVERT
 time travel   @commit/branch      @ref / AS OF ts        AS OF commit/ts
 GC            lakectl gc + rules  nessie gc + orphan rm  dolt gc
 keep forever  tags                tags                   tags
 scale         object store        object store           mid-size RDBMS

Step-by-step explanation.

1. The top rows fix the choice: layer and diff fidelity. If the question is "which engine," you read these two rows — files/objects → lakeFS, table snapshots → Nessie, cells → Dolt. Everything else is operations.
2. The middle rows fix the WAP surface: branch, audit-on, publish. They are structurally identical — create a branch, validate on it, merge to main — differing only in syntax. That sameness is the point: WAP is engine-independent.
3. The rollback and time-travel rows show every engine has both a "revert/reset to prior commit" undo and a "read the past by ref/timestamp" read. If an engine lacked either, it would not be a real version-control system.
4. The GC row is the operating obligation. Each engine has a ref-level expiry and (for the object-backed ones) a separate data-file cleanup; skipping GC is how versioned storage silently balloons.
5. The "keep forever" row is the exception to GC: tags are the durable record and are never reaped. This matrix is the whiteboard answer to "compare and operate these three" — memorise it.

Output.

Question	Answer via matrix
Which engine for a file lake?	lakeFS (objects row)
Which for multi-table Iceberg?	Nessie (table-meta row)
Which for per-cell audit?	Dolt (rows/cells row)
Common operating loop?	branch → audit → merge (WAP)
How to bound storage?	GC row + keep tags

Rule of thumb. Keep this matrix in your head: the top two rows answer "which engine," the middle rows prove WAP is the same everywhere, and the bottom rows are the retention discipline. An interviewer who asks you to "compare and operate lakeFS, Nessie, and Dolt" is asking you to reproduce it.

Worked example — a write-audit-publish CI gate for data

Detailed explanation. The most valuable operating pattern is a CI gate: a pull-request-style flow where a data change runs on a branch, an automated audit runs, and merge is blocked unless it passes — the data equivalent of "tests must pass before merge." Build it engine-agnostically, then show the Nessie instantiation.

• Trigger. A proposed data change opens a branch.
• Audit. CI runs quality checks against the branch.
• Gate. Merge to main only if the audit passes; otherwise the branch is rejected.

Question. Implement a WAP CI gate that blocks a merge to main when branch-level data-quality checks fail.

Input.

Stage	Action
open	create branch, load/edit data
audit	row counts, null PKs, referential checks, drift
gate	pass → merge; fail → reject + alert
publish	atomic merge to main

Code.

# Engine-agnostic WAP CI gate (Nessie instantiation via Spark SQL)
class DataQualityError(Exception): ...

def wap_ci_gate(spark, branch: str):
    # 1. AUDIT the branch in isolation
    spark.sql(f"USE REFERENCE {branch} IN nessie")
    checks = spark.sql("""
        SELECT
          (SELECT count(*) FROM warehouse.db.orders)                          AS rows,
          (SELECT count(*) FROM warehouse.db.orders WHERE order_id IS NULL)    AS null_pk,
          (SELECT count(*) FROM warehouse.db.orders o
             LEFT JOIN warehouse.db.customers c ON c.id=o.customer_id
             WHERE c.id IS NULL)                                              AS orphan_fk,
          (SELECT abs(count(*) - (SELECT baseline FROM meta.expected_counts
                                   WHERE tbl='orders'))
             FROM warehouse.db.orders)                                        AS drift
    """).first()

    problems = []
    if checks.rows == 0:        problems.append("empty batch")
    if checks.null_pk > 0:      problems.append(f"{checks.null_pk} null order_id")
    if checks.orphan_fk > 0:    problems.append(f"{checks.orphan_fk} orphan FKs")
    if checks.drift > 50000:    problems.append(f"row drift {checks.drift} > 50k")

    # 2. GATE
    if problems:
        alert(f"WAP gate FAILED on {branch}: {problems}")
        raise DataQualityError(problems)     # merge is NOT attempted

    # 3. PUBLISH — atomic merge only past the gate
    spark.sql(f"MERGE BRANCH {branch} INTO main IN nessie")
    spark.sql(f"DROP BRANCH {branch} IN nessie")
    print(f"published {branch} -> main")

# The same gate as a pipeline stage (Airflow / CI): fail => branch left for triage
wap_pipeline:
  - task: open_branch          # CREATE BRANCH etl_<run_id> FROM main
  - task: transform_on_branch  # write into the branch
  - task: wap_ci_gate          # audit + gate (raises on failure)
    on_failure: keep_branch_and_alert     # do NOT merge; quarantine for humans
  - task: tag_release          # optional: tag main after a clean publish

Step-by-step explanation.

1. The gate first switches to the branch and runs the audit there — row count, null primary keys, orphan foreign keys, and a drift check against an expected baseline. All of this validates the candidate data while it is still invisible to main.
2. Each failed check appends to problems. The checks encode the platform's data contract: non-empty, valid keys, referential integrity, and volume within tolerance. These four catch the overwhelming majority of real corruption.
3. The gate is a hard branch point: if problems is non-empty it raises before any merge is attempted, so bad data cannot reach main even accidentally. The branch is left intact for humans to triage — the "quarantine" behavior.
4. Only past the gate does the code MERGE BRANCH … INTO main, publishing atomically. This is write-audit-publish expressed as CI: the merge is the deploy, and the audit is the required test suite.
5. Wiring the same gate as an orchestrator stage (on_failure: keep_branch_and_alert) makes it operational: a failing batch pages a human and leaves evidence on its branch, while a passing batch publishes and optionally tags a release. The pattern is engine-agnostic — the identical shape works on lakeFS hooks and a Dolt merge check.

Output.

Branch audit result	Gate	main outcome
rows>0, 0 null, 0 orphan, drift ok	pass	atomic merge; branch dropped
empty batch	fail	no merge; branch quarantined
200 orphan FKs	fail	no merge; alert raised
drift 120k > 50k	fail	no merge; human triage

Rule of thumb. Make merge-to-main the only path to production and gate it with an automated audit — then bad data is unpublishable by construction. Write-audit-publish is CI for data: the branch is the PR, the audit is the test suite, and the merge is the deploy.

Worked example — garbage collection and retention

Detailed explanation. Versioned storage grows with every branch and commit; without GC it balloons. The retention job has two layers on the object-backed engines — expire refs/commits, then remove data no live ref references — and must always run in that order. Walk through a retention policy across the three engines.

• Policy. Keep main 90 days, feature/work branches 14 days, tags forever.
• Two-phase GC. Expire refs first, then remove unreachable data files.
• Safety. Never GC below the newest tag you must reproduce.

Question. Define a retention policy and the GC commands for lakeFS, Nessie+Iceberg, and Dolt.

Input.

Object	Retention
main history	90 days
work / feature branches	14 days
release / training tags	forever
unreachable data files	removed after ref expiry

Code.

# lakeFS — (1) delete stale work branches, (2) run GC to drop unreferenced objects
lakectl branch list lakefs://lake | awk '$1 ~ /^(ingest|etl)-/ {print $1}' \
  | while read b; do lakectl branch delete "lakefs://lake/$b" --yes; done
# GC retention rules (committed): default 14d, main 90d; then run the GC Spark job
#   { "default_retention_days": 14, "branches": [{"branch_id":"main","retention_days":90}] }

# Nessie + Iceberg — (1) expire refs/commits, THEN (2) remove orphan data files
nessie gc --cutoff main=P90D --cutoff-default P14D --delete          # phase 1: refs
# phase 2 (per table, after phase 1 so live refs are known):
#   CALL nessie.system.expire_snapshots(table => 'warehouse.db.orders', older_than => now() - INTERVAL 14 DAYS);
#   CALL nessie.system.remove_orphan_files(table => 'warehouse.db.orders');

-- Dolt — compact unreferenced history; tags/branches you keep are preserved
-- (drop merged work branches first, then gc)
CALL DOLT_BRANCH('-d', 'proposal-tax-2026q3');   -- delete merged proposal branch
CALL DOLT_GC();                                   -- reclaim unreferenced chunks
-- Tagged states (e.g. refdata-2026-08-18) remain reachable and are NOT collected.

Step-by-step explanation.

1. The policy separates churn (work branches, old commits) from record (tags). Churn is reaped on a schedule; record is kept forever. This single distinction prevents both unbounded growth and accidental loss of reproducible states.
2. On lakeFS, phase one deletes stale work branches by name pattern, and phase two runs GC with retention rules so objects no surviving commit references are physically removed. Deleting branches first is what makes their objects unreferenced.
3. On Nessie+Iceberg the order is critical: nessie gc expires refs/commits first so the set of live refs is finalized, and only then does Iceberg remove_orphan_files delete data files no live snapshot points at. Running orphan removal first could delete files a soon-to-survive ref needs.
4. On Dolt, deleting merged proposal branches and running DOLT_GC() compacts unreferenced row-history chunks. Because Dolt keeps full cell history, GC is how you stop an edit-heavy table's storage from growing without bound — while tagged states stay reachable.
5. Across all three, the invariant is the same: reap refs, then reap the data those refs no longer protect, and never below a tag you must reproduce. Retention is the operating cost of version control, and running it is a senior habit, not an afterthought.

Output.

Engine	Phase 1 (refs)	Phase 2 (data)	Protected
lakeFS	delete stale branches	GC unreferenced objects	tags
Nessie+Iceberg	nessie gc cutoffs	remove orphan files	tags
Dolt	delete merged branches	DOLT_GC()	tags
all	expire first	remove second	reproducible tags

Rule of thumb. Run GC on a schedule in two phases — expire refs, then remove the data they no longer protect — and never below a tag you must reproduce. Versioned storage is not free; retention is the discipline that keeps data version control affordable.

Senior interview question on choosing and operating

A senior interviewer might ask: "You're standing up data version control across a platform with a raw file lake, a curated Iceberg warehouse, and a hand-edited reference dataset. Justify the engine per zone, describe the single operating discipline you'd apply across all three, how you'd gate every production change, how rollback works, and how you'd keep versioned storage from growing without bound."

Solution Using per-zone engines under one write-audit-publish discipline with two-phase GC

Per-zone choice (granularity-driven)
====================================
  raw file lake        → lakeFS   (version objects of any format)
  curated Iceberg WH   → Nessie   (multi-table atomic commits, shared by Spark+Trino)
  reference dataset    → Dolt     (row/cell diff, blame, three-way merge)

One operating discipline across all three: WRITE-AUDIT-PUBLISH
  branch  →  audit on the branch  →  gated merge to main  →  (tag)  →  GC the rest

# Single WAP driver, engine-pluggable — same shape, per-engine adapters
def wap(engine, dataset, transform, audit):
    b = engine.branch(f"wap-{dataset}-{run_id()}")   # WRITE
    engine.run(b, transform)
    if not audit(engine.read(b)):                     # AUDIT
        engine.discard(b); alert(f"{dataset} failed audit"); return
    engine.merge(b, "main")                           # PUBLISH (atomic)
    engine.tag(f"{dataset}-{today()}", "main")        # reproducible record
    engine.drop(b)

# adapters implement branch/run/read/merge/discard/tag for lakeFS, Nessie, Dolt
wap(lakefs_adapter, "raw_events", ingest_events, audit_files)
wap(nessie_adapter, "curated",    load_star,     audit_referential)
wap(dolt_adapter,   "reference",  apply_edits,   audit_contract)

# Rollback (any zone): revert/reset main to the prior commit — seconds
lakectl branch revert lakefs://lake/main <bad> --parent-number 1 --yes   # lakeFS
# nessie: CALL nessie.assign_branch('main','main~1')                     # Nessie
# dolt:   CALL DOLT_RESET('--hard','HEAD~1')                             # Dolt

# Two-phase GC on a schedule, per engine (refs first, then data); tags kept forever

Step-by-step trace.

Concern	Mechanism	Result
Engine per zone	granularity decision	files→lakeFS, tables→Nessie, rows→Dolt
One discipline	WAP driver	identical branch→audit→merge everywhere
Production gate	audit before merge	bad data unpublishable
Rollback	revert/reset to prior commit	seconds, any zone
Reproducibility	tag after publish	pinned, immutable states
Storage growth	two-phase GC per engine	bounded; tags exempt

After deployment, each zone uses the engine matched to its granularity, but all three run through one write-audit-publish driver with per-engine adapters; every production change is gated by an audit on its branch; rollback in any zone is a prior-commit revert measured in seconds; releases and training sets are tagged for reproducibility; and a scheduled two-phase GC per engine keeps versioned storage proportional to real change while never touching a tag.

Output:

Metric	Value
Engines	lakeFS + Nessie + Dolt (per zone)
Operating model	one WAP discipline, pluggable adapters
Production gate	branch-level audit before merge
Rollback	prior-commit revert/reset (seconds)
Reproducibility	tags per publish
Storage	two-phase GC; tags exempt

Why this works — concept by concept:

• Granularity-driven engine choice — each zone gets the engine whose versioning layer matches what must be diffed and rolled back, so no zone is forced into too-coarse or too-fine versioning.
• One write-audit-publish discipline — a single driver with per-engine adapters proves WAP is engine-independent: branch, audit, gated merge, tag. Operators learn one model, not three.
• Gated merge as the production boundary — the audit is a hard precondition of merge, so bad data cannot be published regardless of zone; the merge is the only path to main.
• Prior-commit rollback — every engine's undo is a pointer move to an immutable prior commit, so recovery is seconds in any zone and the incident stays auditable.
• Cost — three services to operate plus a scheduled two-phase GC per engine, against the payoff of isolated pipelines, atomic publishes, seconds-long rollback, and reproducible tags across the whole platform. The dominant recurring cost is retention, which GC bounds; tags — the record you must keep — are deliberately exempt. Net: O(real change) storage under one discipline, instead of O(dataset) backups and hours-long restores per zone.

Data
Topic — data-validation
Data-validation problems on audit gates

Practice →

Design
Topic — design
Design problems on multi-engine data platforms

Practice →

---

Cheat sheet — data version control recipes

• Which engine when. Version files/objects of any format → lakeFS (sits in front of S3/GCS/Azure, speaks the S3 API). Version an Iceberg/Delta lakehouse where many tables must commit and roll back together → Nessie (git-like catalog, engine-agnostic across Spark/Flink/Trino/Dremio). Version rows and cells in relational data you edit and audit → Dolt (MySQL-compatible database with Git built in). Choose by the granularity of the thing you must diff and roll back.
• lakeFS branch/commit/merge template. lakectl branch create lakefs://repo/work --source lakefs://repo/main; write via the S3 gateway (s3a://repo/work/path); lakectl commit lakefs://repo/work -m "msg" --meta key=val; lakectl merge lakefs://repo/work lakefs://repo/main. Rollback: lakectl branch revert lakefs://repo/main <hash> --parent-number 1. Time travel: read lakefs://repo/main@<hash>/path. Zero-copy because objects are content-addressed and copy-on-write.
• Nessie multi-table commit + tag template. CREATE BRANCH etl IN nessie FROM main; USE REFERENCE etl IN nessie; write to several tables; audit cross-table invariants on the branch; MERGE BRANCH etl INTO main IN nessie publishes all changed tables in one atomic commit; CREATE TAG release_x IN nessie FROM main for an immutable reproducible state. Time travel: orders@release_x or FOR SYSTEM_TIME AS OF TIMESTAMP '…'.
• Dolt commit/diff/merge template. CALL DOLT_BRANCH('fix'); CALL DOLT_CHECKOUT('fix'); edit rows with SQL; CALL DOLT_COMMIT('-a','-m','msg'); review with SELECT … FROM dolt_diff_<table> WHERE to_commit=HASHOF('fix') AND from_commit=HASHOF('main'); CALL DOLT_MERGE('fix') (three-way; conflicts land in dolt_conflicts_<table>). Audit with dolt_history_<table> + dolt_blame_<table>; time travel with … AS OF 'HEAD~5'.
• Write-audit-publish gate template. Branch → write on the branch → audit the branch (rows>0, no null PKs, no orphan FKs, drift within tolerance, schema contract) → gate: pass = atomic merge to main; fail = discard/quarantine branch + alert. Enforce the gate at the platform layer — a lakeFS pre-merge hook, a CI stage before MERGE BRANCH, or a Dolt merge check — so merge-to-main is the only path to production.
• Rollback primitive (all engines). Undo is a move to a prior immutable commit, not a restore: lakeFS branch revert, Nessie assign_branch('main','main~1'), Dolt DOLT_RESET('--hard','HEAD~1') / DOLT_REVERT. Seconds, and history is preserved so you can still diff what went wrong.
• Time travel (all engines). lakeFS: @<commit|branch>. Nessie: table@<ref> or FOR SYSTEM_TIME AS OF. Dolt: AS OF <commit|timestamp>. Pin reproducible reads (ML training sets, compliance) to immutable tags, never to branches — a branch moves, a tag does not.
• Garbage collection — two phases, in order. Phase 1: expire refs/commits (lakeFS retention rules, nessie gc --cutoff, delete merged Dolt branches). Phase 2: remove unreachable data (lakeFS GC job, Iceberg remove_orphan_files, DOLT_GC()). Always refs-first so live refs are known. Keep main longer than feature branches; keep tags forever.
• Zero-copy explained. A branch copies pointers, not bytes. lakeFS shares content-addressed objects (copy-on-write on write); Nessie shares immutable Iceberg data files across refs and only forks metadata; Dolt uses structural sharing of unchanged rows. Isolating a pipeline therefore costs metadata, which is why per-batch branching is affordable.
• Merge conflict handling. lakeFS/Nessie detect conflicts at the object/table level (both refs changed the same object/table) — resolve by choosing a ref or re-running. Dolt does a true three-way row merge and surfaces cell-level conflicts in dolt_conflicts_<table> (base/ours/theirs) that you resolve in SQL, exactly like a Git conflict.
• Multi-engine platform pattern. Run lakeFS over the raw file zone, Nessie over the curated Iceberg warehouse, and Dolt for reference data — different layers, one write-audit-publish discipline. A single WAP driver with per-engine adapters (branch/run/read/merge/discard/tag) keeps operations uniform across all three.
• Migration cost between models. Adding version control to an existing S3 lake with lakeFS: point tools at the gateway, ~1 sprint. Adopting Nessie on an existing Iceberg warehouse: swap the catalog, migrate table references, ~1–2 sprints. Moving reference data into Dolt: import + wire the MySQL connector, ~days per dataset. Choose the layer once; re-layering later is a real migration.

Frequently asked questions

What is data version control in one sentence?

Data version control is the practice of applying Git-like semantics — branch, commit, diff, merge, revert, and time travel — to data instead of source code, so a pipeline can work on an isolated copy, a change can be reviewed and validated before it publishes, and a bad load can be rolled back to a prior immutable commit in seconds rather than restored from a backup. The three canonical engines version different layers: lakeFS versions the raw objects/files in an object store, Nessie versions Iceberg/Delta table metadata in a catalog, and Dolt versions rows and cells inside a SQL database. Every senior data-engineering interview probes it because it is the load-bearing correctness and reproducibility pattern for the modern lakehouse and ML stack.

lakeFS vs Nessie vs Dolt — when do I pick each?

Pick by the granularity of what you must diff and roll back. Choose lakeFS when your data is files in an object store (Parquet lakes, ML feature files, images, models) and you want to version everything in the bucket regardless of format — it sits in front of S3/GCS/Azure and speaks the S3 API, so Spark, Trino, and ML tools work unchanged. Choose Nessie when your data is an Iceberg or Delta lakehouse and you want git-like branches over the catalog so many tables commit and roll back together atomically — it is engine-agnostic, shared by Spark, Flink, Trino, and Dremio. Choose Dolt when your data is relational and modest-to-mid scale and you need true row-and-cell versioning with diff, blame, and three-way merge — it is a MySQL-compatible database with Git built in, ideal for reference data and human-curated tables. Many platforms run all three, each over the zone it fits, under one write-audit-publish discipline.

What is the write-audit-publish pattern?

Write-audit-publish (WAP) is the operating discipline that makes data version control valuable: write the pipeline's output to an isolated branch (not to production), audit that branch in isolation with data-quality checks (row counts, null/PK checks, referential and cross-table invariants, drift and schema contracts), and publish by merging the branch into main in a single atomic commit only if the audit passes — otherwise the branch is discarded or quarantined and production never saw the data. It is the data equivalent of "tests must pass before merge": the branch is the pull request, the audit is the test suite, and the merge is the deploy. All three engines implement WAP the same way — a lakeFS pre-merge hook, a CI stage before a Nessie MERGE BRANCH, or a Dolt merge check — and enforcing the gate at the platform layer makes bad data unpublishable by construction.

How does time travel work across these tools?

Time travel means reading data as it existed at a past point, and every engine supports it because every commit is a complete, immutable, addressable state. In lakeFS you read the repository @ any commit or branch (lakefs://repo/main@<hash>/path). In Nessie you query a table AT a branch, tag, or timestamp (orders@release_v1 or FOR SYSTEM_TIME AS OF TIMESTAMP '…'), resolving to the snapshot current then. In Dolt you query AS OF a commit, branch, or timestamp (SELECT * FROM customers AS OF 'HEAD~5'), and you additionally get dolt_history_<table> for every version of every row. For reproducibility — pinning an ML training set or a compliance read — always target an immutable tag, never a branch: a branch moves as new commits land, but a tag is a permanent name for one exact state, which is what makes a re-read deterministic.

Do zero-copy branches really not duplicate the data?

Yes — a branch copies pointers, not bytes, which is why creating one is effectively free and why per-batch branching is affordable. In lakeFS, objects are content-addressed and stored once; a branch initially references the exact same physical objects as main, and the two only diverge as new writes create new objects (copy-on-write) while unchanged objects stay shared. In Nessie, the branch forks only the catalog metadata — the immutable Iceberg data files are shared across every ref, so a branch adds metadata, not data files. In Dolt, unchanged rows are structurally shared between commits, so a branch stores only the delta. The practical consequence: isolating a pipeline on its own branch costs metadata proportional to what actually changes, not a full copy of the dataset — the property that makes write-audit-publish cheap enough to run on every batch.

How do merge conflicts work for data?

It depends on the versioning layer. lakeFS and Nessie detect conflicts at the object and table level respectively — if two branches both produced a new version of the same object (lakeFS) or the same table (Nessie), the merge flags a conflict you resolve by choosing a side or re-running the branch against the latest main. There is no row-level merge because those engines version files and metadata, not row data. Dolt performs a genuine three-way merge of rows against the common ancestor: non-overlapping edits (different rows, or different cells of the same row) merge cleanly and automatically, while a true conflict — both branches changed the same cell to different values, or one edited a row the other deleted — is surfaced in dolt_conflicts_<table> with base/ours/theirs values per cell, which you resolve in SQL and then commit, exactly like resolving a Git merge conflict in code. That cell-level three-way merge is unique to Dolt among the three.

Practice on PipeCode

• Drill the ETL practice library → for the branch-based ingestion, write-audit-publish, and incremental-load patterns that data version control is built to support.
• Rehearse on the SQL practice library → for the versioned-table, diff, history, and audit-query problems senior interviewers love.
• Sharpen the modelling and platform axis with the design practice library → for lakehouse, isolation, and rollback system-design scenarios.
• Stress-test the quality gate with the data-validation practice library →, then anchor everything against PipeCode's broader 450+ data-engineering catalogue.

Lock in data version control muscle memory

Docs explain the tools. PipeCode drills explain the decision — when to version files with lakeFS, when Nessie's multi-table atomic commit earns its place, when Dolt's cell-level diff and three-way merge are the only thing that works, and how write-audit-publish gates every change. Pipecode.ai is Leetcode for Data Engineering — pattern-first practice tuned for the production trade-offs senior data engineers actually face.

Practice ETL problems →
Practice design problems →