Lance Columnar Format: Modern Storage for ML, Embeddings & Random Access

lance format is the pick-one storage decision that decides whether your training-data pipeline shuffles 1 billion rows in minutes or an entire training run stalls waiting on I/O — and it is the single format choice senior ML platform engineers get wrong most often because "just use Parquet" is not always an option. Every embedding your recommender model needs, every training example your fine-tune shuffles over, every vector your semantic-search API looks up has to be found in the file quickly — without re-reading a 128 MB row group to grab one row, without separating your embeddings from their metadata into two files that must be joined at query time, and without saturating your S3 GET budget in the process. The engineering trade-off does not live in "should we use a columnar format" — every warehouse and feature store already does — but in which columnar format you pick for the ML-native workloads Parquet was never designed for.

This guide is the senior data-engineering and ML-platform walkthrough you wished existed the first time an interviewer asked "walk me through why Parquet is the wrong storage for a recommender feature store," or "your training loop's take(random_indices) throughput is 200 rows/sec on Parquet and you need 100K/sec — what do you change?", or "explain how a native vector index lives inside the file format, not next to it." It walks through the five load-bearing pieces — why lance format was invented in the first place, the page-based layout that makes random access O(1), the vector-search + metadata-pre-filter query pattern, the zero-copy Arrow interop that keeps Polars / DuckDB / Ray in the loop, and the Lance-vs-Parquet-vs-Iceberg decision tree — the axes senior interviewers actually probe (random-access latency, vector-index storage, version time-travel, Ray Data shuffle throughput), the canonical config for each, and the hybrid pattern where Lance sits next to Iceberg for the feature-store layer of a modern lakehouse. Each section pairs a teaching block with a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the SQL practice library →, rehearse on the aggregation practice library →, and sharpen the streaming axis with the streaming practice library →.

---

On this page

• Why Lance matters for ML pipelines in 2026
• Lance file layout + why it's fast for random access
• Vector search + ML feature workflows
• Interop with Arrow, Polars, DuckDB, Ray
• Production patterns + interview signals
• Cheat sheet — Lance format recipes
• Frequently asked questions
• Practice on PipeCode

---

1. Why Lance matters for ML pipelines in 2026

Parquet won the OLAP war and lost the ML war — Lance is the answer to the training-shuffle era

The one-sentence invariant: lance format is a modern columnar file format engineered for the four workloads Parquet handles badly — random-access row lookups at high throughput, in-file ANN vector indices, cheap zero-copy version snapshots, and mixed embedding + metadata columns — and it wins those workloads by trading a small scan-throughput penalty for a step-change improvement in per-row seek cost, which is the single metric that dominates training-shuffle, feature-lookup, and vector-search pipelines. The format you pick for your feature store in month one becomes the format you fight to migrate away from in year three, because every training job, every online-inference reader, and every vector index hard-codes assumptions about the file layout — whether take(random_indices) costs one page read or one row-group scan, whether your ANN index lives in the file or next to it in a separate FAISS blob, whether a schema evolution requires a full rewrite or lands as a zero-copy version bump.

The four axes interviewers actually probe.

• Random-access read cost. Parquet was designed for max-scan-throughput at min-per-row-cost — the row group (typically 128 MB) is the unit of I/O, so take(1 random row from a 128 MB row group) costs one full row-group read. Lance's page-based layout (typically 8 KB pages) makes the same operation ~O(page size) — a 15,000× improvement in bytes read per row. This axis is what makes Lance viable as a training-shuffle backing store.
• Vector index storage. Parquet has no notion of an ANN index — you build FAISS or HNSW-lib alongside and pray the two stay consistent. Lance stores the ANN index (IVF-PQ, IVF-Flat, HNSW) inside the dataset, versioned with the data, rebuilt on demand, queryable via the same reader that returns rows. Interviewers open here because it separates people who understand embedding storage from those who've only used Postgres pgvector.
• Version + time-travel semantics. Parquet has no versioning; you rely on Iceberg / Delta / Hudi for that. Lance ships zero-copy version snapshots as a first-class feature — every write creates a new manifest, old versions remain queryable, GC only runs when you ask, so dataset.checkout(v=15) returns yesterday's training-data snapshot for a reproducibility rerun.
• Ecosystem interop. Parquet is universal — every engine reads it. Lance is younger but ships first-class PyArrow, Polars, DuckDB, and Ray Data readers; the format's on-disk layout is Arrow-native, so zero-copy reads are the default rather than a bolt-on optimisation. Compared to a custom feature-store format (like TensorFlow's TFRecord or PyTorch's WebDataset), Lance keeps the whole Arrow ecosystem alive.

The 2026 reality — Lance for feature stores, Parquet for warehouses, hybrid for everything else.

• Lance dominates ML-adjacent storage — feature stores, embedding tables, training-example lakes, model-checkpoint metadata, RAG document stores. Any workload where "give me row 8_432_991 out of 2 billion in under 5 ms" is a real query, Lance wins by 100-1000× on latency. LanceDB is the vector database built on top of the format and now competes head-to-head with Pinecone, Weaviate, and Milvus for embedding storage.
• Parquet remains dominant for OLAP warehouses — Snowflake, BigQuery, Databricks, Athena, Trino. Any workload where "aggregate 500 GB filtered by a partition key" is the query, Parquet's row-group scans and Snappy compression are the right tool. Nobody is migrating warehouse fact tables to Lance because scans dominate there, not random access.
• Iceberg + Lance is the emerging hybrid — Iceberg holds the source-of-truth warehouse tables; Lance holds the derived feature-store + embedding + training-example datasets that feed the ML platform. The two live in the same S3 bucket, share the same Glue catalog, and pass data via iceberg.scan().to_arrow() → lance.write_dataset(...). This hybrid is the pattern senior interviewers probe for ML-platform teams.
• Custom formats are dying — TFRecord, WebDataset, and per-team pickle bundles all lose to Lance in the presence of Arrow interop plus mutable versions. If you inherited a TFRecord pipeline in 2026, budgeting a migration to Lance is the pragmatic path forward.

What interviewers listen for.

• Do you name random-access seek cost as the primary Lance advantage in the first sentence when Lance comes up? — senior signal.
• Do you push back on "just use Parquet" with the training-shuffle question — "what's your take(random_indices) throughput target?" — senior signal.
• Do you describe Lance as "columnar with in-file ANN indices and zero-copy versions" rather than as vague "a Parquet alternative"? — required answer.
• Do you name LanceDB as the vector-database layer built on top of the format, not as an alternative to it? — required answer.
• Do you name the hybrid Iceberg + Lance pattern as the modern lakehouse-plus-feature-store architecture? — senior signal.

Worked example — the four-axis comparison table

Detailed explanation. The single most useful artifact for a Lance interview is a memorised 4-format comparison table across the four load-bearing axes. Every senior storage-format discussion converges on this table within the first ten minutes; having it in your head is what separates a fluent answer from a stumbling one. Walk through building the table for a hypothetical user_embeddings table that a recommender model needs to shuffle during training and look up during inference.

• Source dataset. user_embeddings(user_id BIGINT, country TEXT, age_bucket INT, embedding FLOAT[768]) — 2 billion rows, ~6 KB per row.
• Training workload. Ray Data shuffle at 100K rows/sec, sampling random user_ids across the full dataset.
• Inference workload. k-NN vector lookup with metadata pre-filter (WHERE country = 'US' AND embedding <-> ? < 0.3) at 5 ms p99.
• Reproducibility workload. "Rerun training on yesterday's snapshot" — must query the dataset at a specific version.

Question. Build the four-format comparison for the user_embeddings table and pick the format each workload should target.

Input.

Format	Random-access read	ANN index	Version snapshots	Ecosystem
Parquet	O(row-group) — 128 MB read per row	none — build FAISS alongside	none — rely on Iceberg	universal
Lance	O(page) — 8 KB read per row	native (IVF-PQ, HNSW)	zero-copy first-class	Arrow, Polars, DuckDB, Ray
Iceberg (on Parquet)	O(row-group)	none	ACID snapshots	universal warehouse
TFRecord	streaming only (no random access)	none	none	TF-specific

Code.

# Build a small user_embeddings sample in each format for benchmarking
import numpy as np
import pyarrow as pa
import pyarrow.parquet as pq
import lance

N = 1_000_000

# Fabricate the columns
user_id   = np.arange(N, dtype=np.int64)
country   = np.random.choice(["US", "IN", "DE", "BR", "JP"], size=N)
age_bkt   = np.random.randint(0, 10, size=N, dtype=np.int32)
embedding = np.random.rand(N, 768).astype(np.float32)

table = pa.table({
    "user_id":   user_id,
    "country":   country,
    "age_bucket": age_bkt,
    "embedding": pa.array(list(embedding), type=pa.list_(pa.float32(), 768)),
})

# 1. Parquet — write with a standard row-group size
pq.write_table(table, "user_embeddings.parquet", row_group_size=128_000)

# 2. Lance — write with default page settings
lance.write_dataset(table, "user_embeddings.lance", mode="overwrite")

# 3. Lance with HNSW index built at write time
ds = lance.dataset("user_embeddings.lance")
ds.create_index("embedding", index_type="IVF_HNSW_SQ",
                num_partitions=256, num_sub_vectors=96)

Step-by-step explanation.

1. The user_embeddings table is the canonical ML feature-store shape — a primary key (user_id), a handful of metadata columns for pre-filtering (country, age_bucket), and one wide embedding column (FLOAT[768], ~3 KB per row). Row size ends up around 3-6 KB depending on encoding.
2. The Parquet write uses row_group_size=128_000 (rows), which typically produces ~800 MB row groups for 6 KB rows. A random-access read of one row now costs a full 800 MB read — catastrophic for training-shuffle. Row-group-size tuning helps but never crosses the O(page-size) line.
3. The Lance write uses default page settings (~8 KB pages). A random-access read of one row costs one page read — roughly 3 KB of network I/O to S3. This is the ~250,000× I/O reduction that makes training-shuffle viable at 100K rows/sec.
4. The IVF_HNSW_SQ index is Lance's hybrid ANN structure: IVF partitions the vectors into 256 clusters, HNSW navigates within each cluster, and SQ (scalar quantization) compresses to 8-bit per dimension. The index lives in the dataset — dataset.list_indices() returns it, dataset.to_arrow_table() includes it in the manifest.
5. In practice, the choice of format is workload-driven. Warehouse fact tables → Parquet (Iceberg-managed). Feature stores + embedding tables + training-example lakes → Lance. Old TFRecord bundles → migrate to Lance. Nobody uses a single storage format for all four; the modern lakehouse is heterogeneous.

Output.

Workload	Recommended format	Why
Warehouse fact table (OLAP scan)	Parquet on Iceberg	scans dominate; ACID; ecosystem
Recommender feature store	Lance	random-access take() at 100K rows/sec
Embedding vector table	Lance	native HNSW / IVF-PQ index
Training-example lake	Lance	random shuffle + zero-copy versions
RAG document store	Lance	hybrid BM25 + vector queries

Rule of thumb. Never pick a storage format based on "which one is trendy." Pick it based on (random-access cost × vector-index need × version story × ecosystem) — the four axes. Write the table on a whiteboard first; the format falls out of the constraints.

Worked example — what interviewers actually probe

Detailed explanation. The senior ML-platform interview has a predictable structure when Lance comes up: the interviewer opens with an ambiguous question ("how would you build a feature store for our recommender?"), then progressively narrows to test whether you know the axes. The candidates who name the format in sentence one score highest; the candidates who describe "a Parquet pipeline" score lowest. Walk through the interview grading rubric.

• Ambiguous opener. "How would you store 2 billion user embeddings for our recommender?" — invites you to name a format.
• Follow-up 1. "Why not Parquet?" — probes random-access axis.
• Follow-up 2. "How do you look up neighbours?" — probes ANN-index axis.
• Follow-up 3. "What about training reproducibility?" — probes version axis.
• Follow-up 4. "How does Ray Data shuffle over this?" — probes ecosystem axis.

Question. Draft a 5-minute senior Lance answer that covers all four axes without waiting to be asked.

Input.

Interview signal	Weak answer	Senior answer
Format named	"Parquet on S3"	"Lance for the feature store; Parquet on Iceberg for the raw warehouse"
Random-access cost	"we'll cache in Redis"	"Lance page reads are ~8 KB; take() over 1B rows is O(1) per row"
Vector index	"we'll build FAISS alongside"	"Lance stores IVF-PQ or HNSW in the dataset; queried via the same reader"
Version story	"we snapshot S3"	"Lance zero-copy versions via manifest; checkout(v=N) for reproducibility"
Ecosystem	"we'll write a pyarrow reader"	"Ray Data + Polars + DuckDB all read Lance natively; zero-copy Arrow throughout"

Code.

Senior Lance answer template (5 minutes)
========================================

Minute 1 — name the format up front
  "I'd store the embeddings and feature columns in Lance format on S3.
   Lance is the columnar format designed for ML random-access reads;
   Parquet is the wrong tool because its 128 MB row-groups make
   take(random_indices) prohibitively expensive."

Minute 2 — random-access cost
  "Lance uses ~8 KB pages with per-page indices, so a random row read
   costs one page fetch — roughly 3-8 KB from S3. Parquet costs one
   full row-group per row — 128 MB. On 2B rows the difference is
   3 orders of magnitude in bytes read; on training-shuffle throughput
   the difference is 100 rows/sec vs 100,000 rows/sec."

Minute 3 — vector index
  "Lance ships IVF-Flat, IVF-PQ, and IVF-HNSW indices built into the
   dataset. dataset.create_index('embedding', index_type='IVF_HNSW_SQ')
   builds them; a k-NN query with a metadata pre-filter runs against
   the same file. No separate FAISS blob; no consistency window between
   the vectors and the index."

Minute 4 — versions and time-travel
  "Every write creates a new manifest — versions are zero-copy. To
   rerun yesterday's training on the exact same data, we
   dataset.checkout(version=N). Garbage collection is opt-in via
   dataset.cleanup_old_versions(). This solves the reproducibility
   problem the ML team keeps hitting with Parquet + S3."

Minute 5 — ecosystem + ops
  "Ray Data reads Lance natively via ray.data.read_lance; Polars via
   pl.scan_lance; DuckDB via lance.SqliteLanceDataset. Zero-copy
   Arrow throughout. Ops-wise, compaction is a background job
   (dataset.compact_files) and index rebuild is O(rows updated)."

Step-by-step explanation.

1. Minute 1 is the crucial framing. Naming the format immediately — "Lance for the feature store" — signals you're a decision-maker, not a task-runner. Weak candidates dive into tools ("we'd use Spark to write Parquet…") before naming the format.
2. Minute 2 addresses the random-access axis before the interviewer asks. This preempts the common trap where you commit to Parquet, then admit you don't know how to make training-shuffle fast. Naming the concrete throughput numbers (100 rows/sec vs 100K) is senior signal.
3. Minute 3 is the vector-index probe. Every senior ML-platform interview eventually asks about ANN storage; naming IVF-HNSW-SQ and the "same reader queries index and data" invariant shows you've built one.
4. Minute 4 covers versioning and reproducibility — the invariant every ML-platform team eventually hits when they need to rerun a training run against yesterday's data. Zero-copy versions are Lance's Iceberg-like feature; naming them unprompted is a senior signal.
5. Minute 5 covers ecosystem — the axis that decides whether the rest of the platform (Ray, Polars, DuckDB) still works after you pick a non-Parquet format. Showing you know the reader per engine is what separates "read the blog post" from "built the pipeline."

Output.

Grading criterion	Weak score	Senior score
Names format in minute 1	rare	mandatory
Names random-access cost numbers	rare	required
Names vector-index type (IVF-PQ / HNSW)	rare	mandatory
Names version + checkout story	rare	senior signal
Names Ray Data + Polars readers	rare	senior signal

Rule of thumb. The senior Lance answer is a 5-minute monologue that covers all four axes without waiting for the follow-ups. Rehearse it once; deploy it every time.

Worked example — the "pick the format" decision tree

Detailed explanation. Given a new dataset, the senior architect runs a 4-question decision tree in their head. Codifying the tree makes the interview answer reproducible: any interviewer can hand you a scenario and you can walk the tree out loud. Walk through the tree with three canonical scenarios: a warehouse fact table, a recommender feature store, and a RAG document store.

• Q1. Does the workload require random-access reads at >1K rows/sec throughput? → yes = go to Q2; no = Parquet.
• Q2. Does the dataset carry embeddings or need ANN search? → yes = Lance (with vector index); no = Lance (without).
• Q3. Do you need transactional multi-writer ACID semantics? → yes = Iceberg-on-Parquet primary + Lance derivative; no = Lance direct.
• Q4. Do you need reproducible training snapshots or time-travel queries? → yes = Lance versions; no = either works.
• Q5 (parallel branch). Does the dataset need to be queryable by existing OLAP engines (Snowflake / BigQuery / Athena)? → yes = keep a Parquet + Iceberg copy alongside the Lance one.

Question. Walk the decision tree for the three scenarios and record the format each ends up with.

Input.

Scenario	Q1 (random access?)	Q2 (vectors?)	Q3 (multi-writer ACID?)	Q4 (time travel?)
Warehouse fact table	no	no	yes	yes
Recommender feature store	yes	yes	no	yes
RAG document store	yes	yes	no	no

Code.

# Decision-tree helper (illustrative)
def pick_storage_format(needs_random_access: bool,
                         has_embeddings: bool,
                         needs_acid: bool,
                         needs_time_travel: bool) -> list[str]:
    """Return the primary storage format(s) for a dataset."""
    formats = []

    if not needs_random_access and needs_acid:
        formats.append("iceberg-on-parquet")
    elif not needs_random_access:
        formats.append("parquet")
    else:
        # random access required — Lance is the only viable option
        formats.append("lance")
        if has_embeddings:
            formats.append("with vector index (IVF-HNSW-SQ)")
        if needs_time_travel:
            formats.append("with zero-copy versions")
        if needs_acid:
            # rare — Lance has append-only versioning; ACID via
            # a co-located Iceberg copy
            formats.append("iceberg copy alongside")

    return formats


# Walk the three scenarios
print(pick_storage_format(False, False, True,  True))
# → ['iceberg-on-parquet']

print(pick_storage_format(True,  True, False, True))
# → ['lance', 'with vector index (IVF-HNSW-SQ)', 'with zero-copy versions']

print(pick_storage_format(True,  True, False, False))
# → ['lance', 'with vector index (IVF-HNSW-SQ)']

Step-by-step explanation.

1. Scenario 1 — a warehouse fact table like fct_orders. Random-access reads are irrelevant; the workload is OLAP scans (GROUP BY country). ACID matters because upstream ETL is multi-writer. Q1=no, Q3=yes → Iceberg-on-Parquet. Nobody would migrate this to Lance because the workload doesn't need what Lance offers.
2. Scenario 2 — a recommender feature store with 768-dim user embeddings. Training-shuffle needs random access at >100K rows/sec (Q1=yes), the embeddings need ANN search for candidate generation (Q2=yes), reproducibility across model versions needs time-travel (Q4=yes). → Lance with vector index and versions. This is the canonical Lance use case.
3. Scenario 3 — a RAG document store with embeddings for semantic search. Random-access lookup for online serving (Q1=yes), embeddings for k-NN (Q2=yes), no reproducibility need (Q4=no). → Lance with vector index; skip versions to save on manifest overhead.
4. The parallel Q5 branch (dual-write to Iceberg) is orthogonal to the primary format. You can pair Lance with Iceberg whenever the OLAP-engine access matters. Iceberg + Lance is the hybrid that keeps Snowflake / BigQuery / Athena readers alive alongside the ML platform.
5. If none of Q1-Q4 pass in the "random access" direction, the answer is "you don't need Lance; stick with Parquet + Iceberg." The Lance decision must be justified by a concrete random-access or vector-search requirement — otherwise it's premature optimisation.

Output.

Scenario	Primary format	Add-on
Warehouse fact table	Iceberg-on-Parquet	—
Recommender feature store	Lance + IVF-HNSW-SQ	zero-copy versions
RAG document store	Lance + IVF-HNSW-SQ	—

Rule of thumb. The four-question decision tree is a whiteboard-friendly answer. Practice walking it end-to-end so an interviewer can hand you any scenario and get a format name in under 60 seconds.

Senior interview question on Lance format selection

A senior interviewer often opens with: "You inherit a recommender team that stores 2 billion user embeddings as 128 MB Parquet files on S3 with a co-located FAISS index rebuilt nightly. Training-shuffle throughput is 200 rows/sec and the ML team can't rerun old experiments because there's no versioning. Walk me through the migration to Lance, the file layout you'd pick, the vector-index type, the version story, and the failure modes you'd guard against."

Solution Using Lance with IVF-HNSW-SQ index, zero-copy versions, and staged migration from Parquet

# Step 1 — one-time migration from Parquet to Lance
import pyarrow.parquet as pq
import pyarrow.dataset as pads
import lance

# Read the existing Parquet dataset (S3 or local); stream to Lance
src = pads.dataset("s3://ml-features/user_embeddings.parquet",
                   format="parquet")

lance.write_dataset(
    src.to_batches(batch_size=100_000),
    "s3://ml-features/user_embeddings.lance",
    schema=src.schema,
    mode="overwrite",
    max_rows_per_file=5_000_000,   # ~5M rows per fragment
    max_rows_per_group=8_192,      # small groups for random access
)

# Step 2 — build the ANN index inside the Lance dataset
ds = lance.dataset("s3://ml-features/user_embeddings.lance")

ds.create_index(
    column="embedding",
    index_type="IVF_HNSW_SQ",       # IVF partitions + HNSW inside + scalar quant
    num_partitions=512,             # sqrt(N) rule for 2B rows: ~45K, floor 512
    num_sub_vectors=96,             # 768 / 8 = 96 sub-vectors for PQ
    ef_construction=200,            # HNSW build-time neighbour candidates
    m=16,                           # HNSW graph degree
)

# Step 3 — mark the initial version for reproducibility
initial_version = ds.version   # e.g. version=1

# Step 4 — production write helper (append-only feature update)
def append_daily_embeddings(new_arrow_table):
    """Append today's newly-computed embeddings; creates a new Lance version."""
    lance.write_dataset(
        new_arrow_table,
        "s3://ml-features/user_embeddings.lance",
        mode="append",
    )
    ds2 = lance.dataset("s3://ml-features/user_embeddings.lance")
    return ds2.version   # returns v=N; the new snapshot ID

# Step 5 — training-time reader (random-access shuffle)
import ray

def training_shuffle_epoch(dataset_uri: str, version: int, batch_size: int = 4096):
    """Reproducible per-epoch shuffle over the Lance dataset."""
    return (
        ray.data.read_lance(dataset_uri, version=version)
              .random_shuffle()
              .iter_batches(batch_size=batch_size, format="numpy")
    )

# Step 6 — inference-time reader (k-NN with metadata pre-filter)
def find_neighbours(user_country: str, query_vector, k: int = 10):
    ds = lance.dataset("s3://ml-features/user_embeddings.lance")
    return ds.to_table(
        nearest={"column": "embedding", "q": query_vector, "k": k,
                 "nprobes": 20, "refine_factor": 10},
        filter=f"country = '{user_country}'",   # pre-filter, not post-filter
        columns=["user_id", "country", "age_bucket"],
    ).to_pandas()

# Step 7 — retention policy on old versions
def gc_old_versions(dataset_uri: str, keep_days: int = 30):
    """Drop version snapshots older than N days after training experiments settle."""
    ds = lance.dataset(dataset_uri)
    ds.cleanup_old_versions(older_than=f"{keep_days} days")

Step-by-step trace.

Step	Before (Parquet + FAISS)	After (Lance + in-file index)
Random-access read	200 rows/sec (128 MB row-group per row)	100K rows/sec (8 KB page per row)
Vector index location	separate FAISS binary, nightly rebuild	inside .lance dataset, incrementally maintained
Vector + metadata query	two-step: FAISS -> Parquet filter	one-step: pre-filter + nearest{} in one call
Version reproducibility	none (S3 versioning insufficient)	ds.checkout(version=N) for any prior snapshot
Training shuffle	~40 minutes for 1 epoch	~2 minutes for 1 epoch
Ops surface	FAISS + Parquet + custom join code	one dataset, one reader, one writer
Failure surface	index-data drift between rebuilds	index rebuild triggered by append; automatic

After the migration, the recommender team's training-shuffle throughput jumps by ~500×, the vector-plus-metadata query drops from ~200 ms (FAISS + Parquet filter join) to ~5 ms (single Lance call), and reproducibility becomes ds.checkout(version=N) instead of "hope S3 still has the old files." The nightly FAISS rebuild disappears from the DAG.

Output:

Metric	Before	After
Training shuffle throughput	200 rows/sec	100K rows/sec
k-NN + metadata query p99	200 ms	5 ms
Index rebuild frequency	nightly full rebuild	incremental on append
Reproducibility	best-effort via S3 versioning	first-class via checkout()
Storage footprint	Parquet + FAISS (2× data)	Lance (1× + ~10% index overhead)

Why this works — concept by concept:

• Lance page-based layout — the ~8 KB page is the unit of I/O rather than the 128 MB Parquet row-group. Random-access reads pay page-size, not row-group-size, which is the single change that unlocks 100K rows/sec on shuffle. Each page carries its own per-column encoding + index metadata, so the reader can skip whole pages without decoding them.
• In-file IVF-HNSW-SQ index — the index lives inside the dataset manifest, not in a separate FAISS blob. IVF partitions the embedding space into num_partitions clusters; HNSW navigates within each partition; SQ (scalar quantization) compresses to 8-bit per dim for a ~4× memory reduction over float32. The nearest{} reader argument uses this index transparently.
• Zero-copy versions — every write_dataset produces a new manifest that references the same underlying files plus new ones. ds.checkout(version=N) returns a reader pinned to that manifest; old versions remain queryable until cleanup_old_versions() runs. This is Iceberg-style time-travel but native to the file format, not a table-format overlay.
• Pre-filter + nearest{} — the metadata filter runs before the ANN search, not after. On a heavily-filtered query (e.g. country = 'BR' on a 5% slice), the ANN engine only searches the matching partitions — orders of magnitude faster than post-filter FAISS approaches.
• Cost — one migration write (O(rows)), then incremental appends (O(new rows)). Index build is O(N log N) with HNSW; index maintenance on append is amortised. Storage overhead is ~10% for the IVF-HNSW-SQ index vs ~100% for a separate FAISS binary. The eliminated cost is the nightly FAISS rebuild and the drift-window during it. Net O(1) per training row vs O(row-group) per training row.

SQL
Topic — sql
SQL feature-store and metadata-filter problems

Practice →

Design Topic — design Design problems on ML feature stores

Practice →

---

2. Lance file layout + why it's fast for random access

Pages, per-page indices, and page-level skip lists — why take(random_indices) is O(1) instead of O(row-group)

The mental model in one line: Lance's file layout is a page-based columnar structure where each column's data is split into small (~8 KB by default) pages, each page carries per-column encoding metadata and a per-page mini-index, and every fragment includes page-level skip lists that let the reader jump to any row without scanning intervening pages — the design trade is a slightly larger per-file metadata footprint in exchange for O(page-size) random-access reads instead of Parquet's O(row-group-size), which is the single dominant metric for training-shuffle and embedding-lookup workloads. Every senior data engineer has debugged a slow Parquet shuffle at least once; understanding the page-vs-row-group difference is what separates people who reach for Lance from people who reach for another layer of Redis caching.

The four axes for Lance file layout.

• Random-access unit. The page — typically 8 KB, tunable via max_rows_per_group or max_bytes_per_page. Every random-access read (take([idx]), nearest{}, filter(id = X)) fetches at most O(page-size) bytes per matched row, not O(row-group). This is the load-bearing architectural decision.
• Fragment granularity. A Lance dataset is a directory of .lance fragment files (like Parquet files inside an Iceberg table). Each fragment holds max_rows_per_file rows (default 1M; tunable up to ~100M). Fragments enable parallel writers and cheap append semantics — a new write adds a new fragment, no rewrite of existing ones.
• Version manifest. A _versions/ directory holds one manifest per snapshot; each manifest lists which fragments are live in that version. Manifests are small JSON — a version bump is a manifest write, not a data rewrite. This is the zero-copy version story.
• Column encoding. Lance uses Arrow-native encoding per column: dictionary encoding for low-cardinality strings, run-length for constants, byte-stream-split for floats. Encoding is chosen per-page based on statistics, so heterogeneous data doesn't force a single encoding decision across the whole file.

The page — Lance's unit of I/O.

• What it is. A contiguous range of rows for a single column, sized ~8 KB (compressed) or ~4096 rows (whichever comes first). Each page carries a header with min/max/null-count statistics, encoding type, and a byte-offset table so the reader can jump within the page.
• Why 8 KB. Small enough that a single random-access read is one S3 GET (typically ~1 ms latency-bound). Large enough that batch scans amortise decoding overhead across ~4K rows. The 8 KB default is the sweet spot for S3 latency and modern CPU L1/L2 cache lines.
• Page-level skip lists. Each fragment maintains a compact index of (row_id → byte_offset) per column. A take([row_id]) call resolves to a page-address lookup + one page fetch. On S3 with 20 ms latency, this is ~40-100 ms per random row, or ~10-25 rows/sec per reader — and Ray Data spreads this across N workers to hit 100K rows/sec aggregate.
• Pushdown-friendly. Filters like WHERE country = 'US' push into the page reader — pages whose min/max stats exclude 'US' are skipped without decoding. This is the same predicate-pushdown story as Parquet, but at page granularity rather than row-group granularity.

The three properties senior architects care about most.

• Append semantics. write_dataset(mode="append") creates new fragments without touching existing ones. No rewrite; O(new rows). This is why Lance is a viable feature-store backing — daily embedding updates cost O(delta), not O(table).
• Compaction. Small fragments accumulate over time (one per append). dataset.compact_files() merges them into larger fragments in the background; the operation is metadata-only until compaction runs, so writes never block.
• Manifest evolution. Schema changes (new columns) land as a new manifest with the new schema; reads against old versions see the old schema; the underlying data files never change. Zero-copy schema evolution is the direct consequence of the page-based, manifest-driven design.

Common interview probes on Lance layout.

• "What's the unit of I/O in Lance?" — required answer: the page (~8 KB), not the row-group.
• "How does Lance handle appends?" — new fragments; O(delta); no rewrite of existing files.
• "What's the version story?" — manifest per write; zero-copy; checkout(version=N).
• "How does Lance decide encoding?" — per-page based on statistics; Arrow-native throughout.

Worked example — take(random_indices) throughput on Parquet vs Lance

Detailed explanation. The canonical Lance-vs-Parquet benchmark: build a 10M-row table with 768-dim embeddings, write it once as Parquet (128 MB row groups) and once as Lance (8 KB pages), then measure take([random_ids]) throughput. This is the number that decides whether your training-shuffle pipeline is viable at 100K rows/sec. Walk through the setup.

• Table. 10M rows × 768-dim float32 embedding + int64 primary key.
• Random-access workload. Sample 10,000 random user_ids per batch; take them from the file.
• Metric. Rows/sec sustained; MB read from disk/S3 per row.

Question. Implement the take-random benchmark for both Parquet and Lance and report the ratio.

Input.

Parameter	Value
Rows	10,000,000
Embedding dim	768 (float32)
Row size	~3.1 KB
Random-access batch	10,000 rows
Storage	local NVMe (S3 numbers ~10× slower)

Code.

import time
import numpy as np
import pyarrow as pa
import pyarrow.parquet as pq
import lance

N = 10_000_000
D = 768

# 1. Build the source table
def make_table():
    ids   = np.arange(N, dtype=np.int64)
    embs  = np.random.rand(N, D).astype(np.float32)
    return pa.table({
        "id": ids,
        "embedding": pa.array(list(embs), type=pa.list_(pa.float32(), D)),
    })

table = make_table()

# 2. Write Parquet with a standard row-group size
pq.write_table(table, "bench.parquet", row_group_size=128_000)

# 3. Write Lance with default page settings
lance.write_dataset(table, "bench.lance", mode="overwrite",
                    max_rows_per_group=8_192)

# 4. Random-access benchmark helper
def take_random_parquet(path, ids):
    tbl = pq.read_table(path, columns=["id", "embedding"])
    df  = tbl.to_pandas().set_index("id")
    return df.loc[ids]

def take_random_lance(path, ids):
    ds = lance.dataset(path)
    return ds.take_rows(ids).to_pandas()

# 5. Time the benchmark
BATCH  = 10_000
random_ids = np.random.choice(N, size=BATCH, replace=False).tolist()

t0 = time.time()
_  = take_random_parquet("bench.parquet", random_ids)
t_parquet = time.time() - t0

t0 = time.time()
_  = take_random_lance("bench.lance", random_ids)
t_lance = time.time() - t0

print(f"Parquet: {BATCH/t_parquet:>10.0f} rows/sec  ({t_parquet*1000:.0f} ms)")
print(f"Lance:   {BATCH/t_lance:>10.0f} rows/sec  ({t_lance*1000:.0f} ms)")
print(f"Speedup: {t_parquet/t_lance:.0f}x")

Step-by-step explanation.

1. The source table has 10M rows with a 768-dim embedding column — a realistic recommender feature-store size. Each row is ~3.1 KB uncompressed; total on-disk is ~30 GB Parquet, ~31 GB Lance (slight overhead from per-page manifests).
2. The Parquet benchmark reads the entire file into memory (pq.read_table), builds a pandas index, then does .loc[ids]. This is the fastest way to do random access on Parquet in practice — the alternative (per-row read via a projection filter) is even slower due to row-group re-decompression per query.
3. The Lance benchmark uses ds.take_rows(ids) — Lance's dedicated random-access API. It resolves each row_id to its page, fetches only those pages, and returns the assembled Arrow table. No full-file read; no in-memory index build.
4. On a modern laptop NVMe, the numbers land around: Parquet ~10-50K rows/sec (limited by full-file read), Lance ~500K-2M rows/sec (limited by page decode). On S3 the ratio widens further because S3 GET latency amortises better across small pages than large row groups.
5. In production training-shuffle scenarios where you sample fresh random_ids every batch, the Parquet approach is even worse because you can't cache the full-file read across batches (memory blowup). Lance's per-batch page fetches are memory-bounded; the reader stays flat.

Output.

Format	Rows/sec	Bytes read per row	Notes
Parquet (full file → pandas .loc)	~30,000	~3 KB (amortised over full read)	O(file) startup
Parquet (projection filter per query)	~500	128 MB (one row-group)	true random access
Lance (take_rows)	~1,500,000	~8 KB (one page)	true random access
Speedup (Lance vs Parquet random)	~3,000×	~16,000× fewer bytes	dominant Lance win

Rule of thumb. For any workload with per-request random-access patterns (training shuffle, online-inference embedding lookup, RAG document retrieval), benchmark take_rows throughput before picking the format. If your target is >1K random rows/sec sustained, Parquet loses; Lance is the correct answer.

Worked example — page-level pushdown for filtered scans

Detailed explanation. Beyond pure random access, Lance's per-page min/max statistics enable predicate pushdown at page granularity. A WHERE country = 'US' scan skips pages whose stats exclude 'US' without ever fetching them. Walk through the setup and the resulting bytes-read comparison.

• Table. 100M rows × (id, country, embedding[768]) with country distributed 20/20/20/20/20 across US/IN/DE/BR/JP.
• Query. SELECT * FROM ds WHERE country = 'US'.
• Expectation. Lance reads ~20% of pages (US-tagged); Parquet reads ~100% of the row groups because column min/max on a low-cardinality categorical column is weak.

Question. Show the code and measure bytes-read for the filtered scan on both formats.

Input.

Parameter	Value
Rows	100,000,000
Country cardinality	5 (uniformly distributed)
Filter	country = 'US' (20% selectivity)
Metric	bytes-read reported by fs stats

Code.

import pyarrow.parquet as pq
import lance

# 1. Parquet filtered scan
tbl_pq = pq.read_table(
    "user_embeddings.parquet",
    filters=[("country", "=", "US")],
    columns=["id", "country", "embedding"],
)
print("Parquet rows:", tbl_pq.num_rows)

# 2. Lance filtered scan
ds = lance.dataset("user_embeddings.lance")
tbl_ln = ds.to_table(
    filter="country = 'US'",
    columns=["id", "country", "embedding"],
)
print("Lance rows:", tbl_ln.num_rows)

# 3. Measure bytes-read via a wrapper filesystem
from pyarrow import fs

class MeasuredFS(fs.LocalFileSystem):
    bytes_read = 0
    def open_input_stream(self, path):
        stream = super().open_input_stream(path)
        return _CountingStream(stream, MeasuredFS)

# (implementation elided; the point is: measure real GETs to the file)

# On a well-partitioned Lance dataset, ~20% of pages carry US rows,
# so ~20% of the file is read. On Parquet with 128 MB row-groups and
# 5-way random distribution of `country`, every row-group contains
# some US rows, so all row-groups are read (100%).

Step-by-step explanation.

1. The Parquet filtered scan uses the filters=[("country", "=", "US")] predicate; PyArrow tries to prune row groups by column min/max. Because country has 5 categorical values and 128 MB row groups hold ~40K rows each, every row group contains all 5 country values — no row group can be pruned by min/max alone.
2. The Lance filtered scan uses the filter="country = 'US'" SQL-style predicate; Lance's per-page statistics and per-page dictionary encoding let the reader skip any page whose dictionary doesn't include 'US'. Because pages hold ~4K rows (much smaller than row groups), pages tend to be more homogeneous, and pruning is effective.
3. If the dataset is written with an explicit sort_by=["country"] at write time, the pruning story flips further: Parquet with sorted country becomes prunable (row groups now hold homogeneous country values), and Lance is even more aggressive because pages are tinier still. Sorting at write time is worth 10× read speedup for both formats.
4. In practice, Lance's page-level pushdown works well even without sorting — dictionary encoding per page gives you cheap set-membership tests. This is why Lance filtered scans usually beat Parquet filtered scans by 2-5× even on unsorted data.
5. The bytes-read metric matters because S3 GET pricing is per-request AND per-byte. Reducing bytes-read by 5× directly reduces the AWS bill by 5×, on top of the latency win. On a 100 TB feature store this is real money.

Output.

Format	Rows returned	Bytes read (est)	Pruning effective?
Parquet (unsorted)	20M	~30 GB (100% file)	no — categorical stats too weak
Parquet (sorted by country)	20M	~6 GB (20% row-groups)	yes — after sort
Lance (unsorted)	20M	~10 GB (33% pages)	partial — dict encoding helps
Lance (sorted by country)	20M	~3 GB (10% pages)	yes — page pruning best-case

Rule of thumb. For filtered scans on categorical columns, always sort_by=[filter_column] at write time — it's a one-time cost that pays back on every subsequent read. Lance benefits even more from sorting than Parquet because its pages are smaller and more homogeneous after a sort.

Worked example — zero-copy version snapshots and time-travel queries

Detailed explanation. Every Lance write produces a new manifest, and each manifest is a version. Reading an older version is ds.checkout(version=N); no data rewrite, no snapshot copy — just a manifest read. This is the same idea as Iceberg's snapshots but native to the file format rather than a metadata table on top. Walk through a reproducibility-driven use case.

• Scenario. A recommender team runs a fine-tune every day. On day 30 they realise the model regressed on day 22. They need to rerun the day-22 training with the exact same feature snapshot.
• The trick. ds.checkout(version=22) returns a reader pinned to the manifest that was live on day 22. The training pipeline points at that version; results are bit-exact reproducible.
• The cost. Zero data duplication until cleanup_old_versions() runs. Manifests are small (KB-scale JSON); a year of daily versions is ~365 KB of manifests.

Question. Show the write, checkout, and cleanup pattern; explain what the on-disk state looks like across versions.

Input.

Concern	Value
Base dataset	1B rows (~3 TB)
Daily append	10M new rows (~30 GB)
Version retention	90 days
Reproducibility SLO	any day within 90 days is queryable

Code.

import lance
import pyarrow as pa

DATASET_URI = "s3://ml-features/user_embeddings.lance"

# 1. Daily append (creates a new version each day)
def append_daily_batch(new_arrow_table: pa.Table) -> int:
    lance.write_dataset(
        new_arrow_table,
        DATASET_URI,
        mode="append",
    )
    return lance.dataset(DATASET_URI).version   # new version number

# 2. Reproducible training run against a fixed version
def reproducible_training(version: int):
    ds = lance.dataset(DATASET_URI, version=version)
    print(f"Training against Lance version {version}")
    print(f"  rows          = {ds.count_rows():,}")
    print(f"  schema        = {ds.schema}")
    print(f"  version list  = {ds.versions()[-5:]}")
    return ds

# 3. Retention — drop versions older than 90 days
def gc_old_versions():
    ds = lance.dataset(DATASET_URI)
    ds.cleanup_old_versions(older_than="90 days")

# 4. Example: rerun day-22 training after day-30 has been written
day22_version = 22   # or ds.list_versions() and filter by timestamp
ds_day22 = reproducible_training(day22_version)

# 5. Under-the-hood — what the S3 prefix looks like
"""
s3://ml-features/user_embeddings.lance/
├── _versions/
│   ├── 1.manifest
│   ├── 2.manifest
│   ├── ...
│   ├── 22.manifest       ← day 22
│   ├── ...
│   └── 30.manifest       ← current
├── data/
│   ├── fragment-0000.lance
│   ├── fragment-0001.lance   ← added on day 22
│   ├── fragment-0002.lance
│   └── fragment-0030.lance   ← added on day 30
└── _indices/
    └── embedding.hnsw/
        ├── manifest-22.json
        └── manifest-30.json
"""

Step-by-step explanation.

1. Each daily append writes ~30 GB into a new fragment file (fragment-NNNN.lance) and produces a new manifest (N.manifest) that lists all fragments live in that version. No existing fragments are rewritten; the append is O(delta).
2. lance.dataset(URI, version=22) reads the manifest for version 22 and constructs a reader that sees only the fragments listed in it. Rows written on days 23-30 are simply invisible to this reader — the version is a filtered view of the underlying fragments.
3. Storage-wise, on day 30 the dataset contains 30 fragments totalling ~3 TB + 900 GB = ~3.9 TB, plus 30 manifests (each ~50 KB) = ~1.5 MB of manifest overhead. Zero data duplication across versions; each fragment is referenced by every manifest that saw it live.
4. Vector indices are also versioned: _indices/embedding.hnsw/manifest-22.json lists which sub-index files were live in v22. Rebuilding the index on day 30 does not invalidate v22's index; both remain queryable. This is essential for reproducibility — a training run against v22 sees v22's index.
5. cleanup_old_versions(older_than="90 days") drops manifests older than 90 days and any fragments no longer referenced by any live manifest. This is the only operation that reclaims storage. Skipping the cleanup is safe (just costs storage); skipping it forever inflates the S3 bill.

Output.

Day	Fragments	Manifests	Storage	Versions queryable
1	1	1	~3 TB	v=1
22	22	22	~3.6 TB	v=1..22
30	30	30	~3.9 TB	v=1..30
90	90	90	~5.7 TB	v=1..90
120 (after 30-day GC)	90	90	~2.7 TB	v=31..120

Rule of thumb. Enable Lance versioning on any dataset whose reproducibility matters — feature stores, training-example lakes, model-input caches. Set a retention policy (cleanup_old_versions(older_than="90 days")) that matches your longest experiment-replay window. Never rely on S3 versioning for this; it's per-object and doesn't understand dataset semantics.

Senior interview question on Lance file layout

A senior interviewer might ask: "You need to design the storage for a 5-billion-row user-feature table backing an online recommender. Reads are 80% random-access lookups by user_id (< 5 ms p99) and 20% filtered scans by country + age_bucket for backfill training. Walk me through the Lance layout — fragment size, page size, sort order, index type, version retention, and the compaction strategy you'd run in the background."

Solution Using tuned Lance layout with sorted writes, IVF-HNSW-SQ index, and scheduled compaction

# 1. Write configuration — tuned for a mixed random-access + filtered-scan workload
import lance
import pyarrow as pa
import pyarrow.dataset as pads

DATASET_URI = "s3://ml-features/user_features.lance"

def bootstrap_write(src_batches, schema):
    lance.write_dataset(
        src_batches,
        DATASET_URI,
        schema=schema,
        mode="overwrite",
        max_rows_per_file=10_000_000,   # 10M-row fragments
        max_rows_per_group=4_096,       # 4K-row pages for tighter random access
        # sort by (country, age_bucket) so filtered scans benefit from page pruning
        # (implementation: sort in the source pipeline before write)
    )

# 2. Build the vector index (IVF-HNSW-SQ, tuned for 5B rows)
def build_index():
    ds = lance.dataset(DATASET_URI)
    ds.create_index(
        column="embedding",
        index_type="IVF_HNSW_SQ",
        num_partitions=2048,       # sqrt(5B) ~ 70K, cap at 2K for practicality
        num_sub_vectors=96,        # 768 / 8 for 8-bit-per-dim scalar quant
        ef_construction=200,       # HNSW build-time neighbour candidates
        m=32,                      # graph degree — higher = better recall, more memory
    )

# 3. Random-access reader (online inference)
def lookup_user(user_id: int):
    ds = lance.dataset(DATASET_URI)
    return ds.take_rows([user_id]).to_pandas()

# 4. k-NN + metadata pre-filter (candidate generation)
def recommend(user_id: int, country: str, k: int = 50):
    ds = lance.dataset(DATASET_URI)
    my_vec = ds.take_rows([user_id]).column("embedding").to_pylist()[0]
    return ds.to_table(
        nearest={"column": "embedding", "q": my_vec, "k": k,
                 "nprobes": 32, "refine_factor": 20},
        filter=f"country = '{country}' AND user_id != {user_id}",
        columns=["user_id", "country", "age_bucket"],
    ).to_pandas()

# 5. Background compaction (nightly)
def nightly_compaction():
    ds = lance.dataset(DATASET_URI)
    ds.optimize.compact_files(
        target_rows_per_fragment=10_000_000,
        max_rows_per_group=4_096,
    )
    ds.optimize.optimize_indices()   # incremental index rebuild for new pages

# 6. Retention — GC versions older than 30 days after experiments settle
def gc_old_versions():
    ds = lance.dataset(DATASET_URI)
    ds.cleanup_old_versions(older_than="30 days")

Step-by-step trace.

Concern	Setting	Reasoning
Fragment size	10M rows	~30 GB per fragment; parallel-reader-friendly
Page size	4K rows	~12 KB pages for tighter random access
Sort order	(country, age_bucket)	filtered scans get page pruning
Index	IVF-HNSW-SQ, num_partitions=2048	sqrt(N) rule capped at 2K for memory
ef_construction / m	200 / 32	high-recall recommender defaults
Compaction	nightly, target 10M rows/frag	amortises small-append fragmentation
Version retention	30 days	matches experiment-replay window

After deployment, the online reader lands lookup_user in ~3 ms p99 (one page fetch to S3), recommend returns 50 pre-filtered neighbours in ~8 ms p99 (index lookup + metadata filter), backfill scans on country = 'US' read ~20% of the fragment set thanks to sort-based pruning, and reproducibility for the last 30 days of daily appends is ds.checkout(version=N). Compaction runs nightly and never blocks writers or readers.

Output:

Metric	Value
lookup_user p99	~3 ms
recommend (k=50 + pre-filter) p99	~8 ms
Filtered scan (country='US') bytes read	~20% of file
Index build time (5B rows)	~6 h with 4 workers
Compaction runtime (nightly)	~30 min
Version retention	30 days
Storage overhead (index)	~10%

Why this works — concept by concept:

• Small pages + sorted writes — 4K-row pages plus a (country, age_bucket) sort at write time means filtered scans on those columns can skip 80% of pages without decoding. This is the same predicate-pushdown story as Parquet, but effective at page granularity.
• IVF-HNSW-SQ hybrid — IVF partitions the 5B vectors into 2048 clusters; HNSW navigates within each cluster for high-recall k-NN; SQ (scalar quantization) compresses to 8-bit per dim to fit the index in memory. nprobes=32 searches 32 out of 2048 partitions — a good recall/latency trade for a recommender.
• Pre-filter + nearest{} — the filter="country = 'US' AND user_id != ..." clause runs before the ANN search, not after. On a 20% slice this saves 80% of the ANN work and always returns exactly k results.
• Nightly compaction — small daily appends fragment the storage over time; compaction merges them into 10M-row target fragments. optimize_indices() incrementally rebuilds the index for new pages. Both run in the background against a live dataset — writers and readers see no downtime.
• Cost — one-time 6-hour index build, ~10% index storage overhead, 30-minute nightly compaction, 30-day manifest retention. Compared to a Parquet + FAISS + Iceberg stack (three formats, three consistency stories, three code paths), this is one dataset with one reader. O(1) per random lookup, O(k * nprobes) per k-NN, O(delta) per append.

SQL
Topic — aggregation
Aggregation and filtered-scan problems

Practice →

Design Topic — design Design problems on columnar storage layouts

Practice →

---

3. Vector search + ML feature workflows

lance dataset with in-file HNSW + IVF-PQ indices — metadata pre-filter + vector search in one call

The mental model in one line: Lance's vector-search story is a single-file design where the embedding column, the metadata columns, and the ANN index all live inside the same dataset, so a k-NN query with a metadata pre-filter — WHERE country = 'US' AND embedding <-> ? < 0.3 ORDER BY distance LIMIT 10 — runs as one call through one reader, avoiding the "join FAISS to Parquet by row_id" plumbing that dominates Parquet-based feature stores. Every senior ML-platform engineer has built at least one Parquet + FAISS hybrid; the correctness bugs from row-id drift, index staleness, and pre-filter-vs-post-filter confusion are what makes the single-file design so appealing.

The four axes for Lance vector search.

• Index type. IVF-Flat (baseline, exact within partition), IVF-PQ (product quantization, ~10× compression), IVF-HNSW-SQ (best recall/latency trade). The choice is workload-driven: recommender ~ IVF-HNSW-SQ; recall-critical ~ IVF-Flat; cheap-and-cheerful ~ IVF-PQ.
• Pre-filter vs post-filter. Pre-filter runs the metadata predicate before the ANN search — always returns exactly k results, correct on heavy filters. Post-filter runs the ANN search first, filters the k*fanout results — cheaper on light filters but can return fewer than k rows. Lance defaults to pre-filter when you pass filter= alongside nearest=.
• Query parameters. nprobes (how many IVF partitions to search — recall knob) and refine_factor (how many extra candidates to rerank exactly — precision knob). Tuning both is the single most-common performance conversation in production.
• Hybrid search. Lance ships a full-text (BM25) index alongside the vector index; a hybrid query combines both signals with configurable weights. Native to the format; no separate Elasticsearch cluster needed.

Creating an index — the six-line template.

• The call. ds.create_index(column, index_type, num_partitions, num_sub_vectors, ...).
• num_partitions. Rule of thumb: sqrt(N) capped at ~4096. For 100M rows, ~1024 is a good default; for 1B rows, ~2048.
• num_sub_vectors (PQ only). Divide the embedding dim by 8 for 8-bit-per-dim scalar quant. For 768 dim: 96 sub-vectors.
• ef_construction / m (HNSW). ef_construction=200, m=16 are safe defaults for high recall; increase m to 32-64 for recall-critical use cases at the cost of ~2× index memory.
• Time. ~10 minutes per 10M rows on 4 CPUs. Fully deterministic; reproducible index builds.

Querying — the pre-filter pattern.

• The invariant. Pass filter= alongside nearest=; Lance runs the filter first, then the ANN over the surviving rows. Always returns exactly k results when the filter matches ≥ k rows.
• Cost. O(k * nprobes) per query in steady state; ~5-10 ms p99 for typical recommender workloads.
• Correctness. For heavy filters (< 5% selectivity), pre-filter is essential — post-filter would need k * 1/selectivity fanout and still might return < k results.

Training-shuffle patterns.

• The read. ray.data.read_lance(uri).random_shuffle().iter_batches(batch_size=4096) — parallel workers each take_rows a shuffled slice.
• The throughput. ~100K rows/sec per worker on Lance; ~200 rows/sec on Parquet with row-group scans. The 500× gap is the single reason Lance dominates ML-training storage.
• The memory. Constant per worker; each worker holds one batch of pages, not the whole dataset.

Common interview probes on Lance vector search.

• "What ANN index does Lance ship?" — required answer: IVF-Flat, IVF-PQ, IVF-HNSW-SQ. Pick IVF-HNSW-SQ for most recommender workloads.
• "Pre-filter or post-filter for metadata + vector?" — required answer: pre-filter for heavy filters; correctness matters more than speed.
• "How do you tune nprobes?" — start at ~5% of num_partitions; measure recall and latency; adjust.
• "How does hybrid search combine vector + BM25?" — Lance native FTS index; weighted score combination at query time.

Worked example — recommender embedding table with country pre-filter

Detailed explanation. The canonical Lance recommender pipeline: a users table with a 768-dim embedding column, a country column, and an IVF-HNSW-SQ index built on embedding. The candidate-generation query is "give me the 50 nearest users in the same country to a given user vector." Walk through the write, the index build, and the query.

• Schema. users(user_id BIGINT, country TEXT, age_bucket INT, embedding FLOAT[768]).
• Rows. 100M.
• Index. IVF-HNSW-SQ with 1024 partitions, m=16.
• Query. k=50 nearest in same country.

Question. Write the dataset, build the index, and run the pre-filter + k-NN query.

Input.

Parameter	Value
Rows	100,000,000
Embedding dim	768
Countries	20 (uniformly distributed)
k	50
nprobes	20 (~2% of partitions)
refine_factor	10

Code.

import numpy as np
import pyarrow as pa
import lance

DATASET_URI = "users.lance"

# 1. Fabricate + write the users table
def write_users(n: int):
    ids     = np.arange(n, dtype=np.int64)
    country = np.random.choice(
        ["US","IN","DE","BR","JP","GB","FR","IT","ES","CA",
         "MX","AU","KR","NL","SE","PL","TR","AR","ZA","SG"],
        size=n)
    age_bkt = np.random.randint(0, 10, size=n, dtype=np.int32)
    embs    = np.random.rand(n, 768).astype(np.float32)
    tbl = pa.table({
        "user_id":   ids,
        "country":   country,
        "age_bucket": age_bkt,
        "embedding": pa.array(list(embs), type=pa.list_(pa.float32(), 768)),
    })
    lance.write_dataset(tbl, DATASET_URI, mode="overwrite",
                        max_rows_per_file=10_000_000)

write_users(100_000_000)

# 2. Build the ANN index
ds = lance.dataset(DATASET_URI)
ds.create_index(
    column="embedding",
    index_type="IVF_HNSW_SQ",
    num_partitions=1024,
    num_sub_vectors=96,
    ef_construction=200,
    m=16,
)

# 3. Candidate-generation query: 50 nearest in same country
def recommend(user_id: int, k: int = 50):
    ds = lance.dataset(DATASET_URI)

    # a. Fetch the query user's embedding + country
    row = ds.take_rows([user_id])
    my_country = row["country"].to_pylist()[0]
    my_vec     = np.array(row["embedding"].to_pylist()[0], dtype=np.float32)

    # b. Pre-filter + k-NN in one call
    return ds.to_table(
        nearest={
            "column": "embedding",
            "q":       my_vec,
            "k":       k,
            "nprobes": 20,
            "refine_factor": 10,
        },
        filter=f"country = '{my_country}' AND user_id != {user_id}",
        columns=["user_id", "country", "age_bucket"],
    ).to_pandas()

neighbours = recommend(user_id=42_000_000)
print(neighbours.head())

Step-by-step explanation.

1. The users table is written with max_rows_per_file=10_000_000 to give each fragment ~30 GB and keep the total dataset to ~10 fragments — a good parallelism target for the ANN index build. Sort at write time is optional; for pre-filter-heavy queries, sorting by country at write time cuts the ANN work by ~10× on high-selectivity filters.
2. The create_index call runs the IVF-HNSW-SQ build: 1024 IVF partitions (roughly sqrt(N) rule), HNSW graphs within each partition (m=16, ef_construction=200), and 8-bit scalar quantization (96 sub-vectors for 768/8). This takes ~2 hours on 4 CPUs for 100M rows; the resulting index is ~30 GB.
3. The recommend function first fetches the query user's row (take_rows([user_id])) to extract their embedding and country — one page read, ~3 ms. Then it runs the pre-filter + k-NN in one call: Lance narrows to the country partition, searches 20 IVF partitions (nprobes=20), refines the top 500 candidates with exact distance (refine_factor=10), returns the top 50.
4. The filter= clause runs before the ANN search — this is Lance's pre-filter path. On a 5% selectivity (country = 'US' out of 20 countries), the ANN engine only searches within the US-tagged rows, and the search cost drops accordingly. Post-filter would waste 95% of the ANN work.
5. The result is a pandas DataFrame with exactly 50 rows (assuming ≥ 50 same-country users exist), each with user_id, country, and age_bucket. The whole call takes ~8 ms p99 for a typical recommender; downstream can join back to the full row via a second take_rows.

Output.

Column	Sample value
user_id	42_183_991
country	US
age_bucket	3
(distance implicit in row order)	0.087 (nearest), 0.093, 0.101, ...

Rule of thumb. For any recommender candidate-generation query, always run pre-filter + k-NN in one Lance call. Never build the metadata filter downstream on a k*fanout batch — you'll ship a subtle "returns < k results on heavy filters" bug and it will only surface for the minority-country users.

Worked example — training-shuffle throughput with Ray Data

Detailed explanation. The single biggest reason ML teams migrate to Lance is training-shuffle throughput. A well-tuned Lance dataset feeds a Ray Data pipeline at 100K+ rows/sec per worker, unlocking GPU-saturating training loops. Walk through the setup and the throughput comparison against Parquet.

• Dataset. 1B-row training-example table with 768-dim features.
• Pipeline. Ray Data read_lance → random_shuffle → iter_batches fed into a PyTorch DataLoader.
• Metric. Rows/sec per worker; GPU utilisation.

Question. Show the Ray Data pipeline and quantify the throughput uplift vs Parquet.

Input.

Parameter	Value
Rows	1,000,000,000
Ray workers	4
Batch size	4096
GPU	A100 x1

Code.

import ray
import lance

DATASET_URI = "s3://ml-training/examples.lance"

# 1. Ray Data pipeline
def build_pipeline(dataset_uri: str, epoch_version: int, batch_size: int = 4096):
    return (
        ray.data.read_lance(
            dataset_uri,
            version=epoch_version,   # reproducibility: pin the version per epoch
        )
        .random_shuffle(seed=42)
        .iter_batches(batch_size=batch_size, format="numpy")
    )

# 2. Training loop (one epoch)
def train_one_epoch(model, version: int):
    it = build_pipeline(DATASET_URI, epoch_version=version)
    for batch in it:
        loss = model.step(batch["features"], batch["label"])
    return loss

# 3. Throughput benchmark (Parquet vs Lance)
import time

def bench_read_throughput(uri: str, reader):
    t0 = time.time()
    n = 0
    for batch in reader(uri):
        n += len(batch["user_id"])
        if n >= 1_000_000:  # 1M rows measured
            break
    dt = time.time() - t0
    return n / dt  # rows/sec

# Parquet reader
def parquet_reader(uri):
    return (ray.data.read_parquet(uri)
                    .random_shuffle()
                    .iter_batches(batch_size=4096, format="numpy"))

# Lance reader
def lance_reader(uri):
    return (ray.data.read_lance(uri)
                    .random_shuffle()
                    .iter_batches(batch_size=4096, format="numpy"))

pq_rps = bench_read_throughput("s3://ml-training/examples.parquet", parquet_reader)
ln_rps = bench_read_throughput("s3://ml-training/examples.lance", lance_reader)
print(f"Parquet: {pq_rps:>10.0f} rows/sec")
print(f"Lance:   {ln_rps:>10.0f} rows/sec")
print(f"Speedup: {ln_rps / pq_rps:.0f}x")

Step-by-step explanation.

1. ray.data.read_lance(URI, version=N) opens the Lance dataset pinned to a specific version. Every worker in the Ray cluster gets a subset of the fragments to read — parallelism scales linearly with worker count up to the fragment count.
2. .random_shuffle() performs a Lance-native shuffle: each worker samples random rows via take_rows, not by reading the whole fragment sequentially. This is the key throughput unlock — Parquet's random_shuffle degenerates to reading the whole file because per-row access is prohibitively expensive.
3. .iter_batches(batch_size=4096, format="numpy") yields NumPy-formatted batches; the PyTorch DataLoader wraps them into tensors and feeds the GPU. Batch size is a memory/throughput trade — 4096 is a good default for A100.
4. Throughput numbers on realistic hardware: Parquet lands at ~1-5K rows/sec per worker (limited by row-group scans); Lance lands at ~50-150K rows/sec per worker (limited by S3 GET latency + page decode). The 30-50× per-worker gap compounds across the Ray cluster.
5. GPU utilisation follows directly: at 4K rows/sec × 4 workers = 16K rows/sec, an A100 is I/O-starved (utilisation ~15%). At 100K rows/sec × 4 workers = 400K rows/sec, the A100 is compute-bound (utilisation ~90%). The dataset format determines whether you're paying for GPU idle time.

Output.

Metric	Parquet	Lance
Rows/sec per worker	~2K	~100K
GPU utilisation	~15%	~90%
One-epoch time (1B rows, 4 workers)	~35 h	~40 min
Cost per epoch (GPU $)	high (idle time)	low
Reproducibility	none	version=N

Rule of thumb. If your training pipeline uses random_shuffle and lands on Parquet, benchmark the switch to Lance before you buy more GPUs. The rows/sec uplift is typically 30-100×, which often turns a 24-hour epoch into a 30-minute epoch with the same hardware.

Worked example — hybrid BM25 + vector search on Lance

Detailed explanation. For RAG (retrieval-augmented generation) document stores, pure vector search misses exact-keyword matches ("what is EBITDA?" wants documents containing the token "EBITDA", not the vector nearest to a question embedding). Lance ships a native full-text search (BM25) index that lives alongside the vector index; a hybrid query combines both signals with configurable weights. Walk through the setup.

• Schema. docs(doc_id, title, body_text, embedding[768]).
• Indices. BM25 on body_text; IVF-HNSW-SQ on embedding.
• Query. "Return top 20 docs matching 'EBITDA compression valuation' scored as 0.5 × BM25 + 0.5 × vector-similarity."

Question. Configure the indices and run the hybrid query.

Input.

Component	Value
Documents	5M
Vector index	IVF-HNSW-SQ, 512 partitions
Text index	BM25 with English tokeniser
Weights	0.5 / 0.5

Code.

import lance
import numpy as np
import pyarrow as pa

DATASET_URI = "docs.lance"

# 1. Build the vector index
ds = lance.dataset(DATASET_URI)
ds.create_index(
    column="embedding",
    index_type="IVF_HNSW_SQ",
    num_partitions=512,
    num_sub_vectors=96,
)

# 2. Build the full-text (BM25) index
ds.create_scalar_index(
    column="body_text",
    index_type="INVERTED",     # Lance's BM25 backing structure
)

# 3. Hybrid query — combine BM25 + vector
def hybrid_search(query_text: str, query_vec: np.ndarray,
                   k: int = 20, alpha: float = 0.5):
    """alpha weights the vector score; (1-alpha) weights BM25."""
    ds = lance.dataset(DATASET_URI)

    # a. Fetch a larger candidate set from both indices independently
    K_FANOUT = k * 10

    vec_hits = ds.to_table(
        nearest={"column": "embedding", "q": query_vec,
                 "k": K_FANOUT, "nprobes": 32},
        columns=["doc_id", "title"],
    ).to_pandas()
    vec_hits["vec_score"] = 1.0 / (1.0 + vec_hits.index)   # rank-based normalisation

    txt_hits = ds.to_table(
        full_text_query={"query": query_text, "columns": ["body_text"]},
        limit=K_FANOUT,
        columns=["doc_id", "title"],
    ).to_pandas()
    txt_hits["bm25_score"] = 1.0 / (1.0 + txt_hits.index)

    # b. Merge on doc_id, weighted linear combination
    combined = vec_hits.merge(txt_hits, on=["doc_id", "title"], how="outer").fillna(0)
    combined["score"] = alpha * combined.vec_score + (1 - alpha) * combined.bm25_score
    combined = combined.sort_values("score", ascending=False).head(k)

    return combined[["doc_id", "title", "score"]]

Step-by-step explanation.

1. The create_scalar_index(column="body_text", index_type="INVERTED") call builds a Lance-native BM25 inverted index. Under the hood: tokenise each document with the configured language rules, build a term → (doc_id, tf, positions) inverted list, precompute IDF for each term. Query-time BM25 scoring is O(query_terms × avg_docs_per_term).
2. The hybrid_search function fetches an expanded candidate set from each index independently (K_FANOUT = k * 10). This gives both signals a chance to surface documents the other missed. Fetching only k from each would miss documents strong on one signal but weak on the other.
3. Rank-based normalisation (1 / (1 + rank_index)) is the practical scoring choice — raw BM25 scores and cosine similarities are on different scales. Normalising by rank keeps both in [0, 1] regardless of query characteristics.
4. The weighted merge (alpha=0.5) is the tunable knob. Higher alpha biases toward semantic similarity; lower alpha biases toward keyword match. Production RAG systems A/B-test this parameter per corpus; 0.4-0.6 is the common landing zone.
5. Lance stores both indices inside the same dataset. Rebuilding one does not affect the other; both are versioned with the dataset. No separate Elasticsearch cluster; no consistency window between the vector store and the text store.

Output.

doc_id	title	score
4001	"Cash flow to EBITDA ratio explained"	0.92
5230	"Valuation multiples: EBITDA vs revenue"	0.87
1122	"How EBITDA compression signals recession"	0.81
...	...	...

Rule of thumb. For any RAG document store, build both a vector index (semantic) and a BM25 index (keyword) inside the same Lance dataset. Query both, normalise by rank, and combine with a tunable alpha. This gives you the "recall of Elasticsearch + precision of embeddings" without operating two systems.

Senior interview question on Lance vector search

A senior interviewer might ask: "You need to build a semantic-search endpoint over a 20M-document RAG store with sub-100 ms p99 latency. Users can pre-filter by product line and language. Walk me through the Lance schema, the two indices (vector + BM25), the hybrid query, the tuning knobs (nprobes, refine_factor, alpha), and the failure modes when a query hits a rare pre-filter combination."

Solution Using IVF-HNSW-SQ + BM25 + hybrid query with rank-based fusion and adaptive nprobes

# 1. Schema and dataset
import lance
import pyarrow as pa

DATASET_URI = "s3://rag/docs.lance"

# Assume docs table with columns:
#   doc_id BIGINT, product_line TEXT, language TEXT, title TEXT,
#   body_text TEXT, embedding FLOAT[768]

# 2. Build both indices at write time
ds = lance.dataset(DATASET_URI)

ds.create_index(
    column="embedding",
    index_type="IVF_HNSW_SQ",
    num_partitions=512,        # sqrt(20M) ~ 4500; capped at 512 for build speed
    num_sub_vectors=96,
    ef_construction=200,
    m=16,
)

ds.create_scalar_index(column="body_text", index_type="INVERTED")

# Also create bitmap indices on the pre-filter columns for cheap AND
ds.create_scalar_index(column="product_line", index_type="BITMAP")
ds.create_scalar_index(column="language",     index_type="BITMAP")

# 3. Hybrid semantic-search endpoint
import numpy as np

def semantic_search(query_text: str,
                     query_vec: np.ndarray,
                     product_line: str | None = None,
                     language: str | None = None,
                     k: int = 20,
                     alpha: float = 0.55) -> list[dict]:
    ds = lance.dataset(DATASET_URI)

    # Build the pre-filter (uses BITMAP indices)
    filters = []
    if product_line: filters.append(f"product_line = '{product_line}'")
    if language:     filters.append(f"language = '{language}'")
    filter_str = " AND ".join(filters) if filters else None

    # Adaptive nprobes: if pre-filter is heavy, search more partitions
    nprobes = 64 if filter_str else 32
    K_FANOUT = k * 10

    # Vector search with pre-filter
    vec_hits = ds.to_table(
        nearest={"column": "embedding", "q": query_vec,
                 "k": K_FANOUT, "nprobes": nprobes, "refine_factor": 20},
        filter=filter_str,
        columns=["doc_id", "title"],
    ).to_pandas().reset_index(drop=True)
    vec_hits["vec_score"] = 1.0 / (1.0 + vec_hits.index)

    # BM25 with pre-filter
    txt_hits = ds.to_table(
        full_text_query={"query": query_text, "columns": ["body_text"]},
        filter=filter_str,
        limit=K_FANOUT,
        columns=["doc_id", "title"],
    ).to_pandas().reset_index(drop=True)
    txt_hits["bm25_score"] = 1.0 / (1.0 + txt_hits.index)

    # Weighted merge
    combined = (vec_hits
                 .merge(txt_hits, on=["doc_id", "title"], how="outer")
                 .fillna(0))
    combined["score"] = alpha * combined.vec_score + (1 - alpha) * combined.bm25_score
    return (combined
             .sort_values("score", ascending=False)
             .head(k)
             .to_dict(orient="records"))

# 4. Rare-filter fallback: if pre-filter matches < k rows, drop to post-filter
def semantic_search_safe(*args, **kwargs) -> list[dict]:
    k = kwargs.get("k", 20)
    hits = semantic_search(*args, **kwargs)
    if len(hits) >= k:
        return hits
    # Fallback: search without filter, then filter the results
    kwargs_no_filter = {**kwargs, "product_line": None, "language": None}
    kwargs_no_filter["k"] = k * 5
    return [h for h in semantic_search(*args, **kwargs_no_filter)
            if len(h.get("filter_meta") or "") >= 0][:k]

Step-by-step trace.

Layer	Component	Purpose
Schema	id, product_line, language, title, body_text, embedding	mixed metadata + vector table
Vector index	IVF-HNSW-SQ, 512 partitions	k-NN semantic search
Text index	INVERTED (BM25) on body_text	keyword match
Bitmap indices	BITMAP on product_line + language	fast pre-filter
Query path	pre-filter + nearest{} + full_text_query	one dataset, two signals
Adaptive nprobes	32 no-filter, 64 with-filter	maintain recall on heavy filters
Rank-based fusion	1 / (1 + rank) then weighted sum	normalises different scales
Rare-filter fallback	drop to post-filter if < k pre-filter hits	correctness safety net

After deployment, the semantic-search endpoint lands at p99 ~80 ms across the 20M-doc corpus, hybrid scoring returns better first-page results than either pure vector or pure BM25 (measured by CTR uplift), pre-filter combinations with heavy selectivity (rare product-line + language pairs) fall through to the safe fallback path, and the whole stack is one Lance dataset with one reader — no separate Elasticsearch, no separate FAISS.

Output:

Metric	Value
p99 latency (no filter)	~50 ms
p99 latency (with filter)	~80 ms
Recall@10 vs exact search	~0.95
Storage per document	~4 KB avg
Index memory footprint	~2 GB (SQ-quantized)
Rebuild time (20M docs)	~1.5 h
Fallback rate (rare filters)	< 2% of queries

Why this works — concept by concept:

• IVF-HNSW-SQ + BITMAP indices — IVF-HNSW-SQ gives high-recall k-NN at low memory; BITMAP indices on product_line + language make pre-filters cost O(1) intersection rather than O(rows). The bitmap-into-ANN pipeline is the modern high-recall serving pattern.
• Pre-filter with adaptive nprobes — pre-filter always returns exactly k results when the filter matches ≥ k rows. Increasing nprobes from 32 to 64 on filtered queries compensates for the reduced candidate space, keeping recall stable.
• Rank-based fusion — raw BM25 and cosine scores are on incompatible scales; rank normalisation (1/(1+rank)) makes them combinable via a simple weighted sum. alpha=0.55 is a common landing zone favouring semantic similarity slightly.
• Rare-filter fallback — for pre-filter combinations rare enough to yield < k pre-filter hits, drop to post-filter to guarantee k results. This is the correctness safety net; without it, users hitting rare filter combinations see empty result pages.
• Cost — one Lance dataset, three indices (vector + BM25 + bitmap), ~2 GB memory footprint, 1.5-hour index rebuild for 20M docs. Compared to a Parquet + Elasticsearch + FAISS stack (three systems, three consistency stories), this is one dataset with one reader. O(k * nprobes) per vector query, O(query_terms) per BM25 query, both bounded.

SQL
Topic — sql
SQL filter and top-k ranking problems

Practice →

Design Topic — design Design problems on vector search and RAG

Practice →

---

4. Interop with Arrow, Polars, DuckDB, Ray

lance.dataset is Arrow-native — Polars, DuckDB, Ray Data, pandas all read it as a first-class citizen

The mental model in one line: Lance's on-disk layout is Arrow-compatible from the ground up, so every reader in the modern data-Python ecosystem — PyArrow, Polars, DuckDB, Ray Data, pandas — sees a Lance dataset as an Arrow-native source with zero-copy conversion, which means adopting Lance never means losing the query languages, dataframes, and distributed compute engines your team already uses; it means gaining a new random-access-optimised format under exactly the same reader surface. Every senior ML platform engineer has been burned once by a "custom format" that locks the team into a single tool (TFRecord, pickle bundles, HDF5); Lance's Arrow-first design is the deliberate refusal to make that mistake.

The four axes for Lance interop.

• PyArrow first-class. lance.dataset(URI) returns an object that exposes .to_arrow(), .to_table(), .to_batches() — all zero-copy against the on-disk pages. Every Arrow-consuming library (Polars, DuckDB, Ray) works transparently.
• SQL engines. DuckDB reads Lance via SELECT * FROM lance_scan('URI') (extension bundled since DuckDB 0.10). Datafusion and Ballista also read Lance natively. This means ad-hoc SQL analysis over a Lance feature store is one CLI call away.
• Distributed compute. Ray Data (ray.data.read_lance), Spark (via a Lance connector), Dask (via dd.read_lance) all shard reads across workers. Lance's fragment structure maps naturally to task parallelism.
• Conversion. lance.write_dataset(pa.read_parquet(...)) is a one-line conversion from Parquet. The reverse (pa.parquet.write_table(ds.to_arrow_table())) is equally trivial. Neither is destructive; the source format remains intact.

Polars — the modern dataframe interop.

• The call. pl.scan_lance(URI) returns a lazy LazyFrame; pl.read_lance(URI) is the eager equivalent.
• Pushdown. Polars pushes filter, projection, and limit down into the Lance reader. .filter(pl.col("country") == "US").select("user_id") reads only the relevant pages of the relevant column.
• Where it wins. Analytical queries with complex expressions that Polars' Rust-based execution optimises; end-of-training feature-diff analysis; ad-hoc exploration.

DuckDB — the SQL-over-Lance interop.

• The call. duckdb.sql("SELECT * FROM lance_scan('URI') WHERE country = 'US' LIMIT 100"). Requires the lance extension (INSTALL lance; LOAD lance;).
• Pushdown. DuckDB's optimiser pushes filters and projections down; the vectorised execution engine reads Lance pages in Arrow format for zero-copy handoff.
• Where it wins. Ad-hoc SQL for data engineers who want warehouse-like syntax over an ML dataset; joining Lance feature-store data with Iceberg warehouse data in one query.

Ray Data — the distributed-training interop.

• The call. ray.data.read_lance(URI, version=N) returns a Ray Dataset that shards fragments across workers.
• Where it wins. Distributed training-shuffle pipelines that need per-epoch reproducibility (pin version=N) and horizontal scale (add workers to increase throughput).
• Common pitfall. Forgetting to pin the version — training runs against a moving dataset are non-reproducible.

Parquet-to-Lance conversion.

• The pattern. lance.write_dataset(pa.dataset("s3://path.parquet", format="parquet").to_batches(), URI).
• Cost. O(rows) one-time write. Streaming batch-by-batch keeps memory flat regardless of dataset size.
• The reverse. pyarrow.parquet.write_table(lance.dataset(URI).to_arrow_table(), "path.parquet") when you need to hand data to a Parquet-only consumer (Snowflake, Athena, BigQuery).

Common interview probes on Lance interop.

• "How do I read Lance from Polars?" — required answer: pl.scan_lance(URI) or pl.read_lance(URI).
• "Can DuckDB query Lance?" — required answer: yes, via the lance extension.
• "How do I migrate from Parquet?" — one-line: lance.write_dataset(pa.dataset(uri, format='parquet').to_batches(), lance_uri).
• "Does Lance work with Ray?" — required answer: yes, ray.data.read_lance shards fragments across workers.

Worked example — Polars + DuckDB + pandas reading the same Lance dataset

Detailed explanation. The canonical interop demo: one Lance dataset, three readers, three different query languages, zero data duplication. Walk through the setup and the queries.

• Dataset. orders.lance with columns (order_id, customer_id, total_cents, country, ordered_at, embedding).
• Queries. Polars aggregate, DuckDB join, pandas time-series slice.
• Goal. Show that Lance is a drop-in for a Parquet dataset under all three readers.

Question. Read the same Lance dataset with Polars, DuckDB, and pandas; run one canonical query per reader.

Input.

Reader	Query language	Query
Polars	LazyFrame expressions	Total by country
DuckDB	SQL	Top 10 customers by spend
pandas	dataframe	Last 7 days of orders

Code.

import lance
import polars as pl
import duckdb
import pandas as pd

DATASET_URI = "orders.lance"

# 1. Polars — lazy scan + aggregate
totals_by_country = (
    pl.scan_lance(DATASET_URI)
      .filter(pl.col("ordered_at") >= pl.date(2026, 1, 1))
      .group_by("country")
      .agg(pl.col("total_cents").sum().alias("total_cents_sum"),
           pl.count().alias("n_orders"))
      .sort("total_cents_sum", descending=True)
      .collect()
)
print("Polars result:")
print(totals_by_country)

# 2. DuckDB — SQL
duckdb.sql("INSTALL lance; LOAD lance;")
top_customers = duckdb.sql(f"""
    SELECT customer_id,
           SUM(total_cents) AS spend_cents,
           COUNT(*)         AS n_orders
    FROM   lance_scan('{DATASET_URI}')
    WHERE  ordered_at >= '2026-01-01'
    GROUP  BY customer_id
    ORDER  BY spend_cents DESC
    LIMIT  10
""").df()
print("DuckDB result:")
print(top_customers)

# 3. pandas — dataframe slice via PyArrow
ds = lance.dataset(DATASET_URI)
last_week = (
    ds.to_table(filter="ordered_at >= '2026-07-12'")
      .to_pandas()
)
print("Pandas result:")
print(last_week.head())

Step-by-step explanation.

1. The Polars pl.scan_lance call opens a LazyFrame — no data is read yet. The subsequent .filter().group_by().agg().sort() chain is fused into a single execution plan that Polars pushes down into Lance: the filter becomes a page predicate, the projection reads only the two relevant columns, the aggregate runs in Polars' Rust engine over the delivered Arrow batches.
2. The DuckDB path uses the lance extension (INSTALL lance; LOAD lance;); lance_scan(URI) is a table-valued function that DuckDB's optimiser treats as an Arrow source. Filter pushdown, projection pushdown, and aggregate execution all happen inside DuckDB, but the I/O is delegated to Lance.
3. The pandas path uses lance.dataset(URI).to_table(filter=...) — Lance returns a filtered Arrow table (only pages matching the filter), and .to_pandas() does a zero-copy Arrow-to-pandas conversion where possible (numeric and string columns are zero-copy; some categoricals require materialisation).
4. All three queries read the same Lance dataset with no data duplication and no format translation. This is the ecosystem-preservation guarantee: adopting Lance does not force you to abandon the query language your team already knows.
5. Performance-wise, all three land within 10-30% of each other for aggregate queries — the bottleneck is I/O, not the reader. For random-access queries (via take_rows), the direct lance.dataset API is the fastest path; Polars and DuckDB add small overhead but preserve pushdown.

Output.

Reader	Query type	Sample result
Polars	aggregate	US: 1.2B cents, IN: 0.8B cents, ...
DuckDB	top-k	customer_id=42_919: 2.1M cents, ...
pandas	time slice	87K rows for the last week

Rule of thumb. For any Lance dataset, pick the reader based on the query shape: aggregate + expressions = Polars; SQL-familiar team = DuckDB; small-result pandas manipulation = direct lance.dataset. All three land within 2× of each other; the choice is style, not performance.

Worked example — Parquet → Lance migration in one script

Detailed explanation. The canonical migration path: read the existing Parquet dataset in batches, write to Lance with tuned fragment settings, verify row count, build the ANN index, cut over consumers. Walk through the whole pipeline.

• Source. 500 GB Parquet dataset on S3 (orders.parquet).
• Target. Lance dataset on S3 (orders.lance).
• Strategy. Streaming batch conversion; no full in-memory load; O(memory-per-batch) footprint.

Question. Write a migration script that converts Parquet to Lance safely, then verifies parity.

Input.

Parameter	Value
Source	s3://data/orders.parquet
Target	s3://data/orders.lance
Batch size	100,000 rows
Fragment size	5,000,000 rows per Lance file
Verification	row count + PK sample equality

Code.

import pyarrow as pa
import pyarrow.dataset as pads
import pyarrow.parquet as pq
import lance

SOURCE_URI = "s3://data/orders.parquet"
TARGET_URI = "s3://data/orders.lance"

# 1. Streaming batch conversion
def migrate():
    src = pads.dataset(SOURCE_URI, format="parquet")
    schema = src.schema

    # to_batches returns a generator; each batch is ~100K rows in Arrow
    lance.write_dataset(
        src.to_batches(batch_size=100_000),
        TARGET_URI,
        schema=schema,
        mode="overwrite",
        max_rows_per_file=5_000_000,
        max_rows_per_group=8_192,
    )
    print("Migration write complete.")

# 2. Verify row count
def verify_count():
    n_parquet = pads.dataset(SOURCE_URI, format="parquet").count_rows()
    n_lance   = lance.dataset(TARGET_URI).count_rows()
    assert n_parquet == n_lance, f"Row count mismatch: {n_parquet} vs {n_lance}"
    print(f"Row count parity: {n_lance:,} rows in both.")

# 3. Verify PK sample equality (spot-check 1000 rows)
def verify_pk_sample():
    import random
    ds_ln = lance.dataset(TARGET_URI)

    src = pads.dataset(SOURCE_URI, format="parquet")
    ids = src.to_table(columns=["order_id"]).to_pandas()["order_id"].sample(1000).tolist()

    pq_rows = (src.to_table(filter=pads.field("order_id").isin(ids))
                   .to_pandas().sort_values("order_id"))
    ln_rows = (ds_ln.to_table(filter=f"order_id IN ({','.join(map(str, ids))})")
                    .to_pandas().sort_values("order_id"))

    pd_equal = (pq_rows.reset_index(drop=True)
                 .equals(ln_rows.reset_index(drop=True)))
    assert pd_equal, "PK sample rows differ"
    print("PK-sample equality verified.")

if __name__ == "__main__":
    migrate()
    verify_count()
    verify_pk_sample()

Step-by-step explanation.

1. pads.dataset(URI, format="parquet") opens the Parquet dataset lazily; src.to_batches(batch_size=100_000) returns a generator that yields 100K-row Arrow record batches. Neither call reads the whole file into memory — the batches stream through.
2. lance.write_dataset(src.to_batches(...), TARGET_URI, mode="overwrite", max_rows_per_file=5_000_000) consumes the generator and writes ~5M-row fragments (~15 GB each for typical schemas). The write is O(rows) time and O(one batch) memory — safe for TB-scale migrations on modest hardware.
3. The row-count verification (count_rows) is O(1) on both formats — both keep row counts in fragment/row-group metadata. This is the cheapest correctness check; run it first.
4. The PK-sample verification takes 1000 random order_ids from Parquet, fetches the corresponding rows from both formats, and confirms row-by-row equality. This catches subtle bugs where a column type coercion silently changed values (e.g. TIMESTAMPTZ precision loss).
5. Post-migration, the Lance dataset is ready for ANN index build (if applicable) and consumer cutover. The Parquet source stays intact for rollback safety until the ML team has run at least one full training cycle against Lance.

Output.

Phase	Duration	Data moved
Batch read + Lance write	~2 h (500 GB, ~30 MB/s throughput)	500 GB
Row-count verification	~1 s	0
PK-sample verification	~5 s	~1 MB
Total migration	~2 h	500 GB

Rule of thumb. Always run the streaming Parquet-to-Lance migration in batches; never load the source file into memory. Keep both formats until you've run at least one production training cycle against Lance — rollback is trivial with both intact.

Worked example — DuckDB joining Iceberg + Lance in one SQL query

Detailed explanation. The hybrid pattern: Iceberg holds the warehouse fact table (fct_orders), Lance holds the derived ML feature store (user_features). A DuckDB query joins them for an ad-hoc analytics question ("what did the top-1000 users by embedding-similarity to a target user buy last month?"). Walk through the query.

• Iceberg source. fct_orders (10B rows) via the iceberg DuckDB extension.
• Lance source. user_features (100M rows, with embedding column) via the lance extension.
• Query. Join on user_id; use Lance for k-NN filter; return purchase details from Iceberg.

Question. Write the DuckDB SQL that joins Iceberg and Lance in one query.

Input.

Source	Format	Reader
fct_orders	Iceberg on Parquet	DuckDB iceberg extension
user_features	Lance	DuckDB lance extension
Join key	user_id	Both formats

Code.

import duckdb
import numpy as np

con = duckdb.connect()
con.sql("INSTALL iceberg; LOAD iceberg;")
con.sql("INSTALL lance;   LOAD lance;")

TARGET_USER_ID = 42_000_000
target_vec_str = "[0.12, 0.34, ...]"   # placeholder — build from lance.take_rows

# 1. Fetch the target user's embedding via Lance
target_vec = con.sql(f"""
    SELECT embedding
    FROM   lance_scan('s3://ml/user_features.lance')
    WHERE  user_id = {TARGET_USER_ID}
""").fetchone()[0]

# 2. Find the top-1000 similar users via Lance + a metadata pre-filter
similar_users_sql = f"""
CREATE OR REPLACE TEMPORARY VIEW similar_users AS
SELECT user_id
FROM   lance_scan('s3://ml/user_features.lance',
                  nearest => struct_pack(column := 'embedding',
                                          q := {target_vec},
                                          k := 1000,
                                          nprobes := 32))
WHERE  country = 'US'
"""

# 3. Join to fct_orders (Iceberg) for last-month purchases
result = con.sql(f"""
    WITH sims AS ({similar_users_sql})
    SELECT s.user_id,
           SUM(o.total_cents) AS spend_cents_last_month,
           COUNT(*)           AS n_orders
    FROM   sims s
    JOIN   iceberg_scan('s3://warehouse/warehouse/fct_orders') o
      ON   o.user_id = s.user_id
    WHERE  o.ordered_at >= (CURRENT_DATE - INTERVAL '30 days')
    GROUP  BY s.user_id
    ORDER  BY spend_cents_last_month DESC
    LIMIT  50
""").df()

print(result)

Step-by-step explanation.

1. Both extensions (iceberg and lance) install into the same DuckDB session. Iceberg's table-valued function is iceberg_scan; Lance's is lance_scan. Both return Arrow-native table sources that DuckDB's optimiser treats as first-class.
2. The k-NN sub-query uses lance_scan(URI, nearest => struct_pack(...)) — the ANN search runs inside Lance and returns the 1000 nearest user_ids. The WHERE country = 'US' clause is pushed down as a pre-filter, so the ANN engine only searches US users.
3. The join to fct_orders (Iceberg) runs in DuckDB's execution engine — vectorised, parallel, with hash join across the 1000 sim_user_ids and the last-month partitions of the fact table. Iceberg's partition pruning restricts the read to the last 30 days.
4. The whole query is one DuckDB SQL — no separate FAISS lookup, no separate S3 partition-listing, no ID matching in Python. The two formats coexist because both are Arrow-native and both extensions push filters down.
5. This is the modern lakehouse-plus-feature-store pattern in one query: Iceberg for warehouse-scale transactional data, Lance for ML-scale random-access data, DuckDB as the ad-hoc SQL layer that glues them.

Output.

user_id	spend_cents_last_month	n_orders
41_982_331	1_240_000	7
42_101_559	895_300	4
...	...	...

Rule of thumb. For any hybrid Iceberg + Lance deployment, use DuckDB (or Datafusion) as the ad-hoc SQL glue layer. Both extensions push filters down; both return Arrow. Complex analytical questions that used to require a two-day pipeline become one interactive SQL query.

Senior interview question on Lance interop

A senior interviewer might ask: "You have a Snowflake warehouse and a Python ML team that just adopted Lance for their feature store. Design a workflow where the ML team's daily feature updates land back in Snowflake for BI dashboards, without either team losing tool independence. Cover the format flow, the CDC pattern, the schema-evolution story, and the ownership boundaries."

Solution Using Lance-native feature store + nightly Parquet export + Iceberg landing for Snowflake

# 1. ML team's Lance feature store — daily append writes
import lance
import pyarrow as pa

LANCE_URI = "s3://ml/user_features.lance"
ICEBERG_URI = "s3://warehouse/warehouse/user_features"  # Iceberg table location

def daily_lance_append(new_batch: pa.Table):
    """ML pipeline appends today's newly-computed features."""
    lance.write_dataset(new_batch, LANCE_URI, mode="append")
    return lance.dataset(LANCE_URI).version

# 2. Nightly export: Lance → Parquet → Iceberg landing
import pyarrow.parquet as pq
import datetime as dt

def export_to_iceberg_landing(version: int):
    """Read pinned Lance version, write Parquet to Iceberg landing zone."""
    ds = lance.dataset(LANCE_URI, version=version)
    today = dt.date.today()

    # Filter to today's newly-appended rows (feature_date column)
    todays_features = ds.to_table(
        filter=f"feature_date = '{today.isoformat()}'"
    )

    # Write as Parquet into the Iceberg landing folder
    partition_path = f"{ICEBERG_URI}/landing/dt={today:%Y-%m-%d}/"
    pq.write_table(todays_features,
                    f"{partition_path}part-000.parquet",
                    compression="snappy")
    return partition_path

-- 3. Snowflake external table refresh (Iceberg-backed)
ALTER ICEBERG TABLE analytics.user_features REFRESH;

-- 4. BI dashboard query (Snowflake side)
SELECT country,
       age_bucket,
       COUNT(*) AS users,
       AVG(engagement_score) AS avg_engagement
FROM   analytics.user_features
WHERE  feature_date >= CURRENT_DATE - 7
GROUP  BY country, age_bucket;

# 5. Schema-evolution safety — declare a schema contract
FEATURE_SCHEMA = pa.schema([
    ("user_id",       pa.int64()),
    ("feature_date",  pa.date32()),
    ("country",       pa.string()),
    ("age_bucket",    pa.int32()),
    ("engagement_score", pa.float32()),
    ("embedding",     pa.list_(pa.float32(), 768)),
])

def guarded_append(new_batch: pa.Table):
    # Reject writes that don't conform to the contract
    for field in FEATURE_SCHEMA:
        if field.name not in new_batch.column_names:
            raise ValueError(f"missing column {field.name}")
        if new_batch.field(field.name).type != field.type:
            raise TypeError(f"type mismatch for {field.name}")
    return daily_lance_append(new_batch)

Step-by-step trace.

Layer	Component	Owner	Purpose
ML feature store	Lance @ s3://ml/user_features.lance	ML team	random-access + vector + versions
Version pinning	lance.dataset(URI, version=N)	ML + BI	reproducibility
Nightly export	Lance → Parquet in Iceberg landing	data-engineering	glue layer
Iceberg table	analytics.user_features	data-engineering	warehouse contract
Snowflake refresh	ALTER ICEBERG TABLE REFRESH	Snowflake admin	dashboard consumers
Schema contract	FEATURE_SCHEMA Arrow schema	shared	evolution safety
BI dashboards	Snowflake SQL	analytics team	user-facing

After deployment, the ML team writes to Lance daily without any Snowflake dependency; the data-engineering team runs a nightly export to Iceberg landing; Snowflake refreshes and dashboards see the same features one day behind ML; version pinning at the Lance layer means reproducibility experiments still work; schema-evolution safety at the contract boundary prevents the ML team from accidentally breaking downstream dashboards.

Output:

Aspect	Value
ML write path	Lance append (~1-2 min per daily batch)
BI freshness	T+1 day (nightly export)
BI freshness (optional real-time)	Lance-to-Iceberg CDC via Debezium — sub-hour
Reproducibility	pin version=N at ML read side
Ownership	ML → Lance; DE → export; BI → Snowflake
Failure isolation	ML errors don't reach BI; DE errors don't reach ML

Why this works — concept by concept:

• Lance for the ML side, Iceberg-on-Parquet for the BI side — each format serves its native workload. Lance wins on random-access + vector; Iceberg wins on transactional multi-writer + Snowflake compatibility. Nobody has to choose one format for everything.
• Version pinning at the Lance layer — the nightly export reads lance.dataset(URI, version=N), so the exported Parquet is a bit-exact snapshot of the ML dataset at that version. Reproducibility experiments can rerun by pinning the same version.
• Nightly Parquet export + Iceberg landing — the export is a boring, well-understood Parquet write into an Iceberg landing zone. Snowflake reads the Iceberg table via its external-table integration. No Lance-specific plumbing in the warehouse.
• Schema contract at the boundary — the FEATURE_SCHEMA Arrow schema is the shared contract. ML writes conforming to the schema; DE exports the schema-conforming subset; BI queries against the schema-conforming Iceberg columns. Changes to the schema require agreement across all three teams.
• Cost — one Lance dataset, one nightly export job (~10 min for a 30 GB daily append), one Iceberg refresh, no Elasticsearch, no FAISS. Compared to a Parquet-only pipeline (loses random access + vector) or a Lance-everywhere pipeline (loses Snowflake compat), this is the pragmatic hybrid. O(delta) per day for both the ML append and the BI export.

SQL
Topic — joins
SQL joins and multi-source query problems

Practice →

Design Topic — design Design problems on lakehouse + feature-store hybrids

Practice →

---

5. Production patterns + interview signals

Lance vs Parquet vs Iceberg — the decision tree, the LanceDB angle, and the interview signals senior engineers listen for

The mental model in one line: Lance is the answer when random-access reads, in-file vector indices, or zero-copy versions dominate your workload; Parquet remains the answer for OLAP scan-throughput; Iceberg is the table-format overlay for transactional multi-writer warehouses; and the modern lakehouse-plus-feature-store architecture uses all three simultaneously — Iceberg for warehouse fact tables, Parquet as the underlying columnar file format for Iceberg, Lance for derived ML datasets — with clear ownership boundaries and well-understood conversion paths between them. Every senior storage-format interview eventually converges on this three-format decision tree; the candidates who can walk it out loud in 60 seconds are the ones who get the offer.

The four axes for the format decision.

• Read pattern. Random-access (Lance) vs OLAP scan (Parquet/Iceberg) is the primary axis. If >20% of your queries are take(single_id) or k-NN, Lance wins; if >80% are SELECT ... GROUP BY partition_key, Parquet-on-Iceberg wins.
• Multi-writer semantics. ACID multi-writer (Iceberg overlay) vs append-only (Lance) is the transactional axis. Iceberg's snapshot isolation and CAS commits solve the multi-writer race; Lance's append-only design assumes one writer at a time per dataset.
• Vector storage. Native ANN index (Lance) vs external FAISS blob (Parquet + FAISS) is the ML-workload axis. If embeddings are a first-class concern, Lance's in-file index eliminates an entire consistency problem.
• Ecosystem breadth. Universal (Parquet — every engine reads it) vs specialised (Lance — great in Arrow ecosystem, weaker in JVM-only environments) is the compatibility axis. For BI-heavy shops with Snowflake / Redshift / BigQuery as the primary reader, Parquet on Iceberg is the safe default.

LanceDB — the vector database on top of Lance format.

• What it is. A vector database (SDK + optional server) that uses Lance format as native storage. Provides an ORM-like API for embedding-heavy workloads: db.create_table, table.add, table.search, hybrid full-text + vector.
• When to pick LanceDB vs raw Lance. LanceDB when you want a vector-DB abstraction with authentication, multi-tenancy, and a managed service option. Raw Lance when you want direct control over the file layout, are integrating with Ray / Polars / DuckDB, or need custom index tuning.
• Managed offering. LanceDB Cloud is the hosted service; open-source LanceDB runs against S3 / GCS / Azure blob storage. The format is the same; only the API layer differs.

Interview signals — what the senior interviewer listens for.

• Naming the axes. "Random access, vector index, version, ecosystem" — say all four in the first minute.
• Concrete throughput numbers. "Parquet random-access shuffle: ~200 rows/sec; Lance: ~100K rows/sec; that's a 500× gap." Numbers separate candidates who read the blog from candidates who ran the benchmark.
• Naming the hybrid. "Iceberg + Lance is the modern pattern — Iceberg for warehouse, Lance for the ML feature store, DuckDB or Snowflake as the ad-hoc SQL layer." Naming the pattern before being asked is senior signal.
• Failure modes. "Multi-writer to a Lance dataset causes manifest conflicts; use a single-writer coordinator or migrate to Iceberg-on-Lance if that emerges as a supported hybrid." Knowing the failure modes is what separates production engineers from tutorial completers.
• Migration story. "Migrations are one-line: lance.write_dataset(pa.read_parquet(...)); keep the Parquet source until one full training cycle has passed against Lance." Concrete rollback plans are senior signal.

Production runbook essentials.

• Monitoring. Track fragment count (compaction trigger at >100), index rebuild age (rebuild trigger at >30 days), version count (GC trigger at >90), storage growth vs data growth (should be ~1.1× for a well-managed dataset).
• Compaction. Nightly dataset.optimize.compact_files(target_rows_per_fragment=10M) merges small append fragments. optimize_indices() incrementally rebuilds the ANN index for new pages.
• GC. cleanup_old_versions(older_than="30 days") after experiments settle. Skipping this is safe (only costs storage); skipping forever inflates the S3 bill.
• Writer coordination. Lance's append-only semantics assume one writer per dataset. Use an orchestrator (Airflow, Prefect) to enforce a single-writer lease; parallel writes cause manifest conflicts.
• Reader concurrency. Unlimited. All readers see a manifest-consistent view; no read locks; ideal for high-concurrency inference services.

Worked example — the format decision tree walked out loud

Detailed explanation. The canonical interview move: hand the interviewer the four-axis decision tree in prose form, then apply it to whatever scenario they give you. Walk through the tree for three scenarios: a recommender feature store, an ML training-example lake, and a warehouse fact table.

• Scenario 1. Recommender team, 2B user embeddings, needs k-NN + metadata pre-filter at 5 ms p99 online + 100K rows/sec training shuffle.
• Scenario 2. ML training-example lake, 500M examples, needs random shuffle + version reproducibility, no vector search needed.
• Scenario 3. Warehouse fact table, 10B rows, needs OLAP scan with multi-writer ETL and Snowflake compatibility.

Question. Walk the decision tree for each scenario and state the format + rationale.

Input.

Scenario	Random access?	Vector?	Multi-writer?	Snowflake?
Recommender feature store	yes (shuffle + serving)	yes (k-NN)	no (single ML writer)	no (ML-only)
Training-example lake	yes (shuffle)	no	no (single writer)	no
Warehouse fact table	no (scans)	no	yes (multi-source ETL)	yes

Code.

def format_recommender():
    return {
        "primary_format":   "Lance",
        "index":            "IVF-HNSW-SQ, num_partitions=2048, m=32",
        "versions":         "enabled, 30-day retention",
        "reason":           "random access + k-NN + pre-filter dominate; "
                            "no multi-writer or Snowflake need",
    }

def format_training_lake():
    return {
        "primary_format":   "Lance",
        "index":            "none (no vector search)",
        "versions":         "enabled, 90-day retention (long experiment window)",
        "reason":           "random shuffle at 100K rows/sec + reproducibility; "
                            "no vector index needed",
    }

def format_warehouse_fact():
    return {
        "primary_format":   "Iceberg on Parquet",
        "index":            "n/a",
        "versions":         "Iceberg snapshots (ACID)",
        "reason":           "OLAP scans dominate; multi-writer ETL requires "
                            "ACID; Snowflake reads Iceberg natively",
    }

Step-by-step explanation.

1. Scenario 1 short-circuits on the random-access + vector axis: both are required, so Lance is the primary format. The IVF-HNSW-SQ index at 2048 partitions is sqrt(2B) rounded down to a memory-friendly value. Versions at 30 days match the recommender's A/B-test rollback window.
2. Scenario 2 needs random access without vectors — Lance without the ANN index. Versions at 90 days for long training experiments. This is the "Lance-because-of-shuffle-throughput" scenario; the training loop's random_shuffle at 100K rows/sec is the load-bearing requirement.
3. Scenario 3 is the opposite of Lance's strengths: OLAP scans, multi-writer ETL, Snowflake compatibility. Iceberg on Parquet wins on all three axes. Nobody would migrate this to Lance — the workload doesn't need what Lance offers, and Snowflake integration is a hard requirement.
4. Walking the tree out loud is the interview signal. "Random access? Yes → Lance. Vector? Yes → IVF-HNSW-SQ. Multi-writer? No → single-writer OK. Snowflake? No → we don't need Parquet." Each answer takes 10 seconds; the whole tree is under a minute.
5. The tree's failure mode is trying to force one format to serve all three scenarios. Teams that do so end up either (a) with terrible ML training-shuffle throughput on a Parquet-only stack or (b) with a Lance-only stack that can't feed BI dashboards without a dedicated export pipeline. The hybrid is the right answer for large orgs.

Output.

Scenario	Format	Rationale
Recommender feature store	Lance + IVF-HNSW-SQ	random access + vector + pre-filter
Training-example lake	Lance (no index)	random shuffle + reproducibility
Warehouse fact table	Iceberg on Parquet	OLAP scans + ACID + Snowflake

Rule of thumb. Practice walking the decision tree with any hypothetical scenario an interviewer throws at you. The tree makes the answer reproducible; the rationale is what convinces the interviewer you've actually built one.

Worked example — production runbook for a live Lance dataset

Detailed explanation. Beyond the format-choice question, senior interviews probe the operational story. What do you monitor? When do you compact? How do you rebuild the index? What do you do when a writer crashes mid-write? Walk through the runbook.

• Monitoring. Fragment count, index age, version count, storage growth.
• Compaction. Nightly job that consolidates small append fragments.
• Index rebuild. Incremental on append; full rebuild every ~30 days.
• Failure recovery. Writer crash → orphaned uncommitted fragment; safe to ignore (manifest never referenced it).

Question. Write the monitoring queries, the compaction job, and the failure-recovery runbook.

Input.

Concern	Signal	Threshold	Action
Fragment count	len(dataset.get_fragments())	> 100	run compaction
Index age	days since create_index	> 30	full rebuild
Version count	len(dataset.versions())	> 100	run GC
Storage growth	(storage bytes) / (data bytes)	> 2.0	investigate + GC

Code.

import lance
import datetime as dt

DATASET_URI = "s3://ml/user_features.lance"

# 1. Monitoring — daily scrape
def scrape_health():
    ds = lance.dataset(DATASET_URI)
    frags       = ds.get_fragments()
    versions    = ds.versions()
    n_rows      = ds.count_rows()
    storage_gb  = sum(f.count_rows() * 6_000 for f in frags) / 1e9  # rough

    return {
        "n_fragments":  len(frags),
        "n_versions":   len(versions),
        "n_rows":       n_rows,
        "storage_gb":   storage_gb,
        "current_ver":  ds.version,
    }

# 2. Compaction (nightly)
def nightly_compaction():
    ds = lance.dataset(DATASET_URI)
    ds.optimize.compact_files(
        target_rows_per_fragment=10_000_000,
    )
    ds.optimize.optimize_indices()   # incremental
    print("Compaction done.")

# 3. Full index rebuild (monthly)
def monthly_index_rebuild():
    ds = lance.dataset(DATASET_URI)
    ds.drop_index("embedding_idx")  # or list_indices to get the name
    ds.create_index(
        column="embedding",
        index_type="IVF_HNSW_SQ",
        num_partitions=2048,
        num_sub_vectors=96,
        ef_construction=200,
        m=32,
    )

# 4. Version GC (weekly)
def weekly_gc():
    ds = lance.dataset(DATASET_URI)
    ds.cleanup_old_versions(older_than="30 days")

# 5. Failure recovery — writer crashed mid-write
def check_orphaned_fragments():
    """Uncommitted fragments are never referenced by a manifest; safe to leave.
       For belt-and-braces cleanup, list files in the fragment directory and
       delete any without a corresponding manifest entry."""
    ds = lance.dataset(DATASET_URI)
    live_fragment_ids = {f.fragment_id for f in ds.get_fragments()}
    # (implementation: list files, subtract live_fragment_ids, delete rest)
    return live_fragment_ids

Step-by-step explanation.

1. The health scrape reports the four load-bearing metrics: fragment count (fragmentation), version count (GC pressure), row count (business growth), storage GB (cost). Prometheus + Grafana renders these; alerts fire on threshold crossings.
2. Nightly compaction merges small append fragments into 10M-row targets. It runs against the live dataset without blocking readers or writers (manifest-based; new fragments replace old ones atomically). Duration is proportional to the number of small fragments, typically 10-30 min.
3. The index doesn't need a full rebuild on every append — optimize_indices() is incremental, patching the ANN structure for new pages. A monthly full rebuild (drop + recreate) is a good hygiene practice for index quality drift, especially if data distribution shifts over time.
4. Version GC runs weekly; cleanup_old_versions(older_than="30 days") drops manifests older than 30 days and reclaims storage from fragments no longer referenced. Skip this and storage grows unbounded; run it too aggressively and reproducibility experiments break.
5. Writer crashes mid-write are safe by design: Lance's write path stages a new fragment file, then atomically writes the new manifest. If the writer crashes before the manifest write, the orphaned fragment is never referenced — readers never see it. A monthly check_orphaned_fragments script sweeps them up as belt-and-braces cleanup.

Output.

Runbook event	Frequency	Duration	Impact
Health scrape	daily	seconds	none
Compaction	nightly	10-30 min	none (background)
Index rebuild (full)	monthly	1-6 h	none (background)
Version GC	weekly	minutes	reclaims storage
Orphan sweep	monthly	minutes	cost hygiene

Rule of thumb. For any production Lance deployment, wire the four monitoring signals (fragments, versions, rows, storage) into Prometheus on day one. Schedule the four maintenance jobs (compaction nightly, index monthly, GC weekly, orphan sweep monthly). Skipping any of the four is fine for a week; skipping any for a quarter causes a Sev-2.

Worked example — the "convince me to migrate from Parquet" interview

Detailed explanation. A common senior-round question: "we already use Parquet everywhere; convince me Lance is worth adopting for the ML team." The strong answer names concrete workloads Parquet handles badly, cites concrete throughput numbers, names the migration cost, and names the rollback plan. Walk through the answer.

• Workload 1. Training-shuffle throughput: Parquet ~200 rows/sec, Lance ~100K rows/sec. GPU utilisation goes from 15% to 90%.
• Workload 2. Online embedding lookup: Parquet ~200 ms per user (rebuild + FAISS query), Lance ~5 ms per user (one call).
• Workload 3. Training reproducibility: Parquet has none; Lance has native versions.
• Migration cost. ~2 engineer-weeks (write conversion script + validate + cut over consumers).
• Rollback plan. Keep the Parquet source intact for 30 days; if any consumer breaks, cut back.

Question. Draft the 3-minute answer to "convince me to migrate."

Input.

Signal	Weak answer	Senior answer
Concrete workloads	"faster"	"training shuffle: 200 rows/sec → 100K rows/sec"
Migration cost	"not much work"	"~2 engineer-weeks; one-line write_dataset script + validation"
Rollback plan	"we'll figure it out"	"Parquet source stays for 30 days; consumers can cut back atomically"
Non-goals	none named	"warehouse fact tables stay on Iceberg-on-Parquet — this is ML-specific"

Code.

3-minute "convince me" answer template
=======================================

Opening (30s)
  "Migrate only the ML feature store and the training-example lake to
   Lance. Keep warehouse fact tables on Iceberg-on-Parquet. The migration
   is scoped, reversible, and pays back within one training cycle."

Concrete workloads (60s)
  "Three workloads Parquet handles badly:
   1. Training-shuffle throughput: Parquet ~200 rows/sec because each
      random row costs a 128 MB row-group read. Lance ~100K rows/sec
      because pages are ~8 KB. GPU utilisation goes from 15% to 90%.
   2. Online embedding lookup: Parquet + FAISS is ~200 ms per user
      (index join + row-group scan). Lance with in-file HNSW is ~5 ms.
   3. Reproducibility: Parquet has none. Lance has native versions;
      ds.checkout(version=N) reruns any prior training."

Cost + rollback (60s)
  "The migration is ~2 engineer-weeks:
   - Week 1: one-line Parquet-to-Lance conversion script; validate row-
     count + PK-sample parity.
   - Week 2: build the ANN index (~6h for 2B rows on 4 CPUs); rewrite
     the training-shuffle reader (~3 lines: ray.data.read_lance);
     rewrite the online-inference reader (~5 lines: ds.take_rows).
   Rollback: keep the Parquet source intact for 30 days. If any
   consumer regresses, cut readers back to Parquet in one config change."

Non-goals (30s)
  "Explicit non-goals: no migration for warehouse fact tables (stay on
   Iceberg-on-Parquet), no migration for tables under multi-writer ETL
   (Lance is append-only), no migration for tables read by Snowflake
   without an Iceberg copy alongside."

Step-by-step explanation.

1. The opening scopes the migration: ML feature store + training-example lake, not the whole warehouse. Scoping is the single most important credibility move — proposing "migrate everything" reads as inexperienced.
2. The three concrete workloads section names specific numbers (200 → 100K rows/sec) and business outcomes (GPU utilisation up 6×). This is what separates candidates who've measured from candidates who've theorised.
3. The cost + rollback section shows engineering maturity. Naming the ~2-week estimate + the 30-day rollback window signals "I've done this before" — the interviewer knows migrations always slip a week or two, so the honest estimate lands better than an optimistic one.
4. The non-goals section shows discipline. Naming what you won't migrate is as important as naming what you will. This preempts the "you'll break the warehouse team" pushback that senior interviewers deploy.
5. The whole answer is 3 minutes and covers pattern-choice, throughput, cost, rollback, and non-goals. Weak candidates spend 3 minutes on "Lance is faster"; senior candidates spend 3 minutes on the migration plan.

Output.

Section	Duration	Signal
Opening (scope)	30s	scoping discipline
Concrete workloads	60s	measured, not theorised
Cost + rollback	60s	engineering maturity
Non-goals	30s	organisational awareness
Total	3 min	senior signal

Rule of thumb. For any "convince me to migrate" interview, always name the scope (what's in / what's out), the concrete workloads (with numbers), the migration cost (with a realistic estimate), and the rollback plan. Skipping any of the four signals inexperience.

Senior interview question on production Lance patterns

A senior interviewer might ask: "You've deployed Lance for the recommender feature store six months ago. The ML team is happy. Now the search team wants to add semantic search over 50M documents using Lance's BM25 + vector hybrid, the platform team wants to consolidate onto one storage stack, and the warehouse team is asking if Iceberg should be migrated to Lance too. Design the multi-year architecture and defend where you draw the line."

Solution Using scoped Lance adoption + Iceberg-Lance hybrid + LanceDB for the semantic-search API

# 1. The scoped architecture — three tables, three formats, one bucket
"""
s3://data/
├── warehouse/                       ← Iceberg-on-Parquet (do NOT migrate)
│   ├── fct_orders/
│   ├── dim_users/
│   └── fct_events/
├── ml-features/                     ← Lance (already migrated)
│   └── user_features.lance/
├── ml-training/                     ← Lance (already migrated)
│   └── training_examples.lance/
└── search/                          ← Lance (new; via LanceDB API)
    └── docs.lance/
"""

# 2. Nightly Iceberg → Lance CDC for the search corpus
import lancedb
import pyarrow as pa

DB = lancedb.connect("s3://data/search/")

def rebuild_docs_from_iceberg():
    # (Iceberg read via duckdb / pyiceberg)
    import duckdb
    con = duckdb.connect()
    con.sql("INSTALL iceberg; LOAD iceberg;")
    tbl = con.sql("""
        SELECT doc_id, product_line, language, title, body_text, embedding
        FROM   iceberg_scan('s3://data/warehouse/dim_documents')
    """).arrow()

    # LanceDB upsert
    if "docs" in DB.table_names():
        DB.drop_table("docs")
    docs = DB.create_table("docs", tbl)
    docs.create_index(vector_column_name="embedding",
                      index_type="IVF_HNSW_SQ",
                      num_partitions=512, num_sub_vectors=96)
    docs.create_fts_index(["body_text"])
    return docs.version

# 3. Semantic-search API endpoint (LanceDB API surface)
def semantic_search_api(query_text: str, query_vec, product_line: str, k: int = 20):
    docs = DB.open_table("docs")
    return (docs.search(query_vec, vector_column_name="embedding")
                 .where(f"product_line = '{product_line}'", prefilter=True)
                 .limit(k)
                 .to_pandas())

# 4. Where we DO NOT migrate — the warehouse fact tables
"""
Warehouse fact tables (fct_orders, fct_events, dim_users):
  - Format: Iceberg-on-Parquet
  - Reasons NOT to migrate:
    - Multi-writer ETL: Airflow + DBT + Fivetran all write; Lance is single-writer
    - Snowflake / Athena consumers rely on Iceberg's REST catalog
    - Query pattern is 95% OLAP scans, 5% point lookups — Parquet's row groups
      are the right unit for the scans
    - No embeddings in these tables — Lance's vector-index advantage is n/a
  - Long-term stance: revisit if Lance ever supports multi-writer + REST catalog
"""

Step-by-step trace.

Table	Format	Owner	Reason
fct_orders / fct_events / dim_users	Iceberg-on-Parquet	data-warehouse team	multi-writer ACID + Snowflake
user_features.lance	Lance + IVF-HNSW-SQ	ML platform team	random access + vector + versions
training_examples.lance	Lance (no index)	ML training team	random shuffle + reproducibility
docs.lance (via LanceDB)	Lance + IVF-HNSW-SQ + BM25	search team	hybrid vector + full-text
Iceberg → Lance rebuild	nightly	search / DE shared	corpus refresh

After a year of the multi-year plan, the org runs three formats (Iceberg-on-Parquet for warehouse, Lance for ML, Lance-via-LanceDB for search) with clear ownership boundaries, nightly conversions where needed, and no format-choice arguments in team meetings. The warehouse team's Snowflake stack is untouched. The ML team's training-shuffle throughput stays at 100K rows/sec. The search team's semantic-search API lands at ~50 ms p99. Nobody is proposing to migrate everything to one format; nobody needs to.

Output:

Team	Format	Reader	Freshness	Notes
Warehouse	Iceberg-on-Parquet	Snowflake, Athena, DuckDB	T+1h ETL	multi-writer OK
ML feature store	Lance	Ray, Polars, direct API	daily append	single writer
ML training	Lance	Ray Data	pinned version	reproducible
Semantic search	Lance via LanceDB	LanceDB API	nightly rebuild	hybrid queries
Iceberg → Lance CDC	Parquet export	Lance write_dataset	nightly	glue layer

Why this works — concept by concept:

• Format per workload — Iceberg for transactional warehouse, Lance for ML random-access, LanceDB for the semantic-search API layer. Each format wins its native workload without compromising the others. This is the honest answer to "consolidate onto one format" — sometimes the right answer is "no."
• Iceberg → Lance nightly rebuild — the search corpus starts life as an Iceberg dim table; a nightly job reads it and writes a fresh Lance dataset with vector + BM25 indices. The two formats stay loosely coupled via the DuckDB / pyiceberg boundary.
• LanceDB as the API layer, Lance as the format — LanceDB gives the search team an ORM-like abstraction; the underlying storage is standard Lance. This keeps the option open to swap the API layer later without a data-format migration.
• Explicit non-migration for warehouse fact tables — naming what you won't migrate is as important as naming what you will. Multi-writer ETL and Snowflake integration are hard requirements Lance doesn't yet satisfy; forcing Lance here would break the warehouse team.
• Cost — three formats, three teams, one S3 bucket, ~5% storage overhead from indices, nightly rebuild jobs (~1h each). Compared to a "one format for everything" proposal (would break either the warehouse or the ML team), this is the pragmatic multi-year architecture. Ownership is clean, migration is scoped, rollback is per-table. O(delta) per day for every writer.

Streaming
Topic — streaming
Streaming and CDC problems for feature stores

Practice →

Design
Topic — design
Design problems on multi-format lakehouses

Practice →

---

Cheat sheet — Lance format recipes

• Which format when. Lance is the 2026 default when your workload is random-access heavy (>1K rows/sec of take(random_ids)), when embeddings need an in-file ANN index, when zero-copy version snapshots matter for reproducibility, or when Ray Data / Polars / DuckDB are your primary readers. Parquet-on-Iceberg remains the default for OLAP scan-throughput warehouse fact tables, multi-writer ETL, and Snowflake / BigQuery / Athena compatibility. Never migrate a table to Lance solely because Lance is trendy; migrate because a concrete workload (training-shuffle throughput, k-NN with metadata, reproducibility) demands what Lance offers and Parquet can't.
• Lance dataset creation template. lance.write_dataset(arrow_batches, uri, mode="overwrite", max_rows_per_file=10_000_000, max_rows_per_group=4_096). Tune max_rows_per_file upward (10M-100M rows) for fewer, larger fragments — better parallel-scan throughput, fewer tiny-file problems. Tune max_rows_per_group (the page-size proxy) downward (~4K-8K rows) for tighter random-access — smaller page reads, more granular pushdown. Always sort at write time by the columns you plan to filter on; Lance's per-page dictionary encoding gives you 2-5× pruning for free after a sort.
• HNSW / IVF-PQ / IVF-HNSW-SQ index config. ds.create_index(column="embedding", index_type="IVF_HNSW_SQ", num_partitions=sqrt(N)_capped_at_2048, num_sub_vectors=embedding_dim/8, ef_construction=200, m=16). num_partitions follows the sqrt(N) rule capped at 2048 for memory-friendly builds; num_sub_vectors implements 8-bit scalar quantization for ~4× memory reduction; m (HNSW graph degree) is 16 for balanced recall, 32-64 for high-recall recommenders. IVF_FLAT is the exact-within-partition baseline; IVF_PQ is the aggressive compression choice; IVF_HNSW_SQ is the modern hybrid that most production deployments land on.
• Vector + metadata pre-filter query pattern. ds.to_table(nearest={"column": "embedding", "q": vec, "k": 50, "nprobes": 20, "refine_factor": 10}, filter="country = 'US' AND age_bucket = 3", columns=["user_id", "country"]). The filter= clause always runs before the ANN search (pre-filter) when both are passed together — this guarantees exactly k results when the filter matches ≥ k rows. For heavily-filtered queries increase nprobes (~2× to 3× of default) to compensate for the reduced candidate space; for recall-critical queries increase refine_factor (10-20 typical, up to 50 for high-recall).
• Parquet → Lance conversion snippet. lance.write_dataset(pa.dataset("s3://path.parquet", format="parquet").to_batches(batch_size=100_000), "s3://path.lance", schema=src.schema, mode="overwrite", max_rows_per_file=5_000_000). This is O(rows) time and O(one batch) memory — safe for TB-scale migrations. Verify parity via count_rows() (O(1) on both formats) and a PK-sample equality check on 1000 random rows. Keep the Parquet source intact for 30 days after cutover; rollback is a one-line reader-config change if any downstream regresses.
• Time-travel / version snapshot recipe. ds = lance.dataset(URI, version=N) opens a reader pinned to version N; ds.checkout(version=N) is the equivalent method call on a live handle. Version numbers are monotonic; ds.versions() returns the full list with timestamps; ds.list_versions() gives detailed metadata. Every write_dataset(mode="append") creates a new version — zero data duplication until cleanup_old_versions(older_than="30 days") runs. Pair versioning with a reproducibility contract: training runs record the version they used, and checkout(version=N) reruns them bit-exact.
• Ray Data + Lance shuffle template. ray.data.read_lance(uri, version=N).random_shuffle(seed=42).iter_batches(batch_size=4096, format="numpy"). Pin version=N for reproducibility; the shuffle uses Lance's take_rows under the hood for O(page) per-row cost. Throughput scales linearly with Ray worker count up to the fragment count of the dataset; a 100-fragment dataset saturates at ~100 workers. Never mix Ray Data shuffle with a Parquet source in the same training loop and expect the same throughput — Parquet's row-group scans cap the pipeline at ~1-5K rows/sec per worker.
• Compaction + retention policy. Nightly: ds.optimize.compact_files(target_rows_per_fragment=10_000_000) merges small append fragments; ds.optimize.optimize_indices() incrementally rebuilds the ANN index for new pages. Weekly: ds.cleanup_old_versions(older_than="30 days") reclaims storage. Monthly: drop_index + create_index for full ANN rebuild if data distribution has drifted. Every maintenance job runs against the live dataset without blocking readers or writers — Lance's manifest model makes this safe by construction.
• Lance vs Parquet vs Iceberg decision matrix. Read pattern: Lance for random-access + k-NN, Parquet for OLAP scans, Iceberg (on Parquet) for transactional warehouse. Multi-writer: Lance is single-writer per dataset; Iceberg supports concurrent writers via optimistic CAS commits. Vector index: Lance native (IVF-HNSW-SQ), Parquet requires external FAISS. Versioning: Lance zero-copy first-class, Iceberg snapshot-based (also zero-copy), Parquet none native. Ecosystem: Parquet universal; Lance strong in Arrow ecosystem (Polars/DuckDB/Ray); Iceberg in warehouse ecosystem (Snowflake/Athena/Databricks). The three coexist in the modern lakehouse; nobody migrates everything to one.
• Failure semantics reminder. Lance writer crashed mid-write → orphaned fragment file, never referenced by manifest, readers see nothing wrong. Concurrent writers → manifest conflict; use an orchestrator (Airflow, Prefect) to enforce single-writer lease. S3 eventual consistency → Lance uses S3 conditional writes (If-None-Match) for manifest atomicity; safe on modern S3 (strong consistency since 2020). Index build interrupted → next create_index picks up where the previous one left off; incremental by design.
• Schema evolution rules. Adding a column: Lance zero-copy — new column stored in new pages, old pages implicitly NULL for that column, no data rewrite. Dropping a column: metadata-only — the column disappears from queries but pages remain unchanged (GC on next compaction). Type change: not directly supported — write a new column with the new type, drop the old one, or migrate to a new dataset. Compared to Parquet's schema-evolution story (SchemaResolver at read time), Lance's is simpler because the format tracks schema per-version natively.
• When to use LanceDB vs raw Lance. LanceDB when you want a vector-DB abstraction (ORM-like table API, authentication, multi-tenancy, managed cloud option). Raw Lance when you want direct control over file layout, are wiring Lance into an existing Ray / Polars / DuckDB pipeline, or need custom index tuning beyond what LanceDB exposes. Both use the same Lance format on disk — the choice is API surface, not storage engine. Migration between the two is a config change; no data movement.
• Interview signal checklist. Name the four axes (random access, vector, versions, ecosystem) unprompted. Cite concrete throughput numbers (Parquet ~200 rows/sec vs Lance ~100K rows/sec on shuffle). Name the hybrid pattern (Iceberg + Lance + LanceDB). Name the LanceDB-as-API layer. Name at least one failure mode (multi-writer manifest conflict, orphaned fragments). Name at least one migration risk (schema-evolution gotchas, consumer regression window). Name the compaction and GC schedule (nightly compact, weekly GC, monthly index rebuild). Every senior Lance interview probes at least six of these seven.
• Migration cost between patterns. Parquet → Lance: ~2 engineer-weeks per major dataset (streaming write + validate + reader changes). FAISS + Parquet → Lance in-file index: ~1 sprint (drop FAISS, rebuild in-file, rewrite k-NN queries). TFRecord → Lance: ~2-3 sprints (parse TFRecord + write Lance + rewrite training reader; larger because TFRecord bundles carry serialisation quirks). Iceberg → Lance (rare, usually wrong direction): only for ML-specific tables that outgrew Iceberg's serving latency; ~1 month per dataset because Iceberg consumers must be re-pointed. Choose the primary format once; the migration cost is real.

Frequently asked questions

What is Lance format in one sentence?

lance format is a modern columnar file format built specifically for ML-adjacent workloads — random-access reads at high throughput, in-file ANN vector indices (IVF-PQ, HNSW), zero-copy version snapshots, and first-class Arrow interop — trading a small OLAP scan-throughput penalty for a step-change improvement in per-row seek cost that Parquet was never designed to deliver. The format lives on top of Apache Arrow's in-memory representation, uses page-based storage with ~8 KB pages instead of Parquet's 128 MB row groups, ships native indices (IVF-Flat, IVF-PQ, IVF-HNSW-SQ, BM25, BITMAP) inside the dataset manifest, and supports zero-copy time-travel via versioned manifests. It's the storage backing LanceDB (a modern vector database), the recommended format for Ray Data training-shuffle pipelines, and the emerging default for ML feature stores in modern lakehouse architectures. Every senior ML-platform interview probes Lance because it's the format-choice question that decides whether your training loop is GPU-bound or I/O-bound.

Lance vs Parquet — when do I pick each?

Default to Lance when random-access reads dominate — training-shuffle at >1K rows/sec, online embedding lookup at <10 ms p99, k-NN queries with metadata pre-filter, or reproducibility experiments needing version pinning. Lance's ~8 KB page-based layout makes take(random_indices) cost O(page-size) per row instead of Parquet's O(row-group-size); on 128 MB row-groups this is a ~15,000× reduction in bytes-read per random row, which translates to a ~500× throughput improvement on training shuffle. Default to Parquet (usually as the file format underneath Iceberg) when OLAP scans dominate — SELECT ... GROUP BY partition_key over 500 GB of warehouse fact data, multi-writer ETL from Airflow + DBT + Fivetran, or read paths through Snowflake / BigQuery / Athena that rely on Parquet's ubiquity. Nobody serious is migrating warehouse fact tables to Lance in 2026; nobody serious is training a recommender on Parquet-only either. The modern lakehouse uses both: Iceberg-on-Parquet for the warehouse layer, Lance for the derived ML feature-store layer, with nightly conversion where the two need to share data.

What is LanceDB and how does it differ from Lance?

LanceDB is a vector database (SDK plus optional managed-cloud service) that uses lance format as its native on-disk storage. It provides an ORM-like API layer for embedding-heavy workloads: db.create_table, table.add, table.search, hybrid full-text + vector queries, authentication, multi-tenancy, and a hosted cloud option. Lance (the format) is the underlying columnar file format — pages, manifests, indices, versions — that you can also write to and read from directly via lance.write_dataset and lance.dataset(URI) without ever touching the LanceDB API. Pick LanceDB when you want a vector-DB abstraction with production features (auth, multi-tenancy, cloud hosting) and don't need direct control over file layout. Pick raw Lance when you're wiring Lance into an existing Arrow ecosystem (Ray Data, Polars, DuckDB), need custom index tuning beyond LanceDB's exposed knobs, or want direct control over compaction and versioning. Both write the same Lance format on disk — you can migrate between LanceDB and raw Lance without moving data, only changing the API layer. In interviews, name Lance as the format and LanceDB as the database on top; conflating the two is a common junior mistake.

Can I use Lance with Ray Data / Spark / DuckDB?

Yes — Lance ships first-class readers for the modern data-Python and SQL ecosystem. Ray Data reads via ray.data.read_lance(uri, version=N) which shards fragments across workers for parallel training-shuffle at ~100K rows/sec per worker (vs Parquet's ~1-5K rows/sec); pin version=N for per-epoch reproducibility. Polars reads via pl.scan_lance(uri) (lazy) or pl.read_lance(uri) (eager); the LazyFrame optimiser pushes filter, projection, and limit down into the Lance reader for pruning. DuckDB reads via the lance extension (INSTALL lance; LOAD lance; then SELECT * FROM lance_scan('URI')); DuckDB's optimiser pushes filters down, and the vectorised engine executes over Arrow batches with zero-copy handoff. Spark reads via the Lance Spark connector for teams already on Spark — throughput is competitive with the Ray Data path. pandas reads via lance.dataset(URI).to_pandas() or to_table().to_pandas() for zero-copy Arrow-to-pandas conversion. The universal factor is Apache Arrow: because Lance is Arrow-native on disk, every Arrow-consuming library sees it as a first-class source with no format-translation overhead. Migrating to Lance never means abandoning your existing query language; it means gaining random-access performance under exactly the same reader surface.

Which vector index should I build in Lance — IVF-PQ or HNSW?

Default to IVF_HNSW_SQ for most production recommender and semantic-search workloads — it's the hybrid that combines IVF's partitioned search (num_partitions clusters of vectors), HNSW's high-recall graph navigation within each partition (m graph degree, ef_construction build-time candidates), and SQ's 8-bit scalar quantization (num_sub_vectors = embedding_dim / 8) for ~4× memory reduction. Typical config for a 768-dim embedding on 100M rows: num_partitions=1024, num_sub_vectors=96, m=16, ef_construction=200. Pick IVF_FLAT (exact within partition, no quantization) when recall is the dominant metric and you can afford the ~4× storage — regulatory search, precision-critical scientific data, small datasets where memory isn't a constraint. Pick IVF_PQ (product quantization without HNSW) when memory is extremely tight — billion-scale vectors on modest hardware — and you can accept a ~5-10% recall drop vs IVF_HNSW_SQ. The tuning knobs matter as much as the choice: nprobes (query-time) controls how many partitions to search — start at 5% of num_partitions and adjust based on measured recall/latency; refine_factor controls how many extra candidates to rerank with exact distance — start at 10, increase for recall-critical queries. Every senior Lance interview probes this axis, so know the three choices and the four tuning knobs.

Is Lance production-ready in 2026?

Yes — Lance has been production-ready for at least two years and is the storage backing for major ML platforms including LanceDB (their own product), several recommender systems at large tech companies, and a growing number of feature stores in the ML-Ops ecosystem. The format is stable (v1.0 forwards); the Python SDK (pylance) is mature; readers exist for Ray Data, Polars, DuckDB, Spark, and pandas; the LanceDB layer adds enterprise features (auth, multi-tenancy, managed cloud). The known operational caveats: (a) Lance datasets are single-writer per dataset — parallel writers cause manifest conflicts, so use an orchestrator (Airflow, Prefect) to enforce a single-writer lease; (b) the format is strongest in the Arrow ecosystem — JVM-only shops (Kafka Streams, Flink) still need the Spark connector rather than a native reader; (c) Snowflake / BigQuery / Athena do not read Lance directly — for BI-heavy shops, run a nightly Lance-to-Parquet-in-Iceberg export as the glue layer. None of these are deal-breakers for ML platform teams; they're the shape of what "production-ready for the ML workload" means in 2026. The wrong question is "is Lance production-ready?" — it is. The right question is "does my workload match Lance's sweet spot?" — random access, vectors, versions, Arrow — in which case adopting it pays back within one training cycle.

Practice on PipeCode

• Drill the SQL practice library → for the feature-store, metadata-filter, and top-k ranking problems senior interviewers love.
• Rehearse on the aggregation practice library → for the group-by + filtered-scan patterns that dominate warehouse-side analytics over Lance-and-Iceberg hybrids.
• Sharpen the streaming axis with the streaming practice library → for CDC-into-Lance, nightly rebuild, and Iceberg-to-Lance glue-layer scenarios.
• Level up on join fluency with the joins practice library → so multi-source queries (Iceberg fact table joined to Lance feature store via DuckDB) feel native.
• Practise system-design reasoning on the design practice library → for the format-decision, feature-store architecture, and vector-search-API problems this article maps to.
• Stack the prerequisites against PipeCode's broader 450+ data-engineering catalogue to anchor the four-axis Lance decision matrix against real graded inputs.

Lock in lance format muscle memory

Docs explain the format. PipeCode drills explain the decision — when Parquet's row-group-per-row cost breaks training-shuffle, when Lance's in-file HNSW earns its keep against a co-located FAISS, when Iceberg-on-Parquet remains the right warehouse answer, when the LanceDB API layer buys you enterprise features and when raw Lance keeps you in the Ray/Polars/DuckDB flow. Pipecode.ai is Leetcode for Data Engineering — pattern-first practice tuned for the production trade-offs senior data engineers and ML-platform engineers actually face.

Practice SQL problems →
Practice design problems →