ORC vs Parquet vs Lance: Choosing a Columnar File Format in 2026

orc vs parquet is the columnar-storage decision most senior data engineers made by inertia a decade ago — "the warehouse team uses ORC, so we use ORC" or "Spark defaults to Parquet, so we use Parquet" — and it is exactly the decision that quietly caps how fast your analytical scans run, how small your storage bill is, and whether your ML team can serve embeddings without bolting a second system onto the side. Every byte your pipeline writes to a lake lands in some physical layout, and that layout decides three things you cannot renegotiate cheaply later: whether a query that touches two of forty columns reads two columns or forty, whether a WHERE order_date = '2026-08-01' scan skips 99% of the file or reads all of it, and whether a point lookup of row 4,021,887 is a millisecond take or a full-file decode. The columnar file format you pick is not a cosmetic choice — it is the read-pattern contract every downstream engine inherits.

This guide is the senior-DE walkthrough you wished existed the first time an interviewer asked "walk me through orc vs parquet and when you'd reach for either," or "why is Parquet the de-facto default if ORC compresses better on Hive," or "your ML team wants random access to 200 million training rows plus vector search — what format serves that?" It walks through the three contenders — apache orc (stripes, built-in indexes, Hive ACID), apache parquet (row groups, Dremel nesting, ecosystem gravity), and the lance format (fast random access, zero-copy versioning, native vector search) — the four axes interviewers actually probe (read pattern, ecosystem, evolution, random access), and the head-to-head benchmarks and decision matrix that turn "it depends" into a defensible pick. Each section pairs a teaching block with a Solution-Tail interview answer — real runnable pyarrow / orc / lance code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the optimization practice library →, rehearse on the ETL practice library →, and sharpen the storage-design axis with the design practice library →.

---

On this page

• Why the columnar format choice still matters in 2026
• Apache ORC — stripes, indexes, ACID in Hive
• Apache Parquet — row groups, ecosystem gravity
• Lance — the modern columnar + vector format
• Head-to-head — benchmarks and the decision matrix
• Cheat sheet — columnar format recipes
• Frequently asked questions
• Practice on PipeCode

---

1. Why the columnar format choice still matters in 2026

Three columnar formats, three design centers — the choice binds your read pattern, ecosystem, and evolution story for years

The one-sentence invariant: a columnar file format is the physical contract that decides whether your engine reads only the columns and row ranges a query touches or the whole file, and the three live contenders in 2026 — apache orc, apache parquet, and lance format — were each designed around a different read pattern (Hive-warehouse scans, universal analytics, and ML random access respectively), so the right pick falls out of your workload, not out of a benchmark leaderboard. All three are columnar, all three store min/max statistics for pruning, and all three compress well — but they diverge sharply on ecosystem breadth, versioning semantics, and how cheaply they serve a single-row lookup. The format you pick becomes the default that every query engine, every catalog, and every downstream consumer inherits, and migrating a petabyte of one format to another is a real multi-week project, so the choice compounds.

The three contenders — one paragraph each.

• Apache ORC (Optimized Row Columnar). Born in 2013 inside the Hive ecosystem to replace RCFile. Its differentiators are aggressive lightweight compression, built-in multi-level indexes (file, stripe, and row-group), optional per-column bloom filters, and first-class Hive ACID transactional tables. If your world is Hive / Tez / on-prem Hadoop, ORC is native.
• Apache Parquet. Born the same year out of Twitter + Cloudera, based on Google's Dremel record-shredding paper. Its differentiator is not any single feature — it is ecosystem gravity. Spark, Arrow, DuckDB, Polars, pandas, Delta Lake, Apache Iceberg, Apache Hudi, Snowflake external tables, BigQuery external tables, and Trino all read and write Parquet as a first-class citizen. It is the safe default for analytics.
• Lance. The modern entrant (Rust, built on Apache Arrow, from the LanceDB team). Its differentiator is fast random access plus zero-copy versioning plus native vector search — it is a vector search format designed for ML/AI training data and embeddings, where you need point lookups, reproducible dataset versions, and approximate-nearest-neighbor queries in one file format.

The four axes interviewers actually probe.

• Read pattern. Full-column scans (analytics) vs single-row point lookups (random access, serving, ML sampling). Parquet and ORC are tuned for scans; a point lookup forces them to decode a whole row group / stripe. Lance is tuned for random access — a take of scattered row ids is O(rows requested), not O(row group).
• Ecosystem gravity. How many engines read the format without a plugin? Parquet wins by a wide margin; ORC is strong in Hive/Trino/Spark; Lance is newer but has pyarrow, DuckDB, pandas, Polars, and PyTorch/TensorFlow data-loader integrations.
• Evolution / versioning. Can the format add columns, and does it track dataset versions for reproducibility and time travel? Parquet and ORC evolve schema at the table layer (Iceberg/Delta/Hive metastore), not in the file. Lance versions natively — every write produces a new manifest, and you can check out an older version by number.
• Random access. The sharpest divider. predicate pushdown skips row groups you don't need, but within a matched row group both Parquet and ORC decode contiguously. Lance's layout supports true random take(row_ids) cheaply, which is why it wins ML sampling and vector re-ranking.

The 2026 reality — Parquet is the default, ORC holds Hive, Lance is the ML riser.

• Parquet is the correct default for any greenfield lakehouse. It is what Delta Lake, Iceberg, and Hudi write underneath; picking Parquet means every engine in the ecosystem can read your data tomorrow.
• ORC remains dominant where Hive is dominant — mature on-prem Hadoop, Hive ACID transactional tables, and shops where ORC's compression edge on wide warehouse fact tables translates to a real storage-cost line item.
• Lance is the riser for ML/AI: training-data catalogs, embedding stores, and RAG pipelines where vector search format semantics, reproducible versioning, and cheap random access matter more than the last drop of ecosystem breadth.

What interviewers listen for.

• Do you say "it depends on the read pattern" and then name the axes — rather than declaring a universal winner? — senior signal.
• Do you know why Parquet won despite ORC compressing better on Hive — ecosystem gravity, Arrow integration, and the Dremel nesting model? — senior signal.
• Do you name Lance (or another random-access format) when the workload is ML sampling + vector search, instead of forcing Parquet? — senior signal.
• Do you describe a format as "a read-pattern contract" rather than "a way to store data"? — required answer.

Worked example — the three-format comparison table

Detailed explanation. The single most useful artifact for a columnar-format interview is a memorised 3×N comparison table. Every senior storage discussion converges on this table within the first ten minutes; having it in your head is what separates a fluent answer from a stumbling one. Walk through building the table for a hypothetical events table that has to serve three consumers: an analytics dashboard, a Hive-legacy reporting job, and an ML feature-sampling loop.

• Source data. A 2-billion-row events(event_id, user_id, event_ts, event_type, payload, embedding) dataset, ~40 columns wide, with a 768-dim float embedding column.
• Consumer 1 — analytics. SELECT event_type, count(*) ... GROUP BY event_type scans over a few columns; needs full-scan throughput and small storage.
• Consumer 2 — Hive legacy. An existing Hive ACID reporting table that upserts late-arriving corrections.
• Consumer 3 — ML. A training loop that samples 10,000 scattered rows per batch and runs vector similarity re-ranking on the embedding column.

Question. Build the three-format comparison for the events workload and pick the format each consumer should use.

Input.

Axis	Apache ORC	Apache Parquet	Lance
Design center	Hive warehouse scans	universal analytics	ML random access + vectors
Physical unit	stripe (row-group index inside)	row group (pages inside)	fragment (pages, row-id addressable)
Built-in indexes	file/stripe/row-group + bloom	row-group + page + bloom	scalar (btree/bitmap) + vector (IVF_PQ/HNSW)
Ecosystem breadth	Hive/Trino/Spark strong	widest (every engine)	newer; pyarrow/DuckDB/pandas/Polars
Versioning	at table layer (Iceberg/Delta/Hive)	at table layer	native in-format manifest versions
Random access	decode whole stripe	decode whole row group	cheap take(row_ids)
Vector search	no	no (external index)	native ANN

Code.

# Build the same Arrow table once; write it three ways.
import numpy as np
import pyarrow as pa

n = 100_000
rng = np.random.default_rng(42)
events = pa.table({
    "event_id":  pa.array(np.arange(n, dtype=np.int64)),
    "user_id":   pa.array(rng.integers(1, 50_000, size=n, dtype=np.int64)),
    "event_ts":  pa.array(rng.integers(1_722_000_000, 1_724_000_000, size=n, dtype=np.int64)),
    "event_type":pa.array(rng.choice(["view", "click", "purchase", "refund"], size=n)),
    "payload":   pa.array([f'{{"k":{int(v)}}}' for v in rng.integers(0, 1000, size=n)]),
    "embedding": pa.array(rng.standard_normal((n, 8)).tolist(),
                          type=pa.list_(pa.float32(), 8)),
})

# ORC  — Hive-native
import pyarrow.orc as orc
orc.write_table(events.drop_columns(["embedding"]), "events.orc", compression="ZSTD")

# Parquet — universal analytics
import pyarrow.parquet as pq
pq.write_table(events.drop_columns(["embedding"]), "events.parquet", compression="zstd")

# Lance — ML random access + vectors (keeps the embedding column)
import lance
lance.write_dataset(events, "events.lance")

print("ORC     stripes:", orc.ORCFile("events.orc").nstripes)
print("Parquet row groups:", pq.ParquetFile("events.parquet").num_row_groups)
print("Lance   versions:", lance.dataset("events.lance").version)

Step-by-step explanation.

• One Arrow table, three writers. Because ORC, Parquet, and Lance all interoperate with Apache Arrow, the same in-memory pa.table serialises to all three. This is the practical reason to think in Arrow first: the format becomes a write-time decision, not a data-modeling decision.
• The embedding column is the divider. ORC and Parquet can store a list<float> column, but neither can index it for nearest-neighbor search — so the analytics and Hive writes drop it. Lance keeps it because it can build a vector index on it later.
• Physical units differ. orc.ORCFile(...).nstripes and pq.ParquetFile(...).num_row_groups expose the coarse pruning unit each format skips at; Lance exposes a version because versioning is in-format, not in an external catalog.
• The pick is consumer-driven. Analytics → Parquet (widest engine support, great scans). Hive legacy → ORC (native ACID, native to the existing metastore). ML sampling + vectors → Lance (random access + native ANN).

Output.

Consumer	Recommended format	Why
Analytics dashboard	Apache Parquet	widest engine support; excellent scan + projection
Hive-legacy reporting	Apache ORC	native ACID; native to the Hive metastore; compression edge
ML feature sampling + vectors	Lance	cheap random access; native vector index; in-format versioning

Rule of thumb. Never pick a columnar format by "which benchmark won." Pick it by (read pattern × ecosystem × versioning × random access). Write the four-axis table on the whiteboard first; the format falls out of the workload.

Worked example — why columnar beats row-oriented for analytics

Detailed explanation. Before comparing the three formats, an interviewer often checks that you understand why columnar wins at all. The two mechanisms are column projection (read only the columns a query touches) and predicate pushdown (skip row ranges whose statistics prove they can't match). A row-oriented file (CSV, JSON, Avro) must read every byte of every row even to answer SELECT event_type, count(*). Walk through the byte-accounting.

• Row layout. All columns of row 1, then all columns of row 2. Reading one column means seeking past every other column, row by row — effectively a full file read.
• Columnar layout. All values of event_type contiguously, then all values of user_id, etc. Reading one column reads one contiguous run; the other 39 columns are never touched.
• Compression compounds it. A column holds values of one type with low cardinality (e.g. four event_type strings), so dictionary + run-length encoding shrinks it 10–50×. Row layout interleaves types and compresses worse.

Question. Quantify the bytes read for SELECT event_type, count(*) FROM events GROUP BY event_type against a row format vs a columnar format for a 40-column, 2-billion-row table.

Input.

Quantity	Row format (CSV/Avro)	Columnar (Parquet/ORC)
Columns read	40 of 40	1 of 40
Bytes scanned (uncompressed)	~800 GB	~20 GB
After columnar compression	n/a (~200 GB gzip)	~1–2 GB (dict + RLE)
Row-group / stripe pruning	none	applies if predicate present

Code.

import os
import pyarrow as pa
import pyarrow.parquet as pq
import pyarrow.csv as pacsv

tbl = pq.read_table("events.parquet")

# Row-oriented baseline: CSV must be fully read to answer any query.
pacsv.write_csv(tbl, "events.csv")

# Columnar: read ONLY the event_type column.
only_type = pq.read_table("events.parquet", columns=["event_type"])

def mb(path):
    return os.path.getsize(path) / 1024 / 1024

print(f"CSV file on disk:        {mb('events.csv'):8.2f} MB (all columns, row-major)")
print(f"Parquet file on disk:    {mb('events.parquet'):8.2f} MB (all columns, compressed)")
print(f"event_type column bytes: {only_type.nbytes / 1024 / 1024:8.2f} MB (projected read)")
# The projected read touches a single column chunk, not the whole file.

Step-by-step explanation.

• Column projection is a byte filter. pq.read_table(..., columns=["event_type"]) reads only that column's chunk in each row group. The engine seeks to the column-chunk offset recorded in the footer and never decodes the other 39 columns.
• CSV has no projection. To compute the same aggregate from CSV, the reader must parse every row's every field to find the event_type position — there is no way to skip columns in a row-major text file.
• Compression is per-column and type-aware. event_type has four distinct values, so dictionary encoding stores them once plus a stream of small integer codes, then run-length-encodes the codes. That is the 10–50× shrink columnar formats get and row formats cannot.
• This is the substrate all three formats share. ORC, Parquet, and Lance all get projection + per-column compression; the differences between them (indexes, versioning, random access) sit on top of this shared columnar substrate.

Output.

Metric	CSV (row)	Parquet (columnar)
Bytes to answer GROUP BY event_type	whole file	one column chunk
Compression of event_type	poor (interleaved)	dictionary + RLE, 10–50×
Predicate pushdown available	no	yes (row-group stats)
Typical scan speedup	1×	10–100×

Rule of thumb. Columnar wins analytics for two reasons — projection (read fewer columns) and pushdown (read fewer rows). Any format comparison starts from this shared baseline; the interesting differences between ORC, Parquet, and Lance are what they layer on top.

Worked example — the "pick the format" decision tree

Detailed explanation. Given a new dataset, the senior architect runs a short decision tree in their head. Codifying it makes the interview answer reproducible: an interviewer hands you a scenario and you walk the tree out loud. Walk through the tree with three canonical scenarios: a greenfield lakehouse fact table, a Hive ACID reporting table, and an ML embedding store.

• Q1. Is this an ML/AI workload needing random access, vector search, or reproducible dataset versions? → yes = Lance; no = Q2.
• Q2. Are you locked into a Hive metastore with ACID transactional tables (upserts/deletes)? → yes = ORC; no = Q3.
• Q3. Do you need the widest possible engine support and a lakehouse table format (Delta/Iceberg/Hudi) underneath? → yes = Parquet (the default); no = still Parquet unless a specific axis overrides.
• Q4 (override branch). Is storage cost the single dominant constraint on wide Hive fact tables and you already run ORC readers? → ORC may edge out Parquet on compression.

Question. Walk the decision tree for the three scenarios and record the format each ends up with.

Input.

Scenario	Q1 (ML/vectors?)	Q2 (Hive ACID?)	Q3 (widest support?)
Greenfield lakehouse fact table	no	no	yes
Hive ACID reporting table	no	yes	—
ML embedding + training store	yes	—	—

Code.

def pick_format(ml_or_vectors: bool,
                hive_acid: bool,
                widest_support: bool = True) -> str:
    """Return the columnar file format for a dataset."""
    if ml_or_vectors:
        return "lance"          # random access + vector search + versioning
    if hive_acid:
        return "orc"            # Hive-native ACID transactional tables
    # default: analytics / lakehouse
    return "parquet"            # widest ecosystem; Delta/Iceberg/Hudi write it

print(pick_format(False, False))          # -> parquet
print(pick_format(False, True))           # -> orc
print(pick_format(True,  False))          # -> lance

Step-by-step explanation.

• Scenario 1 — greenfield lakehouse. No ML, no Hive lock-in, needs broad support → Parquet. This is the modern default and the safe answer when nothing else overrides.
• Scenario 2 — Hive ACID reporting. The table already lives in a Hive metastore and needs transactional upserts → ORC, which has the most mature Hive ACID implementation (base + delta files, native compaction).
• Scenario 3 — ML embedding store. Random-access sampling and vector similarity dominate → Lance, whose whole design center is exactly this.
• The tree is order-sensitive. ML/vectors is checked first because it is the most format-specific requirement; Hive ACID second because it is a hard ecosystem constraint; the Parquet default catches everything else.

Output.

Scenario	Format	Deciding axis
Greenfield lakehouse	Parquet	ecosystem gravity
Hive ACID reporting	ORC	Hive-native ACID
ML embedding store	Lance	random access + vector search

Rule of thumb. The decision tree is a whiteboard-friendly answer. Practice walking it end-to-end so an interviewer can hand you any scenario and get a format name in under 60 seconds — with the deciding axis named, not just the format.

Senior interview question on columnar format selection

A senior interviewer often opens with: "You're standing up a new data platform with three consumers — a BI dashboard over a 5-TB fact table, a legacy Hive reporting job with transactional corrections, and an ML team that needs to sample training rows and run vector search over 300 million embeddings. One team wants to standardise on a single file format 'to keep things simple.' Talk me out of it, and give me the format for each consumer with the trade-off you're making."

Solution Using a workload-driven format decision matrix

# format_decision.py — encode the three-consumer decision as a matrix
from dataclasses import dataclass

@dataclass
class Workload:
    name: str
    read_pattern: str     # "scan" | "point" | "mixed"
    hive_acid: bool
    needs_vectors: bool
    needs_versioning: bool

def choose(w: Workload) -> tuple[str, str]:
    if w.needs_vectors or (w.read_pattern == "point" and w.needs_versioning):
        return "lance", "random access + native vector index + in-format versions"
    if w.hive_acid:
        return "orc", "native Hive ACID (base/delta + compaction) and compression edge"
    return "parquet", "widest ecosystem; Delta/Iceberg/Hudi write it; strong scans"

workloads = [
    Workload("BI dashboard",     "scan",  hive_acid=False, needs_vectors=False, needs_versioning=False),
    Workload("Hive reporting",   "scan",  hive_acid=True,  needs_vectors=False, needs_versioning=False),
    Workload("ML train + vector","point", hive_acid=False, needs_vectors=True,  needs_versioning=True),
]

for w in workloads:
    fmt, why = choose(w)
    print(f"{w.name:22s} -> {fmt:8s} | {why}")

Step-by-step trace.

Workload	read_pattern	hive_acid	needs_vectors	Chosen format
BI dashboard	scan	False	False	parquet
Hive reporting	scan	True	False	orc
ML train + vector	point	False	True	lance

Running the script walks each workload through the same branch order the mental decision tree uses. The BI dashboard has no overriding constraint, so it lands on the Parquet default. The Hive reporting job trips the hive_acid branch and lands on ORC. The ML workload trips the needs_vectors branch and lands on Lance. The "one format to rule them all" request fails because no single format is simultaneously the widest-supported and the Hive-ACID-native and the random-access + vector-native option — those are three different design centers.

Output:

Workload	Format	Trade-off accepted
BI dashboard	Parquet	slightly worse compression than ORC on wide fact tables
Hive reporting	ORC	narrower engine support outside Hive/Trino/Spark
ML train + vector	Lance	newest ecosystem; fewer third-party integrations than Parquet

Why this works — concept by concept:

• Read pattern first — scan-heavy analytics favors Parquet/ORC; point-heavy ML sampling favors Lance. The branch order encodes this: vectors and point+versioning are checked before the scan defaults.
• Hive ACID is an ecosystem constraint — if the data must live in a Hive metastore with transactional semantics, ORC's base/delta/compaction implementation is the most battle-tested, so it overrides the Parquet default.
• One format for three design centers is a category error — "simplicity" that ignores read pattern buys operational uniformity at the cost of 10–100× slower ML sampling or a lost vector index. Naming that trade-off explicitly is the senior signal.
• Cost — three formats mean three sets of readers and three sets of tuning knobs, but each consumer runs on its optimal substrate. The alternative — one format — saves operational surface but taxes the worst-fit consumer forever. Choose per read pattern; standardise the catalog (Iceberg/Unity), not the file format.

Optimization
Topic — optimization
Optimization problems on scan pruning and projection

Practice →

Data Processing Topic — data-processing Data-processing problems on columnar layouts

Practice →

---

2. Apache ORC — stripes, indexes, ACID in Hive

Stripes, row-group indexes, and Hive ACID — where ORC's built-in indexing still wins

The mental model in one line: apache orc is a columnar format that splits a file into large horizontal stripes, embeds three levels of statistics (file, stripe, and 10,000-row row-group index) plus optional per-column bloom filters directly inside the file, encodes each column with type-specialised lightweight compression (run-length, dictionary) under an optional block codec (ZLIB / ZSTD / Snappy), and — uniquely among the three — ships a mature Hive ACID implementation with base + delta files and background compaction, which is why ORC still wins in Hive-centric warehouses even though Parquet has broader reach. Every senior data engineer who has run on-prem Hadoop has tuned an ORC stripe size at least once, and the format remains the correct answer for a specific, still-large slice of the world.

The ORC file anatomy — top to bottom.

• Stripes. The coarse horizontal partition — a contiguous group of rows (default target ~64 MB via pyarrow; Hive historically used ~256 MB). A stripe is the unit an engine can skip entirely if its statistics prove no row matches. Each stripe is self-contained: index data, row data, and a stripe footer.
• Row-group index (row index stride). Inside a stripe, every column is divided into row groups of 10,000 rows (the row.index.stride). For each row group ORC stores min/max/count/sum and a position pointer so a reader can seek directly to a matching row group without decoding the ones before it.
• Bloom filters. Optional, per column. For high-cardinality equality predicates (WHERE user_id = 91237) min/max is useless (the value is in range for almost every stripe), but a bloom filter answers "definitely-not-here" per row group and skips it.
• File footer + postscript. The footer holds the list of stripes, the schema (TypeDescription), and file-level statistics. The postscript (at the very end) records the compression codec and footer length so a reader knows how to bootstrap the file.

Predicate pushdown (PPD) — ORC's original claim to fame.

• Three-level skipping. A predicate is evaluated against file stats (skip the whole file), then stripe stats (skip stripes), then row-group stats (skip 10,000-row groups). Each level prunes before any column data is decoded.
• Sorted data multiplies it. If you write ORC sorted by the predicate column, min/max ranges per stripe/row-group become tight and non-overlapping, so PPD skips almost everything. Unsorted data has overlapping ranges and prunes little — sorting is the single biggest ORC tuning lever.
• Bloom filters for equality. Add bloom_filter_columns for the columns you filter by equality on high-cardinality keys. They cost ~5–10% file size for a large skip win on selective lookups.

Compression + encoding — why ORC is small.

• Lightweight (type-aware) encoding. Integer run-length encoding (RLE v2), dictionary encoding for strings, and delta encoding for sorted sequences — applied before the block codec. This is where most of ORC's size win comes from.
• Block codec. ZLIB (default, best ratio), ZSTD (great ratio + speed, the 2026 choice), Snappy (fastest, larger). The codec compresses the already-lightweight-encoded streams.
• Net effect. On wide, low-cardinality Hive fact tables ORC frequently lands 10–20% smaller than equivalently-configured Parquet — a real storage-cost difference at petabyte scale.

Hive ACID — the feature Parquet doesn't natively match.

• Base + delta files. A transactional ORC table stores a compacted base directory plus per-transaction delta directories. Inserts, updates, and deletes append delta files; readers merge base + deltas at query time.
• Compaction. Minor compaction merges many deltas into fewer; major compaction rewrites base + deltas into a fresh base. Without compaction, read amplification grows with every transaction.
• Why it matters. ORC ACID predates Delta Lake / Iceberg. For shops already on Hive, it is upsert/delete support without adopting a new table format.

Common interview probes on ORC.

• "What's ORC's coarse pruning unit?" — the stripe; then row-group index (10,000 rows) inside it.
• "How do you make ORC predicate pushdown effective?" — sort by the predicate column; add bloom filters for high-cardinality equality.
• "When is ORC better than Parquet?" — Hive-native ACID tables, and compression on wide low-cardinality fact tables.
• "What's the row index stride?" — 10,000 rows; the granularity of ORC's finest statistics.

Worked example — writing ORC and inspecting stripes

Detailed explanation. The canonical first ORC task: write a table, then read back the stripe count and column statistics that the format embedded. pyarrow.orc exposes the stripe count and per-column file statistics; the CLI orc-tools (or Hive) exposes row-group detail. Walk through writing and inspecting.

• Write. orc.write_table(table, path, compression="ZSTD", stripe_size=...).
• Inspect stripe count. orc.ORCFile(path).nstripes.
• Inspect schema + row count. .schema, .nrows.

Question. Write a 100,000-row orders table to ORC with ZSTD and a small stripe size, then report how many stripes the file has and what the per-stripe row count works out to.

Input.

Parameter	Value
Rows	100,000
Columns	id, customer_id, total_cents, status, order_ts
Codec	ZSTD
stripe_size	8 MB (small, to force multiple stripes)

Code.

import numpy as np
import pyarrow as pa
import pyarrow.orc as orc

n = 100_000
rng = np.random.default_rng(7)
orders = pa.table({
    "id":          pa.array(np.arange(n, dtype=np.int64)),
    "customer_id": pa.array(rng.integers(1, 20_000, size=n, dtype=np.int64)),
    "total_cents": pa.array(rng.integers(100, 500_00, size=n, dtype=np.int64)),
    "status":      pa.array(rng.choice(["pending", "shipped", "refunded"], size=n)),
    "order_ts":    pa.array(rng.integers(1_722_000_000, 1_724_000_000, size=n, dtype=np.int64)),
})

# Small stripe_size forces several stripes so we can see the partitioning.
orc.write_table(orders, "orders.orc", compression="ZSTD", stripe_size=8 * 1024 * 1024)

f = orc.ORCFile("orders.orc")
print("stripes:", f.nstripes)
print("rows:   ", f.nrows)
print("rows/stripe (approx):", f.nrows // max(f.nstripes, 1))
print("schema:", f.schema.names)

Step-by-step explanation.

• stripe_size controls the coarse pruning unit. A smaller stripe means more stripes, each holding fewer rows — finer pruning granularity but more per-stripe overhead. The 8 MB value is deliberately small to force multiple stripes on 100k rows.
• nstripes is the skip budget. At query time, PPD can skip whole stripes; more stripes means finer skipping but larger footers. Production Hive fact tables use big stripes (256 MB) to amortise footer overhead across many rows.
• nrows and schema come from the footer. These are read without touching any column data — the reader parses only the footer + postscript to answer "how big, what shape."
• ZSTD is the modern codec choice. It gets close to ZLIB's ratio at much higher decode speed, which is why it has become the default recommendation for both ORC and Parquet in 2026.

Output.

Metric	Value
Stripes	multiple (driven by 8 MB stripe_size)
Rows	100,000
Rows per stripe (approx)	100,000 / nstripes
Footer read cost	O(1) — no column decode

Rule of thumb. Tune stripe_size to your read pattern: big stripes (256 MB) for scan-heavy Hive fact tables to amortise footer overhead; smaller stripes for selective workloads where finer stripe-level pruning pays. The row index stride (10,000) gives you sub-stripe pruning for free.

Worked example — predicate pushdown with stripe stats and a bloom filter

Detailed explanation. ORC's headline feature is skipping data the query can't match. Two mechanisms: min/max stripe/row-group stats (great for range predicates on sorted columns) and bloom filters (great for equality predicates on high-cardinality columns). Write ORC sorted by order_ts with a bloom filter on customer_id, then reason about what a selective query reads.

• Sort by order_ts. Makes per-stripe min/max ranges tight and non-overlapping, so a date-range predicate skips most stripes.
• Bloom filter on customer_id. Makes an equality lookup on a high-cardinality id skip row groups the id is definitely not in.
• Read with a filter. pyarrow's ORC reader accepts a filter= expression and applies pushdown.

Question. Write ORC sorted by order_ts with a customer_id bloom filter, then read only the rows for one customer in a two-day window and describe what pushdown skipped.

Input.

Setting	Value
Sort key	order_ts (ascending)
Bloom filter column	customer_id
Query predicate	customer_id = 4242 AND order_ts in [T0, T0+2d]
Pruning path	stripe min/max (order_ts) + bloom (customer_id)

Code.

import numpy as np
import pyarrow as pa
import pyarrow.orc as orc
import pyarrow.compute as pc
from pyarrow import dataset as ds

n = 500_000
rng = np.random.default_rng(11)
ts = np.sort(rng.integers(1_722_000_000, 1_724_000_000, size=n)).astype(np.int64)
orders = pa.table({
    "id":          pa.array(np.arange(n, dtype=np.int64)),
    "customer_id": pa.array(rng.integers(1, 100_000, size=n, dtype=np.int64)),
    "total_cents": pa.array(rng.integers(100, 500_00, size=n, dtype=np.int64)),
    "order_ts":    pa.array(ts),                       # already sorted -> tight stripe ranges
})

orc.write_table(
    orders, "orders_sorted.orc",
    compression="ZSTD",
    stripe_size=16 * 1024 * 1024,
    bloom_filter_columns=["customer_id"],              # equality-lookup skip
    bloom_filter_fpp=0.02,
)

# Read with predicate pushdown via the dataset API.
dataset = ds.dataset("orders_sorted.orc", format="orc")
T0 = int(ts[len(ts) // 2])                             # a timestamp in the middle
predicate = (
    (pc.field("customer_id") == 4242) &
    (pc.field("order_ts") >= T0) &
    (pc.field("order_ts") < T0 + 2 * 86400)
)
result = dataset.to_table(filter=predicate, columns=["id", "customer_id", "order_ts"])
print("matched rows:", result.num_rows)
print("columns read:", result.column_names)   # only 3 of 4 columns decoded

Step-by-step explanation.

• Sorting by order_ts is what makes range pushdown work. Because rows are ordered, each stripe's order_ts min/max covers a narrow, non-overlapping window. The order_ts in [T0, T0+2d] predicate matches only the one or two stripes whose range overlaps the window; every other stripe is skipped on its min/max alone.
• The bloom filter handles the equality half. customer_id = 4242 cannot use min/max (4242 is within almost every stripe's range), but the per-row-group bloom filter answers "definitely not in this row group" and skips those groups. bloom_filter_fpp=0.02 trades a 2% false-positive rate for a small filter.
• Column projection stacks on top. columns=["id","customer_id","order_ts"] never decodes total_cents. Pruning (fewer rows) and projection (fewer columns) compose.
• The dataset API applies the pushdown. ds.dataset(...).to_table(filter=...) hands the predicate to the ORC reader, which evaluates it against embedded statistics before decoding.

Output.

Pruning stage	Mechanism	Effect
Stripe range skip	order_ts min/max (sorted)	reads ~1–2 stripes of many
Row-group skip	customer_id bloom filter	skips groups lacking 4242
Column projection	columns=[...]	total_cents never decoded
Net data touched	tiny fraction	selective lookup, not full scan

Rule of thumb. ORC PPD is only as good as your data layout. Sort by your most common range-predicate column so min/max ranges are tight, and add bloom filters for the high-cardinality columns you filter by equality. Unsorted ORC with no bloom filters prunes almost nothing — the statistics exist but overlap.

Worked example — Hive ACID base + delta files and compaction

Detailed explanation. ORC's transactional tables are the feature Parquet doesn't natively match. A Hive ACID ORC table stores a compacted base directory plus per-transaction delta directories; an update is a delete + insert recorded in deltas; readers merge base + deltas by row-id; compaction periodically folds deltas back into a base. Walk through the file-layout lifecycle.

• Insert. Writes a delta_* directory of ORC files with an ACID row-id triplet (writeId, bucketId, rowId).
• Update / delete. Writes a delete_delta_* marking the old row-id gone, plus (for update) a new delta_* row.
• Read. Merges base_* + delta_* minus delete_delta_* on the fly.
• Compaction. Minor: merge deltas into fewer deltas. Major: rewrite everything into a new base_*.

Question. Trace the directory layout of a Hive ACID ORC orders table through an insert, an update, and a major compaction, and explain read amplification.

Input.

Operation	Files produced	Reader must merge
Initial load	base_0000005/	base only
Insert 3 rows	delta_0000006_0000006/	base + 1 delta
Update 1 row	delete_delta_...7 + delta..._7	base + 2 deltas + 1 delete-delta
Major compaction	base_0000007/	base only (again)

Code.

-- Hive ACID transactional ORC table
CREATE TABLE orders (
    id          BIGINT,
    customer_id BIGINT,
    total_cents BIGINT,
    status      STRING
)
STORED AS ORC
TBLPROPERTIES ('transactional' = 'true');

-- 1) Initial load  -> base_0000005/
INSERT INTO orders VALUES (1, 10, 1500, 'pending'), (2, 11, 2200, 'pending');

-- 2) Insert        -> delta_0000006_0000006/
INSERT INTO orders VALUES (3, 12, 900, 'pending');

-- 3) Update        -> delete_delta_0000007_0000007/ + delta_0000007_0000007/
UPDATE orders SET status = 'shipped' WHERE id = 1;

-- Read now merges base + deltas - delete_deltas by (writeId, bucketId, rowId)
SELECT * FROM orders;

-- 4) Fold deltas back into a fresh base to cap read amplification
ALTER TABLE orders COMPACT 'major';
-- After compaction the reader sees a single base_0000007/ again.

# Directory layout over the lifecycle (HDFS / object store)
orders/
  base_0000005/                 # after initial load
  delta_0000006_0000006/        # after insert
  delete_delta_0000007_0000007/ # after update (tombstone for old id=1)
  delta_0000007_0000007/        # after update (new id=1 row, status=shipped)
# after ALTER TABLE ... COMPACT 'major':
orders/
  base_0000007/                 # everything merged; deltas removed

Step-by-step explanation.

• Every write is append-only at the file layer. ORC files are immutable, so an update cannot rewrite a row in place. Instead the update writes a delete_delta tombstone (by row-id) plus a fresh delta row — the classic merge-on-read design.
• The reader reconstructs current state on the fly. It reads base + all delta directories, subtracts rows named in delete_delta directories by their (writeId, bucketId, rowId), and returns the survivors. This is why reads get slower as deltas accumulate — that is read amplification.
• Compaction bounds read amplification. COMPACT 'major' rewrites base + deltas into a single new base and drops the old files, so the next reader is back to reading one base directory. Minor compaction is the cheaper "merge many deltas into few" step between majors.
• This predates Delta/Iceberg. For a shop already on Hive, ORC ACID delivers upsert/delete without adopting a new table format — which is precisely why ORC survives in 2026 warehouses.

Output.

Table state	Directories present	Read amplification
After load	base_5	1× (base only)
After insert	base_5 + delta_6	2 dirs merged
After update	base_5 + delta_6 + delta_7 + delete_delta_7	4 dirs merged
After major compaction	base_7	1× again

Rule of thumb. Hive ACID ORC is merge-on-read: every transaction adds delta files and every read merges them, so read latency degrades with un-compacted deltas. Schedule compaction (auto or manual) aggressively on write-heavy transactional tables — an un-compacted ACID table is a slow-read incident waiting to happen.

Senior interview question on Apache ORC

A senior interviewer might ask: "You own a 40-TB Hive ORC fact table (sales) that analysts filter by sale_date range and occasionally by a high-cardinality store_id. Scans are slower than they should be and the storage bill is climbing. Walk me through how you'd re-tune the ORC layout — stripe size, sort order, bloom filters, and codec — and how you'd quantify the win before rolling it out."

Solution Using ORC sort-order, bloom filters, and stripe tuning

# retune_orc.py — rewrite a Hive ORC fact table with a scan-optimised layout
import pyarrow as pa
import pyarrow.orc as orc
import pyarrow.compute as pc
from pyarrow import dataset as ds

# Read the existing (poorly-laid-out) table.
src = ds.dataset("sales_raw.orc", format="orc").to_table()

# 1) Sort by the dominant range-predicate column so stripe min/max ranges are tight.
order = pc.sort_indices(src, sort_keys=[("sale_date", "ascending")])
sorted_tbl = src.take(order)

# 2) Rewrite with a big stripe (scan-heavy), ZSTD, and a bloom filter on store_id.
orc.write_table(
    sorted_tbl, "sales_tuned.orc",
    compression="ZSTD",                    # ratio close to ZLIB, faster decode
    stripe_size=256 * 1024 * 1024,         # amortise footer over many rows
    bloom_filter_columns=["store_id"],     # high-cardinality equality skips
    bloom_filter_fpp=0.05,
)

# 3) Verify the layout the reader will use.
f = orc.ORCFile("sales_tuned.orc")
print("stripes:", f.nstripes, "| rows:", f.nrows)

# 4) Quantify: time a representative date-range + store_id lookup on both files.
def scan(path):
    d = ds.dataset(path, format="orc")
    predicate = (
        (pc.field("sale_date") >= 20260701) &
        (pc.field("sale_date") <  20260703) &
        (pc.field("store_id") == 8123)
    )
    return d.to_table(filter=predicate, columns=["sale_id", "sale_date", "amount"]).num_rows

print("raw   matched:", scan("sales_raw.orc"))
print("tuned matched:", scan("sales_tuned.orc"))   # same rows, far less data scanned

Step-by-step trace.

Step	Change	Why it helps
Sort by sale_date	rows ordered by predicate column	stripe min/max ranges become tight + non-overlapping
stripe_size = 256 MB	fewer, larger stripes	footer overhead amortised over a scan-heavy table
bloom on store_id	per-row-group membership	equality lookups skip groups lacking the id
ZSTD codec	modern block codec	~ZLIB ratio at higher decode speed
Projection in reader	3 of N columns	never decode unused columns

After the rewrite, a sale_date range query prunes down to the handful of stripes whose sorted min/max overlaps the window (instead of scanning all 40 TB), the store_id bloom filter skips row groups that can't contain the store, and ZSTD trims storage while decoding faster than the old ZLIB files. The win is measured by comparing matched-rows-per-bytes-scanned (or wall-clock) between sales_raw.orc and sales_tuned.orc on the same predicate.

Output:

Metric	Raw layout	Tuned layout
Stripe pruning on date range	little (overlapping ranges)	aggressive (tight ranges)
store_id equality skip	none	bloom-filter row-group skip
Codec	ZLIB (slow decode)	ZSTD (fast decode)
Storage	baseline	smaller
Scan wall-clock	baseline	multiples faster

Why this works — concept by concept:

• Sort order — ORC's min/max statistics only prune when ranges are tight. Sorting by the dominant range-predicate column turns overlapping per-stripe ranges into disjoint ones, which is the single largest PPD lever.
• Bloom filters — min/max is useless for high-cardinality equality (store_id = 8123 is in-range for nearly every stripe), so a per-row-group bloom filter provides the "definitely not here" skip that min/max cannot.
• Stripe size — big stripes amortise footer + index overhead across many rows for scan-heavy tables; the 10,000-row row-index stride still gives sub-stripe pruning inside each big stripe.
• ZSTD codec — the lightweight encodings (RLE/dictionary) do most of the shrinking; ZSTD as the block codec gets near-ZLIB ratio with materially faster decode, cutting both storage and scan time.
• Cost — one full rewrite pass (O(rows), one-time) buys tight-range pruning + equality skips on every future query. Compared with leaving the table unsorted, the rewrite pays for itself within days on a frequently-scanned 40-TB table. O(scanned bytes) drops from ~O(table) to ~O(matching stripes) per query.

Optimization
Topic — optimization
Optimization problems on file-layout and pushdown

Practice →

Design Topic — design Design problems on transactional warehouse tables

Practice →

---

3. Apache Parquet — row groups, ecosystem gravity

Row groups, column chunks, pages, and ecosystem gravity — why Parquet is the de-facto default

The mental model in one line: apache parquet is a columnar format that partitions a file into row groups, splits each row group into per-column column chunks, splits each chunk into pages (the smallest encode/compress/skip unit), records min/max/null statistics in a footer plus an optional page index and bloom filters for predicate pushdown, encodes nested and repeated fields with Google Dremel's repetition/definition levels, and — decisively — is read and written as a first-class citizen by nearly every engine and table format in the ecosystem, which is why Parquet, not ORC, became the de-facto default even though ORC compresses better on Hive. When the answer to "what format?" is "the one everything already reads," the answer is Parquet.

The Parquet file anatomy — top to bottom.

• Row groups. The coarse horizontal partition (default target ~128 MB). A row group is the unit an engine skips via row-group statistics. Fewer, larger row groups favor scans; more, smaller row groups favor selective pruning.
• Column chunks. Within a row group, one contiguous chunk per column. The footer records each chunk's byte offset, so a projected read seeks directly to the columns it needs.
• Pages. Within a column chunk, data is split into pages (~1 MB): data pages, dictionary pages, and (with the page index) per-page min/max. Pages are the finest skip and the unit of encoding + compression.
• Footer metadata. At the end of the file: schema, row-group metadata, per-column-chunk statistics (min/max/null_count), and optional page-index + bloom-filter offsets. A reader parses the footer first to plan the read.

Dremel nesting — the model ORC lacks in the same form.

• Repetition + definition levels. Parquet encodes arbitrarily nested and repeated fields (list<struct<...>>) by shredding them into flat columns plus two small integer streams: repetition levels (where a new list starts) and definition levels (how deep a null occurs). This is Google's Dremel record-shredding algorithm.
• Why it matters. Deeply nested JSON-like data stores and scans column-by-column without exploding into rows. Analytics over one nested field reads only that field's flat column plus its level streams.

Encoding + compression — how Parquet stays small.

• Encodings. Dictionary (default for low cardinality), RLE + bit-packing (for the codes and level streams), delta encodings (for sorted integers), and byte-stream-split (for floats). Applied per page, before the block codec.
• Block codecs. Snappy (fast, common default), ZSTD (best ratio/speed balance, the 2026 recommendation), GZIP, LZ4, BROTLI. Chosen per column chunk.
• Net effect. Comparable to ORC; ORC often edges it on wide low-cardinality Hive tables, Parquet closes the gap with ZSTD + dictionary + a well-chosen row-group size.

Predicate pushdown — three levels, like ORC.

• Row-group stats. Min/max/null_count per column chunk in the footer skip whole row groups.
• Page index. The optional column-index + offset-index (Parquet 2.x) stores per-page min/max so the reader skips individual pages inside a matched row group — ORC-style fine pruning.
• Bloom filters. Optional per column for high-cardinality equality, same role as in ORC.

Ecosystem gravity — the real reason Parquet won.

• Table formats write it. Delta Lake, Apache Iceberg, and Apache Hudi all store data files as Parquet. Choosing a lakehouse table format means choosing Parquet underneath.
• Every engine reads it. Spark, Trino, DuckDB, Polars, pandas (via Arrow), Snowflake/BigQuery external tables, ClickHouse, and Dask all read Parquet natively.
• Arrow is the bridge. Parquet ↔ Arrow zero-copy-ish conversion makes it the interchange format of the Python data stack.

Common interview probes on Parquet.

• "What's Parquet's coarse pruning unit?" — the row group; then the page (with the page index) inside it.
• "How does Parquet store nested data?" — Dremel repetition + definition levels.
• "Why did Parquet become the default over ORC?" — ecosystem gravity + Arrow + Dremel nesting; lakehouse table formats standardised on it.
• "How do you tune Parquet for selective queries?" — sort by predicate column, right-size row groups, enable the page index and bloom filters.

Worked example — writing Parquet and inspecting row-group metadata

Detailed explanation. The canonical first Parquet task: write a table and read back the row-group and column-chunk metadata the footer embedded, including the min/max statistics that drive pushdown. pyarrow.parquet exposes all of it via ParquetFile(...).metadata. Walk through it.

• Write. pq.write_table(table, path, row_group_size=..., compression="zstd").
• Inspect row groups. pq.ParquetFile(path).num_row_groups.
• Inspect column stats. metadata.row_group(i).column(j).statistics → min, max, null_count.

Question. Write a 300,000-row orders table to Parquet with a small row-group size, then report the row-group count and the min/max of order_ts in the first row group.

Input.

Parameter	Value
Rows	300,000
row_group_size	50,000 rows
Codec	ZSTD
Stat inspected	order_ts min/max in row group 0

Code.

import numpy as np
import pyarrow as pa
import pyarrow.parquet as pq

n = 300_000
rng = np.random.default_rng(3)
ts = np.sort(rng.integers(1_722_000_000, 1_724_000_000, size=n)).astype(np.int64)
orders = pa.table({
    "id":          pa.array(np.arange(n, dtype=np.int64)),
    "customer_id": pa.array(rng.integers(1, 50_000, size=n, dtype=np.int64)),
    "total_cents": pa.array(rng.integers(100, 500_00, size=n, dtype=np.int64)),
    "order_ts":    pa.array(ts),
})

pq.write_table(
    orders, "orders.parquet",
    row_group_size=50_000,        # 300k rows / 50k -> 6 row groups
    compression="zstd",
    write_statistics=True,        # min/max/null_count in the footer
)

pf = pq.ParquetFile("orders.parquet")
print("row groups:", pf.num_row_groups)

rg0 = pf.metadata.row_group(0)
ts_col_idx = orders.schema.names.index("order_ts")
stats = rg0.column(ts_col_idx).statistics
print("row group 0 order_ts min:", stats.min)
print("row group 0 order_ts max:", stats.max)
print("row group 0 rows:", rg0.num_rows)

Step-by-step explanation.

• row_group_size sets the pruning granularity. 300,000 rows at 50,000 per group yields 6 row groups. Each is independently skippable via its footer statistics — smaller groups prune finer but add footer overhead.
• write_statistics=True is what makes pushdown possible. Without min/max/null_count in the footer, a reader has nothing to prune with and must scan every row group. Statistics are the currency of predicate pushdown.
• The footer is read first, cheaply. pf.metadata parses only the footer — row-group and column-chunk metadata — without decoding any column data, exactly like ORC's footer.
• Sorted order_ts gives tight per-group ranges. Because we sorted the timestamps, row group 0's min/max is a narrow early window; a date-range query can skip the other five groups on their min/max alone.

Output.

Metric	Value
Row groups	6
Row group 0 rows	50,000
Row group 0 order_ts min	earliest timestamp (sorted)
Row group 0 order_ts max	boundary of first 50k window
Footer read cost	O(1) — no column decode

Rule of thumb. Always write Parquet with statistics on and a row-group size matched to your read pattern — large (128 MB+) for scan-heavy tables, smaller for selective lookups. Sort by the dominant predicate column so per-row-group min/max ranges are tight; unsorted data has overlapping ranges and prunes little, exactly as in ORC.

Worked example — predicate pushdown, projection, and the page index

Detailed explanation. Parquet pushdown works at three levels: row group, page (with the page index enabled), and bloom filter. Combined with column projection, a selective query reads a tiny fraction of the file. Write Parquet with the page index and a bloom filter, then run a selective query and reason about what each level skipped.

• Row-group skip. order_ts range predicate skips whole row groups via footer min/max.
• Page skip. The page index skips individual pages inside a matched row group.
• Bloom filter. customer_id equality skips row groups lacking the id.
• Projection. Read only the columns the query returns.

Question. Write Parquet with the page index and a customer_id bloom filter, run customer_id = 777 AND order_ts in [T0, T0+1d], and describe what each pruning level skipped.

Input.

Setting	Value
write_page_index	True
Bloom filter column	customer_id
Predicate	customer_id = 777 AND order_ts window
Projection	id, customer_id, order_ts

Code.

import numpy as np
import pyarrow as pa
import pyarrow.parquet as pq
import pyarrow.compute as pc
from pyarrow import dataset as ds

n = 1_000_000
rng = np.random.default_rng(21)
ts = np.sort(rng.integers(1_722_000_000, 1_724_000_000, size=n)).astype(np.int64)
orders = pa.table({
    "id":          pa.array(np.arange(n, dtype=np.int64)),
    "customer_id": pa.array(rng.integers(1, 200_000, size=n, dtype=np.int64)),
    "total_cents": pa.array(rng.integers(100, 500_00, size=n, dtype=np.int64)),
    "order_ts":    pa.array(ts),
})

pq.write_table(
    orders, "orders_idx.parquet",
    row_group_size=100_000,
    compression="zstd",
    write_statistics=True,
    write_page_index=True,                 # per-page min/max -> page-level skip
    write_bloom_filter=["customer_id"],    # equality skip on high-cardinality id
)

dataset = ds.dataset("orders_idx.parquet", format="parquet")
T0 = int(ts[len(ts) // 2])
predicate = (
    (pc.field("customer_id") == 777) &
    (pc.field("order_ts") >= T0) &
    (pc.field("order_ts") < T0 + 86400)
)
result = dataset.to_table(filter=predicate, columns=["id", "customer_id", "order_ts"])
print("matched rows:", result.num_rows)
print("columns decoded:", result.column_names)   # total_cents skipped

Step-by-step explanation.

• Row-group pruning runs first. The order_ts window is compared against each row group's footer min/max; only the group(s) overlapping the one-day window survive. On sorted data that is one or two groups out of ten.
• The page index prunes inside the survivor. write_page_index=True stores per-page min/max (column index) and page byte offsets (offset index), so within the surviving row group the reader skips straight to the pages whose order_ts range overlaps — it never decodes the earlier pages.
• The bloom filter handles the equality half. customer_id = 777 can't use min/max, so the per-column bloom filter (write_bloom_filter) answers "definitely not in this row group" and skips groups lacking 777.
• Projection composes with all of it. columns=[...] decodes three of four columns. The combined effect: a handful of pages of three columns, out of a million rows of four columns.

Output.

Pruning level	Mechanism	What it skipped
Row group	footer min/max on order_ts	non-overlapping groups
Page	page index (column + offset index)	non-matching pages in the survivor
Row group (equality)	customer_id bloom filter	groups lacking id 777
Column	projection	total_cents column chunk

Rule of thumb. Enable the page index (write_page_index=True) and bloom filters on the columns you filter by equality — they turn Parquet's row-group-level pruning into ORC-style page-level pruning. Combined with sorting and projection, a selective query touches a tiny, bounded slice of the file.

Worked example — nested data with repetition and definition levels

Detailed explanation. Parquet's Dremel model is what lets it store nested, repeated fields columnarly. A list<struct> field is shredded into flat leaf columns plus repetition and definition level streams. Write a table with a nested items column and read one nested leaf back, showing that the nested field is stored and scanned column-by-column.

• Nested schema. order(id, items: list<struct<sku, qty>>).
• Shredding. items.sku and items.qty become flat columns; repetition levels mark list boundaries; definition levels mark nulls/empties.
• Projection into nesting. Reading only items.sku reads only that leaf column plus its level streams.

Question. Write orders with a nested items list-of-structs to Parquet, then read back only the nested items.sku leaf and confirm the other leaf (items.qty) was not required.

Input.

Field	Type	Notes
id	int64	order id
items	list>	nested, repeated
Projection target	items.list.element.sku	one nested leaf

Code.

import pyarrow as pa
import pyarrow.parquet as pq

item_type = pa.struct([("sku", pa.string()), ("qty", pa.int32())])
orders = pa.table({
    "id": pa.array([1, 2, 3], type=pa.int64()),
    "items": pa.array(
        [
            [{"sku": "A1", "qty": 2}, {"sku": "B2", "qty": 1}],  # order 1: 2 items
            [{"sku": "C3", "qty": 5}],                            # order 2: 1 item
            [],                                                    # order 3: empty list
        ],
        type=pa.list_(item_type),
    ),
})

pq.write_table(orders, "orders_nested.parquet", compression="zstd")

# Read only the nested sku leaf. Parquet's Dremel shredding stores it as a
# flat column with repetition/definition levels, so qty is never decoded.
sku_only = pq.read_table("orders_nested.parquet", columns=["items.list.element.sku"])
print(sku_only.schema)
print(sku_only.to_pylist())
# The empty list (order 3) and per-order item counts are reconstructed from
# the repetition/definition level streams, not from a materialised row image.

Step-by-step explanation.

• The list<struct> is shredded into flat leaves. items.sku and items.qty each become their own column. There is no "row image" on disk — the nested structure is reconstructed from level streams at read time.
• Repetition levels rebuild the lists. A repetition level says "this value starts a new list vs continues the current one," which is how the reader knows order 1 had two items and order 2 had one — purely from the level stream.
• Definition levels encode nulls and empties. Order 3's empty list is represented in the definition-level stream, not as a stored row; the reader materialises [] from the level, costing almost no bytes.
• Projection reaches into nesting. columns=["items.list.element.sku"] decodes only the sku leaf column and its levels; the qty leaf is never touched — the same projection win as flat columns, extended to nested fields.

Output.

id	items.sku (projected)	qty decoded?
1	["A1", "B2"]	no
2	["C3"]	no
3		no

Rule of thumb. Parquet stores nested, repeated data columnarly via Dremel repetition + definition levels, so you get projection and pushdown inside nested fields — read one leaf of a deep struct without materialising the rest. This is a structural advantage over formats that would explode nested data into rows.

Senior interview question on Apache Parquet

A senior interviewer might ask: "You're the platform owner for an Iceberg lakehouse whose bronze tables are 20-TB of unsorted Parquet. Analysts complain that WHERE event_date = ? AND user_id = ? queries scan far too much data. Walk me through how you'd re-tune the Parquet files — sort order, row-group size, page index, bloom filters, ZSTD — and how you'd prove the pruning win before compacting the whole table."

Solution Using sort order, right-sized row groups, page index, and bloom filters

# retune_parquet.py — rewrite Iceberg data files with a pruning-optimised layout
import pyarrow as pa
import pyarrow.parquet as pq
import pyarrow.compute as pc
from pyarrow import dataset as ds

src = ds.dataset("events_bronze/", format="parquet").to_table()

# 1) Sort by the dominant range predicate first, then the equality key second.
order = pc.sort_indices(
    src, sort_keys=[("event_date", "ascending"), ("user_id", "ascending")]
)
sorted_tbl = src.take(order)

# 2) Rewrite: right-sized row groups + page index + bloom filter + ZSTD.
pq.write_table(
    sorted_tbl, "events_silver.parquet",
    row_group_size=1_000_000,              # ~128 MB target for scan efficiency
    compression="zstd",
    write_statistics=True,
    write_page_index=True,                 # ORC-style page-level pruning
    write_bloom_filter=["user_id"],        # high-cardinality equality skip
)

# 3) Prove the win: same predicate, count how much each layout decodes.
def matched(path):
    d = ds.dataset(path, format="parquet")
    predicate = (
        (pc.field("event_date") == 20260801) &
        (pc.field("user_id") == 4242)
    )
    return d.to_table(filter=predicate, columns=["event_id", "event_date", "user_id"]).num_rows

print("bronze matched:", matched("events_bronze/"))
print("silver matched:", matched("events_silver.parquet"))  # identical rows, far less scanned

pf = pq.ParquetFile("events_silver.parquet")
print("row groups:", pf.num_row_groups)

Step-by-step trace.

Step	Change	Pruning effect
Sort by (event_date, user_id)	ordered layout	tight per-row-group + per-page min/max
row_group_size = 1M rows	~128 MB groups	scan-efficient; still prunable
write_page_index=True	per-page min/max	skip pages inside a matched group
bloom on user_id	membership test	skip groups lacking the user
ZSTD	modern codec	smaller files, fast decode

After the rewrite, event_date = 20260801 prunes to the contiguous row groups whose sorted date range overlaps that day; inside those groups the page index skips to the pages covering user_id = 4242; the bloom filter eliminates groups that can't contain the user at all. The correctness check is that matched(bronze) and matched(silver) return the same row count — same answer, dramatically less data scanned — which you can confirm before compacting the full 20 TB by testing on one partition.

Output:

Metric	Bronze (unsorted)	Silver (tuned)
Row-group pruning on date	little (overlapping)	aggressive (disjoint)
Page-level pruning	none (no page index)	yes (page index)
user_id equality skip	none	bloom-filter skip
Bytes scanned per query	~O(table)	~O(matching pages)
Rows returned	identical	identical

Why this works — concept by concept:

• Sort order — like ORC, Parquet's min/max only prunes when ranges are tight. Sorting by event_date then user_id makes both row-group and page statistics selective, which is the biggest lever.
• Row-group sizing — ~128 MB groups balance scan efficiency (few footers) against pruning granularity; the page index restores fine-grained skipping inside those large groups.
• Page index — the column-index + offset-index turns row-group-level pruning into page-level pruning, the feature that closes ORC's historical fine-pruning advantage.
• Bloom filter — the "definitely not here" test for the high-cardinality user_id equality predicate that min/max cannot serve.
• Cost — one sort + rewrite pass (O(n log n), one-time) buys tight pruning on every future query; validated on a single partition before committing the full-table compaction. Per-query scanned bytes drop from ~O(table) to ~O(matching pages). This is why the lakehouse table formats standardised on Parquet — the pruning primitives compose cleanly with their file-level statistics.

ETL
Topic — etl
ETL problems on lakehouse compaction and layout

Practice →

Data Processing Topic — data-processing Data-processing problems on nested and columnar data

Practice →

---

4. Lance — the modern columnar + vector format

Fast random access, zero-copy versioning, and vector indexes — the columnar format built for ML/AI

The mental model in one line: the lance format is a modern columnar format (Rust, built on Apache Arrow, from the LanceDB team) whose design center is not full-scan analytics but fast random access — a take of scattered row ids is cheap, not a whole-row-group decode — layered with zero-copy versioning (every write produces a new manifest so you get reproducible dataset versions and time travel for free) and native vector search (scalar btree/bitmap indexes plus IVF_PQ / HNSW approximate-nearest-neighbor indexes), which together make it the columnar format built for ML/AI training data, embeddings, and RAG rather than a Parquet replacement for BI. When the workload is "sample scattered rows, re-rank by vector similarity, and reproduce exactly which rows a model trained on," Lance is the format that serves all three from one file.

The Lance design center — why another format.

• Random access is a first-class read pattern. Parquet and ORC optimise contiguous scans; a point lookup forces decoding a whole row group / stripe. Lance stores data so that reading row ids [2, 88, 4021] fetches roughly those rows, which is what ML sampling, shuffling, and vector re-ranking need.
• Versioning belongs in the format. ML reproducibility demands "which exact rows did version 7 of this dataset contain?" Lance answers it natively — no external table format required.
• Vectors are native. Embeddings are the dominant new column type; Lance indexes them for ANN search instead of bolting on a separate vector database for the index.

The Lance format anatomy.

• Dataset directory. A Lance dataset is a directory of data files plus a _versions/ manifest history and _indices/ for secondary indexes.
• Fragments. Data is grouped into fragments (like row groups) of one or more data files; fragments are the unit of append and deletion.
• Manifest + versions. Each commit writes a new manifest describing which fragments and indices make up that version. Old manifests remain, so any prior version is directly readable — this is the versioning + time-travel primitive.
• Row addressability. Rows are addressable by id within fragments, which is what makes take(row_ids) cheap relative to a scan format.

Fast random access — the headline win.

• take(row_ids). Fetches specific rows by id without scanning the fragments between them — O(rows requested), not O(fragment). This is the operation Parquet/ORC do poorly.
• Why ML needs it. Training shuffles and mini-batch sampling touch scattered rows every epoch; a scan format re-reads far more than requested. Vector re-ranking fetches the top-k candidate rows' full payloads by id.

Zero-copy versioning + time travel.

• Every write is a new version. append, overwrite, and delete each create a new manifest; the data files they don't touch are shared (zero-copy), not rewritten.
• Checkout by version. lance.dataset(path, version=N) reads the dataset exactly as it existed at version N — reproducible training inputs, cheap experiment branching, and rollback.

Vector search — native ANN.

• Scalar indexes. Btree / bitmap indexes on scalar columns for fast filtered lookups (WHERE label = 'cat').
• Vector indexes. IVF_PQ (inverted file + product quantization) and HNSW build approximate-nearest-neighbor indexes on an embedding column, so top-k similarity search is sub-linear instead of a brute-force scan.
• Filtered vector search. Combine a scalar predicate with a vector query — "nearest neighbors among rows where source = 'docs'" — in one call.

Common interview probes on Lance.

• "What does Lance optimise that Parquet doesn't?" — random access (take by row id) plus native versioning and vector indexes.
• "How does Lance do versioning?" — manifest-per-commit; old manifests remain readable for time travel; unchanged data files are shared.
• "What vector indexes does Lance support?" — IVF_PQ and HNSW for ANN, plus scalar btree/bitmap for filtered search.
• "When would you NOT use Lance?" — a pure BI scan workload with a mature Parquet/Iceberg lakehouse and no ML/vector or random-access requirement.

Worked example — writing a Lance dataset and random access with take

Detailed explanation. The canonical first Lance task: write a dataset, then fetch scattered rows by id with take — the operation Lance is built for. Contrast the cost model with Parquet, where the same scattered fetch would decode whole row groups. Walk through it.

• Write. lance.write_dataset(arrow_table, path).
• Open. lance.dataset(path).
• Random access. dataset.take([ids], columns=[...]) fetches those rows directly.

Question. Write a 200,000-row training table to Lance, then fetch 5 scattered row ids and reason about why this is cheaper than the Parquet equivalent.

Input.

Parameter	Value
Rows	200,000
Columns	id, label, features (list)
Access	take([2, 500, 88_000, 150_003, 199_999])
Cost model	O(rows requested), not O(fragment)

Code.

import numpy as np
import pyarrow as pa
import lance

n = 200_000
rng = np.random.default_rng(5)
train = pa.table({
    "id":       pa.array(np.arange(n, dtype=np.int64)),
    "label":    pa.array(rng.integers(0, 10, size=n, dtype=np.int32)),
    "features": pa.array(rng.standard_normal((n, 8)).tolist(),
                         type=pa.list_(pa.float32(), 8)),
})

lance.write_dataset(train, "train.lance", mode="overwrite")

ds = lance.dataset("train.lance")
print("version:", ds.version, "| rows:", ds.count_rows())

# Random access: fetch 5 scattered rows by id, only the columns needed.
rows = ds.take([2, 500, 88_000, 150_003, 199_999], columns=["id", "label"])
print(rows.to_pylist())
# In Parquet this scattered fetch would decode whichever row groups contain
# these ids in full; Lance addresses the rows directly.

Step-by-step explanation.

• lance.write_dataset accepts an Arrow table directly. Like ORC and Parquet, Lance interoperates with Arrow, so the same in-memory table serialises to Lance. mode="overwrite" creates version 1 (or replaces the dataset).
• take([ids]) is the random-access primitive. It resolves each id to its fragment + offset and fetches roughly those rows, rather than scanning the fragments between them. This is O(rows requested).
• The Parquet contrast is the teaching point. In Parquet, ids 2 and 199_999 live in different row groups; a scattered take decodes each containing row group in full to extract one row — dramatically more work than Lance's direct addressing.
• Column projection still applies. columns=["id","label"] skips the features column, composing projection with random access.

Output.

Operation	Lance	Parquet (contrast)
Fetch 5 scattered ids	~5 rows' worth of IO	multiple full row-group decodes
Cost model	O(rows requested)	O(row groups touched)
Projection	yes (columns=)	yes (columns=)
Best for	ML sampling, re-ranking	full-column scans

Rule of thumb. Reach for Lance the moment your read pattern is dominated by scattered point lookups — ML mini-batch sampling, shuffling, vector re-ranking. take(row_ids) is O(rows requested) in Lance and O(row groups touched) in Parquet/ORC, and that gap is the entire reason the format exists.

Worked example — zero-copy versioning and time travel

Detailed explanation. Lance versions natively: each write creates a new manifest, unchanged data files are shared, and any prior version is directly readable by number. This gives reproducible training inputs and cheap rollback without an external table format. Walk through append + overwrite + checkout.

• v1. Initial write.
• v2. Append new rows (a new fragment; v1's fragments are shared, not copied).
• v3. Overwrite (new manifest; old data files remain for time travel).
• Checkout. Read any version by number.

Question. Create a Lance dataset, append rows to make version 2, then read both the latest version and version 1 to demonstrate time travel.

Input.

Step	Operation	Resulting version
1	write 3 rows	v1
2	append 2 rows	v2
3	checkout v1	reads original 3 rows

Code.

import pyarrow as pa
import lance

v1_data = pa.table({"id": [1, 2, 3], "label": ["a", "b", "c"]})
lance.write_dataset(v1_data, "cats.lance", mode="overwrite")   # -> version 1

# Append: creates version 2; version 1's data file is SHARED, not rewritten.
extra = pa.table({"id": [4, 5], "label": ["d", "e"]})
lance.write_dataset(extra, "cats.lance", mode="append")        # -> version 2

latest = lance.dataset("cats.lance")
print("latest version:", latest.version, "| rows:", latest.count_rows())   # 2 | 5

# Time travel: read the dataset exactly as it was at version 1.
old = lance.dataset("cats.lance", version=1)
print("v1 rows:", old.count_rows())                            # 3
print("v1 data:", old.to_table().to_pylist())

# Inspect the version history.
for v in latest.versions():
    print("version", v["version"], "committed at", v["timestamp"])

Step-by-step explanation.

• Each write bumps the version. The initial overwrite is version 1; the append is version 2. Versions are integers, monotonically increasing per commit.
• Appends share unchanged data (zero-copy). Version 2's manifest references version 1's data file plus the new fragment. Version 1 is not rewritten — appends are O(new data), not O(dataset).
• Time travel reads an old manifest. lance.dataset(path, version=1) loads version 1's manifest, which points only at the original data file, so it returns exactly the 3 original rows — reproducibly, forever (until you vacuum old versions).
• This is the reproducibility primitive ML needs. "Model X trained on dataset version 7" is answerable directly: check out version 7 and you have the exact rows. No external snapshotting.

Output.

Read	Version	Rows returned
latest	2	5 (ids 1–5)
time travel	1	3 (ids 1–3)
version history	—	v1 and v2 with timestamps

Rule of thumb. Lance's in-format versioning makes reproducible datasets and rollback free — pin a training run to a version number and you can always reconstruct its exact inputs. Appends are zero-copy, so versioning costs storage only for changed data, and a vacuum step reclaims old versions when you no longer need to time-travel to them.

Worked example — vector search with an IVF_PQ index

Detailed explanation. Lance's differentiator over Parquet/ORC is native vector search. Build an IVF_PQ approximate-nearest-neighbor index on an embedding column, then run a top-k similarity query — optionally filtered by a scalar predicate. This is the vector search format capability that keeps embeddings and their index in one place. Walk through it.

• Embedding column. A fixed-size-list float column (list<float, D>).
• Build index. create_index(column, index_type="IVF_PQ", num_partitions, num_sub_vectors).
• Query. to_table(nearest={column, q, k}) returns the top-k nearest rows.
• Filtered ANN. Add a filter= to search within a subset.

Question. Build an IVF_PQ index on a 128-dim embedding column and run a top-5 nearest-neighbor query, then run the same query filtered to one label.

Input.

Setting	Value
Rows	50,000
Embedding dim	128
Index	IVF_PQ (num_partitions=256, num_sub_vectors=16)
Query	top-5 nearest to a random vector
Filtered query	same, WHERE label = 3

Code.

import numpy as np
import pyarrow as pa
import lance

n, dim = 50_000, 128
rng = np.random.default_rng(9)
emb = rng.standard_normal((n, dim)).astype(np.float32)
data = pa.table({
    "id":     pa.array(np.arange(n, dtype=np.int64)),
    "label":  pa.array(rng.integers(0, 10, size=n, dtype=np.int32)),
    "vector": pa.array(emb.tolist(), type=pa.list_(pa.float32(), dim)),
})

ds = lance.write_dataset(data, "embeddings.lance", mode="overwrite")

# Build an approximate-nearest-neighbor index on the vector column.
ds.create_index(
    "vector",
    index_type="IVF_PQ",
    num_partitions=256,       # IVF coarse quantizer cells
    num_sub_vectors=16,       # PQ sub-vector count (128 / 16 = 8 dims each)
)

q = rng.standard_normal(dim).astype(np.float32)

# Top-5 nearest neighbors across the whole dataset.
knn = ds.to_table(nearest={"column": "vector", "q": q, "k": 5}, columns=["id", "label"])
print("top-5 ids:", knn.column("id").to_pylist())

# Filtered ANN: nearest neighbors AMONG rows where label = 3.
knn_filtered = ds.to_table(
    nearest={"column": "vector", "q": q, "k": 5},
    filter="label = 3",
    columns=["id", "label"],
)
print("top-5 ids (label=3):", knn_filtered.column("id").to_pylist())

Step-by-step explanation.

• IVF_PQ makes ANN sub-linear. IVF (inverted file) partitions the vectors into num_partitions cells so a query only searches the nearest cells; PQ (product quantization) compresses each vector into num_sub_vectors codes so distance computation is fast and memory-light. Together they turn brute-force O(n·d) into approximate sub-linear search.
• The index lives in the dataset. create_index writes into the Lance dataset's _indices/ — the embeddings and their ANN index are one artifact, versioned together, no separate vector database required.
• nearest={...} is the query primitive. It returns the top-k rows by vector distance, decoding only the projected columns for those k rows — random access again, now driven by similarity instead of explicit ids.
• Filtered ANN composes scalar + vector. filter="label = 3" restricts the search to matching rows (accelerated by a scalar index if present), so you get "nearest neighbors within a subset" in a single call — the pattern RAG and recommendation re-ranking need.

Output.

Query	Returns	Mechanism
top-5 nearest	5 ids by vector distance	IVF_PQ ANN
top-5 nearest, label=3	5 ids, all label 3	scalar filter + ANN
index storage	inside the Lance dataset	versioned with the data

Rule of thumb. If embeddings and similarity search are core to the workload, Lance keeps the vectors, their IVF_PQ/HNSW index, and the rest of the columns in one versioned format — no separate vector store to sync. Parquet and ORC can store an embedding column but cannot index it for ANN, which is the sharpest capability line between the old and new columnar formats.

Senior interview question on Lance

A senior interviewer might ask: "Your team ships a RAG system whose document embeddings currently live in Parquet on S3, with a separate vector database re-loaded nightly for similarity search, and no reproducible link between a model version and the exact training rows. Walk me through redesigning this on Lance — the write path, the vector index, the versioning contract, and the filtered-search query — and name the trade-off you accept versus keeping Parquet + an external vector DB."

Solution Using Lance for a versioned embedding + training store

# rag_store.py — a single Lance dataset for embeddings, ANN index, and versioning
import numpy as np
import pyarrow as pa
import lance

DIM = 384

def build_initial(path: str, rows: list[dict]) -> "lance.LanceDataset":
    tbl = pa.table({
        "doc_id":    pa.array([r["doc_id"] for r in rows], type=pa.int64()),
        "source":    pa.array([r["source"] for r in rows]),
        "text":      pa.array([r["text"] for r in rows]),
        "embedding": pa.array([r["embedding"] for r in rows],
                              type=pa.list_(pa.float32(), DIM)),
    })
    ds = lance.write_dataset(tbl, path, mode="overwrite")     # version 1
    ds.create_scalar_index("source", index_type="BTREE")      # fast filtered search
    ds.create_index("embedding", index_type="IVF_PQ",
                    num_partitions=512, num_sub_vectors=48)     # ANN index
    return ds

def ingest_new_docs(path: str, rows: list[dict]) -> int:
    tbl = pa.table({
        "doc_id":    pa.array([r["doc_id"] for r in rows], type=pa.int64()),
        "source":    pa.array([r["source"] for r in rows]),
        "text":      pa.array([r["text"] for r in rows]),
        "embedding": pa.array([r["embedding"] for r in rows],
                              type=pa.list_(pa.float32(), DIM)),
    })
    lance.write_dataset(tbl, path, mode="append")             # new version, zero-copy
    return lance.dataset(path).version

def search(path: str, query_vec, k: int = 5, source: str | None = None,
           version: int | None = None):
    ds = lance.dataset(path, version=version)                 # pin a version for reproducibility
    kw = {"nearest": {"column": "embedding", "q": query_vec, "k": k},
          "columns": ["doc_id", "source", "text"]}
    if source is not None:
        kw["filter"] = f"source = '{source}'"                 # filtered ANN
    return ds.to_table(**kw)

Step-by-step trace.

Layer	Mechanism	Replaces
Embedding storage	Lance list<float,384> column	Parquet embedding column
ANN index	IVF_PQ inside the dataset	external vector DB nightly reload
Filtered search	scalar BTREE on source + ANN	app-side post-filter
Reproducibility	version= checkout	no link between model + rows
Ingest	mode="append" (zero-copy)	full rewrite / re-index

After the redesign, one Lance dataset holds the documents, their embeddings, the IVF_PQ ANN index, and a scalar btree on source. Nightly ingestion is a zero-copy append that bumps the version; a model trained on version 7 can always reconstruct its exact inputs via lance.dataset(path, version=7); and similarity search — including filtered "search within source = 'docs'" — runs directly against the dataset with no external vector database to keep in sync.

Output:

Capability	Parquet + external vector DB	Lance
Embedding storage	Parquet	Lance column
Similarity search	separate system, nightly reload	native, in-format
Filtered ANN	app-side or DB-specific	one call (scalar index + ANN)
Model ↔ data version link	none	version= checkout
Ingest cost	rewrite + re-index	zero-copy append

Why this works — concept by concept:

• Random access — top-k ANN returns scattered candidate rows, and Lance's take-by-id addressing fetches their full payloads cheaply; a Parquet-backed store would decode whole row groups per candidate.
• In-format vector index — IVF_PQ lives in the Lance dataset's _indices/, so embeddings and their ANN index are one versioned artifact instead of a Parquet file plus a separately-synced vector database.
• Zero-copy versioning — each nightly append creates a new manifest sharing prior data files, so version pinning links a model to its exact training rows without snapshot copies.
• Filtered ANN — combining a scalar btree on source with the vector query serves "nearest within a subset" in one call, the core RAG retrieval pattern.
• Cost — the trade-off is ecosystem maturity: you give up Parquet's universal engine support and lean on Lance's newer integrations. In return you delete an entire external system (the vector DB) and its sync job, and gain reproducibility. For an ML/RAG workload dominated by random access + vectors, that is the right trade; for a pure BI scan workload it is not.

Design
Topic — design
Design problems on embedding and feature stores

Practice →

Data Processing Topic — data-processing Data-processing problems on vectors and random access

Practice →

---

5. Head-to-head — benchmarks and the decision matrix

Benchmarks, the decision matrix, and migration — analytics vs ML vs Hive legacy

The mental model in one line: there is no single winner in a file format comparison — Parquet wins full-scan analytics and ecosystem breadth, ORC wins Hive-native ACID and edges compression on wide low-cardinality fact tables, and Lance wins random access, versioning, and vector search — so the head-to-head is not "which format is fastest" but "which axis dominates my workload," and the senior skill is running a small benchmark on your own data, reading the axis scores off it, and defending the migration cost of the pick. Any answer that names a universal winner is the wrong answer; the right answer is a decision matrix keyed on read pattern.

The benchmark axes that matter.

• Full-scan throughput. Rows/sec for SELECT agg(col) ... GROUP BY. Parquet and ORC lead; Lance is competitive but not its design center.
• Point-query / random access. Latency for take(scattered ids). Lance leads by a wide margin; Parquet/ORC pay a whole-row-group decode.
• Compression ratio. Bytes on disk for the same data. ORC often edges Parquet on wide low-cardinality tables; ZSTD narrows the gap; Lance is competitive.
• Write throughput. Rows/sec to write + index. Parquet/ORC are fast to write; Lance adds index-build cost when you create ANN indexes.
• Vector search. Native ANN or not. Only Lance; Parquet/ORC need an external index.

The decision matrix — the artifact to memorise.

• Analytics / BI / lakehouse → Parquet. Widest support, strong scans, written by Delta/Iceberg/Hudi. The default unless an axis overrides.
• Hive-legacy / transactional warehouse → ORC. Native ACID, native metastore, compression edge on fact tables.
• ML / embeddings / RAG / random access → Lance. Point lookups, versioning, vector search in one format.
• Mixed platform → standardise the catalog, not the file. Use Iceberg/Unity as the table layer and let each domain pick its file format underneath where the table format allows.

Migration paths — the cost is real.

• ORC → Parquet (Hive → Spark/lakehouse). Common when moving off Hadoop. Arrow bridges the read/write; the work is repartitioning, sorting, and re-registering in the new catalog. Budget for a full rewrite pass plus catalog migration.
• Parquet → Lance (feature/embedding store). Common when an ML team outgrows "Parquet + external vector DB." Read Parquet into Arrow, write_dataset to Lance, build indexes. The win is deleting the external vector store; the cost is newer tooling.
• Never migrate without a benchmark. Prove the axis win on a sample partition before rewriting a petabyte.

Interview signals on the head-to-head.

• Name the axis before the format — "for a random-access ML workload, Lance; for scan analytics, Parquet."
• Quantify the trade-off — ecosystem breadth vs random access vs compression, not vibes.
• Standardise the catalog, not the file format, across a heterogeneous platform.

Common interview probes on the comparison.

• "Is ORC or Parquet better?" — neither universally; ORC for Hive ACID + compression, Parquet for ecosystem + scans.
• "When does Lance beat Parquet?" — random access, versioning, vector search; not pure BI scans.
• "How do you decide?" — run a small benchmark on your data across the axes; read the winner off the dominant axis.
• "What's the migration cost?" — a full rewrite pass plus catalog re-registration; validate on a sample first.

Worked example — a benchmark harness across the three formats

Detailed explanation. The senior move is to benchmark on your own data rather than trust a leaderboard. Build a small harness that writes the same Arrow table to ORC, Parquet, and Lance, then measures file size, full-scan time, and scattered random-access time. Walk through it.

• Same data, three writers. One Arrow table → ORC, Parquet, Lance.
• Measure size. Bytes on disk per format.
• Measure full scan. Time an aggregate over one column.
• Measure random access. Time a take of scattered ids.

Question. Write a 500,000-row table to all three formats and measure on-disk size, full-scan time, and scattered random-access time.

Input.

Metric	ORC	Parquet	Lance
On-disk size	measured	measured	measured
Full-scan time	fast	fast	competitive
Random-access time	slow (row-group decode)	slow (row-group decode)	fast (take)

Code.

import os, time
import numpy as np
import pyarrow as pa
import pyarrow.orc as orc
import pyarrow.parquet as pq
import pyarrow.compute as pc
import lance

n = 500_000
rng = np.random.default_rng(1)
tbl = pa.table({
    "id":     pa.array(np.arange(n, dtype=np.int64)),
    "amount": pa.array(rng.integers(1, 10_000, size=n, dtype=np.int64)),
    "cat":    pa.array(rng.choice(["a", "b", "c", "d"], size=n)),
})

orc.write_table(tbl, "bench.orc", compression="ZSTD")
pq.write_table(tbl, "bench.parquet", compression="zstd")
lance.write_dataset(tbl, "bench.lance", mode="overwrite")

def size_mb(path):
    if os.path.isdir(path):
        return sum(os.path.getsize(os.path.join(r, f))
                   for r, _, fs in os.walk(path) for f in fs) / 1024 / 1024
    return os.path.getsize(path) / 1024 / 1024

def full_scan_sum(read_amount):
    t = time.perf_counter()
    total = pc.sum(read_amount()).as_py()
    return time.perf_counter() - t, total

ids = rng.integers(0, n, size=1000).tolist()   # 1000 scattered ids

def random_access_orc_pq(path, fmt):
    from pyarrow import dataset as pds
    d = pds.dataset(path, format=fmt)
    t = time.perf_counter()
    d.to_table(filter=pc.field("id").isin(ids), columns=["id", "amount"])
    return time.perf_counter() - t

def random_access_lance(path):
    d = lance.dataset(path)
    t = time.perf_counter()
    d.take(ids, columns=["id", "amount"])
    return time.perf_counter() - t

print(f"{'format':8s} {'size_MB':>8s} {'scan_s':>8s} {'rand_s':>8s}")
for name, path, fmt in [("orc","bench.orc","orc"), ("parquet","bench.parquet","parquet")]:
    scan_s, _ = full_scan_sum(lambda: __import__('pyarrow.dataset', fromlist=['dataset'])
                              .dataset(path, format=fmt).to_table(columns=["amount"]).column("amount"))
    rand_s = random_access_orc_pq(path, fmt)
    print(f"{name:8s} {size_mb(path):8.2f} {scan_s:8.4f} {rand_s:8.4f}")

lance_scan_s, _ = full_scan_sum(lambda: lance.dataset("bench.lance").to_table(columns=["amount"]).column("amount"))
lance_rand_s = random_access_lance("bench.lance")
print(f"{'lance':8s} {size_mb('bench.lance'):8.2f} {lance_scan_s:8.4f} {lance_rand_s:8.4f}")

Step-by-step explanation.

• The same Arrow table removes data-modeling as a variable. Any size or speed difference is attributable to the format, not to different data — the only fair way to benchmark formats.
• Size is a direct os.path.getsize (or directory walk for Lance). Lance and (partitioned) datasets are directories, so the harness sums file sizes; ORC/Parquet single files are one getsize.
• Full scan sums one column. All three do projection, so the scan touches only amount; the timing reflects decode throughput on a single column.
• Random access is the discriminating test. For ORC/Parquet the harness uses an isin filter (which still decodes containing row groups); for Lance it uses take(ids) (direct addressing). The gap on scattered ids is the point of the whole comparison.

Output.

Format	On-disk size	Full scan	Random access (1000 scattered)
ORC	small (compression edge)	fast	slow (row-group decode)
Parquet	small	fast	slow (row-group decode)
Lance	competitive	competitive	fast (take by id)

Rule of thumb. Benchmark on your own data, not a leaderboard, and measure the axis your workload actually stresses. If random access dominates, the scan-format numbers are irrelevant; if full-scan analytics dominates, the random-access gap is irrelevant. The harness makes the trade-off concrete instead of theoretical.

Worked example — the workload-to-format decision matrix

Detailed explanation. Codify the decision as a small function so the interview answer is reproducible. Given a workload's dominant read pattern and constraints, emit the format and the reason. Walk through four canonical workloads.

• BI dashboard. Scan-heavy, needs broad support → Parquet.
• Hive ACID reporting. Transactional, Hive metastore → ORC.
• ML training + vectors. Random access + ANN → Lance.
• Mixed lakehouse. Standardise the table format (Iceberg), pick file format per domain.

Question. Map four workloads to formats with the deciding axis for each.

Input.

Workload	Dominant axis	Constraint
BI dashboard	full scan	widest support
Hive reporting	transactional	Hive metastore
ML train + vectors	random access	ANN + versioning
Mixed platform	heterogeneous	one catalog

Code.

def decide(read_pattern: str, hive_acid: bool, vectors: bool, mixed: bool) -> tuple[str, str]:
    if mixed:
        return "iceberg-catalog + per-domain files", "standardise the catalog, not the file format"
    if vectors or read_pattern == "random":
        return "lance", "random access + native ANN + in-format versioning"
    if hive_acid:
        return "orc", "Hive-native ACID + compression edge on fact tables"
    return "parquet", "widest ecosystem + strong scans (the default)"

cases = [
    ("scan",   False, False, False),   # BI dashboard
    ("scan",   True,  False, False),   # Hive reporting
    ("random", False, True,  False),   # ML train + vectors
    ("scan",   False, False, True),    # mixed platform
]
for c in cases:
    print(c, "->", decide(*c))

Step-by-step explanation.

• Mixed platforms resolve at the catalog layer. When the platform is heterogeneous, the answer isn't a single file format — it's a table format (Iceberg/Unity) that lets each domain store the file format it needs while presenting one governed catalog.
• Vectors / random access resolve to Lance. This branch is checked before the scan defaults because it is the most format-specific requirement.
• Hive ACID resolves to ORC. A hard ecosystem constraint that overrides the Parquet default.
• Everything else is Parquet. The default catches scan-analytics workloads with no overriding axis — which is most of them.

Output.

Workload	Format	Deciding axis
BI dashboard	Parquet	ecosystem + scans
Hive reporting	ORC	Hive ACID
ML train + vectors	Lance	random access + vectors
Mixed platform	Iceberg catalog + per-domain files	heterogeneity

Rule of thumb. The decision matrix has one default (Parquet) and three overrides (Lance for random access/vectors, ORC for Hive ACID, a table format for mixed platforms). Name the override that fires — or say "no override fires, so Parquet" — and you have defended the pick in one sentence.

Worked example — migrating Parquet to Lance for a feature store

Detailed explanation. A common 2026 migration: an ML team's feature store outgrows "Parquet + external vector DB" and moves to Lance for random access, versioning, and native ANN. The migration reads Parquet into Arrow and writes Lance, then builds indexes — validated on a sample first. Walk through it.

• Read. Parquet → Arrow table (or stream in batches for large data).
• Write. Arrow → Lance dataset.
• Index. Build the IVF_PQ + scalar indexes Lance adds over Parquet.
• Validate. Row-count + a spot similarity query before cutover.

Question. Migrate a Parquet feature store with an embedding column to Lance, build the ANN index, and validate the migration before cutover.

Input.

Step	Operation
1	read Parquet feature files into Arrow
2	write Lance dataset
3	build IVF_PQ index on embedding
4	validate row count + sample ANN query

Code.

# migrate_parquet_to_lance.py
import pyarrow.parquet as pq
import pyarrow.dataset as pds
import lance

SRC = "feature_store_parquet/"     # directory of Parquet files
DST = "feature_store.lance"

# 1) Stream the Parquet dataset in batches (avoid loading all of it into RAM).
src_ds = pds.dataset(SRC, format="parquet")

# 2) Write to Lance directly from the Arrow record-batch reader.
reader = src_ds.scanner(batch_size=64_000).to_reader()
lance.write_dataset(reader, DST, mode="overwrite")

dst_ds = lance.dataset(DST)

# 3) Build the indexes Lance adds over Parquet.
dst_ds.create_scalar_index("source", index_type="BTREE")
dst_ds.create_index("embedding", index_type="IVF_PQ",
                    num_partitions=512, num_sub_vectors=48)

# 4) Validate before cutover: row counts must match; ANN query must return k rows.
src_rows = src_ds.count_rows()
dst_rows = dst_ds.count_rows()
assert src_rows == dst_rows, f"row mismatch: {src_rows} != {dst_rows}"

import numpy as np
q = np.random.default_rng(0).standard_normal(384).astype("float32")
hits = dst_ds.to_table(nearest={"column": "embedding", "q": q, "k": 5}, columns=["doc_id"])
assert hits.num_rows == 5
print(f"migrated {dst_rows} rows; ANN returns {hits.num_rows} hits — safe to cut over")

Step-by-step explanation.

• Stream, don't load. scanner(batch_size=...).to_reader() yields Arrow record batches, and lance.write_dataset consumes the reader incrementally — so a feature store larger than RAM migrates without OOM.
• Lance adds what Parquet couldn't index. After the copy, create_index("embedding", "IVF_PQ", ...) builds the ANN index that previously required a separate vector database; create_scalar_index("source", ...) accelerates filtered search.
• Validation gates the cutover. The row-count assert catches a truncated copy; the sample ANN query confirms the index is queryable and returns k hits. Only after both pass do you repoint readers.
• The external vector DB disappears. Post-migration, similarity search runs against the Lance dataset, so the nightly reload job into the external store is deleted — the operational win that justifies the migration.

Output.

Migration stage	Check	Result
Copy	streamed batches	no OOM on large store
Row count	src == dst	equal (assert passes)
ANN index	IVF_PQ built	queryable
Sample query	returns k=5	index validated
Cutover	repoint readers	external vector DB retired

Rule of thumb. Migrate formats by streaming Arrow record batches (never a full load), rebuild the target format's indexes explicitly, and gate the cutover on a row-count assert plus a functional query. Validate on a sample partition before committing the full dataset — a format migration you can't roll back is a format migration you shouldn't start.

Senior interview question on the head-to-head

A senior interviewer might ask: "Leadership wants a one-line policy: 'all data at our company will be stored in format X.' You think that's a mistake. Give me the counter-proposal — the per-workload format assignment, the benchmark you'd run to defend it, the migration cost for the two teams that would move, and the single thing you would standardise across all of them."

Solution Using a benchmark-defended, workload-keyed format policy

# format_policy.py — the counter-proposal, encoded and benchmarked
from dataclasses import dataclass

@dataclass
class Team:
    name: str
    read_pattern: str      # "scan" | "random"
    hive_acid: bool
    vectors: bool

def assign(t: Team) -> str:
    if t.vectors or t.read_pattern == "random":
        return "lance"
    if t.hive_acid:
        return "orc"
    return "parquet"

teams = [
    Team("BI / analytics",   "scan",   hive_acid=False, vectors=False),  # parquet (stays)
    Team("Warehouse / Hive", "scan",   hive_acid=True,  vectors=False),  # orc (stays)
    Team("ML / RAG",         "random", hive_acid=False, vectors=True),   # lance (migrates)
    Team("Fraud features",   "random", hive_acid=False, vectors=False),  # lance (migrates)
]

for t in teams:
    print(f"{t.name:18s} -> {assign(t)}")

# The one thing to standardise across all of them:
STANDARD = "Apache Iceberg catalog + Arrow as the in-memory interchange"
print("\nStandardise:", STANDARD)

Step-by-step trace.

Team	Dominant axis	Assigned format	Moves?
BI / analytics	full scan	parquet	no (already Parquet)
Warehouse / Hive	transactional	orc	no (already ORC)
ML / RAG	random access + vectors	lance	yes (from Parquet + vector DB)
Fraud features	random access	lance	yes (from Parquet)

The counter-proposal is: keep BI on Parquet and the warehouse on ORC (no migration), move the two random-access teams (ML/RAG and fraud features) to Lance, and standardise the catalog (Iceberg) and the interchange (Arrow) — not the file format — across all four. The benchmark to defend it writes each team's representative sample to all three formats and measures the axis that team stresses (scan throughput for BI, random-access latency for ML), proving the assignment on real data. The migration cost is two full rewrite passes (stream Arrow → Lance + build indexes) gated on row-count and functional-query validation.

Output:

Decision	Value
BI / analytics	Parquet (stays)
Warehouse / Hive	ORC (stays)
ML / RAG	Lance (migrate; delete external vector DB)
Fraud features	Lance (migrate; random-access sampling)
Standardised layer	Iceberg catalog + Arrow interchange
Migration cost	2 rewrite passes, validated on samples first

Why this works — concept by concept:

• Workload-keyed assignment — each format serves the read pattern it was designed for, so no team pays the worst-fit tax a single-format mandate would impose.
• Benchmark-defended — the assignment isn't asserted, it's measured on each team's real sample across the axis they stress, which is what turns "it depends" into a defensible policy.
• Standardise the catalog, not the file — Iceberg + Arrow give the platform uniformity (one governance plane, one interchange) without forcing a lossy single file format, which is the actual thing leadership wants when they ask for "one format."
• Bounded migration — only the two random-access teams move, each via a validated streaming rewrite, so the cost is scoped and reversible.
• Cost — two rewrite passes (O(rows) each, one-time) plus the ongoing operation of three readers, in exchange for every team running on its optimal substrate and one deleted external system (the vector DB). Compared with a single-format mandate that permanently taxes the worst-fit workload, the three-format policy is O(1) per query on each team's dominant axis.

Optimization
Topic — optimization
Optimization problems on format and layout trade-offs

Practice →

ETL
Topic — etl
ETL problems on format migration and rewrites

Practice →

---

Cheat sheet — columnar format recipes

• Which format when. Parquet is the 2026 default for analytics and lakehouse tables (widest engine support; Delta/Iceberg/Hudi write it underneath). ORC when you're Hive-native and need ACID transactional tables or the compression edge on wide low-cardinality fact tables. Lance when the workload is ML/AI — random access sampling, reproducible dataset versions, or vector search over embeddings. When the platform is mixed, standardise the catalog (Iceberg/Unity) and interchange (Arrow), not the file format.
• The four axes. Read pattern (scan vs point/random access), ecosystem gravity (how many engines read it), evolution/versioning (schema + dataset versions), random access (cheap take by row id or not). Every orc vs parquet — and every Lance — decision falls out of which axis dominates the workload. Say the axis before the format.
• ORC anatomy + tuning. File → stripes (~64–256 MB) → row-group index (10,000-row stride) → columns with RLE/dictionary under ZLIB/ZSTD/Snappy. Tuning: sort by the dominant range-predicate column (tight min/max), add bloom_filter_columns for high-cardinality equality, big stripes for scan-heavy Hive fact tables, ZSTD codec for ratio + decode speed.
• ORC Hive ACID. Transactional ORC tables store base + delta + delete_delta directories; readers merge base + deltas minus delete_deltas by (writeId, bucketId, rowId); ALTER TABLE ... COMPACT 'major' folds deltas back into a fresh base. Read amplification grows with un-compacted deltas — schedule compaction aggressively on write-heavy transactional tables.
• Parquet anatomy + tuning. File → row groups (~128 MB) → column chunks → pages (~1 MB) → footer (min/max/null_count + optional page index + bloom). Tuning: sort by predicate column, right-size row groups, write_page_index=True for page-level pruning, write_bloom_filter=[cols] for equality, ZSTD + dictionary. write_statistics=True is mandatory — without footer stats there is no predicate pushdown.
• Parquet Dremel nesting. Nested/repeated fields (list<struct>) are shredded into flat leaf columns plus repetition-level (list boundaries) and definition-level (null depth) streams. Projection reaches into nesting — read one leaf of a deep struct (columns=["items.list.element.sku"]) without materialising the rest. This is Parquet's structural edge for JSON-like data.
• Lance anatomy. Dataset directory → fragments (append/delete unit) → manifest-per-version (_versions/) → secondary indexes (_indices/). Rows are addressable by id, which is what makes take(row_ids) O(rows requested) instead of O(row group). Design center is random access, not full-scan analytics.
• Lance random access. lance.dataset(path).take([ids], columns=[...]) fetches scattered rows directly — the operation Parquet/ORC do poorly (they decode whole row groups/stripes). Reach for Lance the moment ML mini-batch sampling, shuffling, or vector re-ranking dominates the read pattern.
• Lance versioning. Every write_dataset(mode="append"|"overwrite") bumps the version; unchanged data files are shared (zero-copy). lance.dataset(path, version=N) time-travels to version N — reproducible training inputs and cheap rollback with no external table format. Vacuum old versions when you no longer need to time-travel.
• Lance vector search. create_index("embedding", index_type="IVF_PQ", num_partitions, num_sub_vectors) builds an ANN index inside the dataset; to_table(nearest={"column","q","k"}) runs top-k similarity; add filter="..." (accelerated by a scalar btree/bitmap index) for filtered ANN. Embeddings + index + data are one versioned artifact — no separate vector DB.
• Predicate pushdown checklist (ORC + Parquet). (1) Sort by the dominant range-predicate column so min/max ranges are disjoint. (2) Enable page index (Parquet) — ORC has the 10,000-row stride natively. (3) Add bloom filters for high-cardinality equality columns. (4) Project only the columns you need. (5) Right-size the row group / stripe to the read pattern. Unsorted data with no bloom filters prunes almost nothing.
• Decision matrix (memorise). Analytics/BI/lakehouse → Parquet. Hive-legacy/transactional → ORC. ML/embeddings/RAG/random access → Lance. Mixed platform → Iceberg/Unity catalog + Arrow, file format per domain. One default (Parquet), three overrides (Lance, ORC, table-format-for-mixed). Name the override that fires.
• Migration cost. ORC → Parquet (Hive → lakehouse): full rewrite + repartition/sort + catalog re-register, budget engineer-weeks at scale. Parquet → Lance (feature/embedding store): stream Arrow batches → write_dataset → build IVF_PQ + scalar indexes, gated on row-count + functional-query validation. Never migrate a petabyte without proving the axis win on a sample partition first.

Frequently asked questions

Is ORC or Parquet better?

Neither is universally better — orc vs parquet is a read-pattern and ecosystem decision, not a leaderboard. Apache ORC wins when you are Hive-native and need ACID transactional tables (base/delta/compaction) or the compression edge on wide, low-cardinality warehouse fact tables, where ORC frequently lands 10–20% smaller than comparably-configured Parquet. Apache Parquet wins on ecosystem gravity — Spark, Arrow, DuckDB, Polars, pandas, Snowflake/BigQuery external tables, and the three lakehouse table formats (Delta, Iceberg, Hudi) all read and write it as a first-class citizen, so picking Parquet means every engine can read your data tomorrow. Both are columnar, both do three-level predicate pushdown (row-group/stripe → page/row-group-index → bloom filter), and both compress well with ZSTD. For a greenfield lakehouse, default to Parquet; for a Hive metastore with transactional corrections, ORC is native.

When should I use Lance instead of Parquet?

Use the lance format when your read pattern is dominated by random access rather than full-column scans — ML mini-batch sampling, dataset shuffling, and vector re-ranking all fetch scattered rows, and Lance's take(row_ids) is O(rows requested) while Parquet must decode whole row groups to extract the same scattered rows. Lance also wins when you need reproducible dataset versioning (each write is a new manifest, so a model can time-travel to its exact training rows via lance.dataset(path, version=N)) or native vector search (IVF_PQ / HNSW ANN indexes built inside the dataset, no external vector database). Keep Parquet for pure BI/analytics scan workloads on a mature lakehouse — Lance's newer ecosystem is a real cost there, and the random-access advantage is irrelevant to a GROUP BY scan. The clean rule: scans and ecosystem breadth → Parquet; random access, versioning, and vectors → Lance.

What is predicate pushdown and which format does it best?

Predicate pushdown is the optimization where the reader evaluates a query's filter against per-chunk statistics before decoding column data, skipping any row group / stripe / page whose min/max proves it cannot match. All three formats do it, and they are close: ORC pioneered it with three levels (file, stripe, and the 10,000-row row-group index) plus optional bloom filters; Parquet matches it with row-group statistics, the optional page index (per-page min/max), and bloom filters; Lance adds scalar btree/bitmap indexes for filtered lookups. The format matters far less than the data layout — pushdown only prunes when min/max ranges are tight, which means you must sort by the dominant range-predicate column and add bloom filters for high-cardinality equality. Unsorted data with overlapping ranges prunes almost nothing regardless of format, so the biggest pushdown lever is sort order, not format choice.

Does Parquet support vector search?

No — Parquet (and ORC) can store an embedding column as a list<float> type, but neither can index it for approximate-nearest-neighbor search, so vector similarity over Parquet requires a separate vector database (or a brute-force scan). This is the sharpest capability line between the classic columnar formats and the modern vector search format: Lance builds IVF_PQ or HNSW ANN indexes directly inside the dataset, keeps them versioned alongside the embeddings, and supports filtered vector search (nearest={...} combined with a scalar filter=) in one call. If embeddings and similarity search are core to your workload — RAG retrieval, recommendation re-ranking, semantic search — Lance eliminates the separate vector store; if you only need to store embeddings for offline batch processing without ANN queries, Parquet is fine and keeps the broader ecosystem.

Is ORC dead in 2026?

No, but it is a specialist rather than a default. Apache ORC remains dominant wherever Hive is dominant — mature on-prem Hadoop clusters, Hive ACID transactional tables (its base/delta/compaction implementation predates Delta Lake and Iceberg), and shops where ORC's compression edge on wide low-cardinality fact tables is a real storage-cost line item at petabyte scale. What changed is that greenfield lakehouses now default to Parquet because the lakehouse table formats (Delta, Iceberg, Hudi) standardised on Parquet underneath and the Arrow ecosystem made Parquet the interchange format of the Python data stack. So ORC is not dead — it is the correct answer for Hive-native ACID and compression-critical warehouse workloads, and senior interviewers still probe it because knowing why you'd pick ORC (Hive ACID + compression) proves you understand all four format-selection axes rather than defaulting by habit.

How do I migrate from Parquet to Lance without downtime?

Migrate by streaming, indexing, validating, and cutting over — never a big-bang load. First, read the Parquet dataset as Arrow record batches (pyarrow.dataset(...).scanner(batch_size=...).to_reader()) and feed the reader straight into lance.write_dataset(reader, dst, mode="overwrite"), so a store larger than RAM migrates without OOM. Second, build the indexes Lance adds over Parquet — create_index("embedding", index_type="IVF_PQ", ...) for ANN and create_scalar_index(col, index_type="BTREE") for filtered search. Third, gate the cutover on validation: assert src.count_rows() == dst.count_rows() to catch a truncated copy, and run a sample nearest-neighbor query to confirm the index is queryable. Fourth, keep the Parquet dataset live and dual-read (or read Parquet as the fallback) until the Lance path is proven, then repoint readers and retire the external vector database the migration makes redundant. Validate on a single partition before committing the full dataset — a format migration you can't roll back is one you shouldn't start.

Practice on PipeCode

• Drill the optimization practice library → for the predicate-pushdown, sort-order, bloom-filter, and scan-pruning problems that make ORC and Parquet layouts fast.
• Rehearse on the ETL practice library → for the compaction, rewrite, and format-migration pipelines that move data between ORC, Parquet, and Lance without downtime.
• Sharpen the storage-design axis with the design practice library → for the format-selection, feature-store, and embedding-store topology questions senior interviewers open with.
• Layer in the data-processing practice library → for the columnar, nested-data, and random-access problems that turn the four-axis decision matrix into muscle memory.
• Stack the prerequisites against PipeCode's broader 450+ data-engineering catalogue to anchor the orc vs parquet vs Lance decision matrix against real graded inputs.

Lock in columnar-format decision muscle memory

Docs describe formats. PipeCode drills explain the decision — when ORC's Hive ACID earns its place, when Parquet's ecosystem gravity makes it the safe default, when Lance's random access and vector indexes beat a Parquet-plus-vector-DB stack, and when to standardise the catalog instead of the file format. Pipecode.ai is Leetcode for Data Engineering — pattern-first practice tuned for the production trade-offs senior data engineers actually face.

Practice optimization problems →
Practice design problems →