DEV Community

Cover image for Delta-rs & PyIceberg: Rust-Backed Table Format Libraries That Skip the JVM
Gowtham Potureddi
Gowtham Potureddi

Posted on

Delta-rs & PyIceberg: Rust-Backed Table Format Libraries That Skip the JVM

#python #sql #interview #dataengineering

delta-rs is the Rust-native crate (with Python bindings shipped as the deltalake package) that finally lets Python-first data teams read, write, MERGE, OPTIMIZE and VACUUM Delta Lake tables without ever launching a JVM — and alongside it PyIceberg has grown from a curious side project into the canonical way to touch Apache Iceberg tables from Python, notebooks, Lambda functions, and small services where a 5-second Spark cold-start was killing throughput and pushing every "just read one table" job onto a cluster that was never meant to exist. Every table your organisation writes into an open lakehouse — a Delta table produced by a Databricks job, an Iceberg table maintained by a Flink stream, a partitioned fact table shared across three engines — has to be reachable from a Python analyst's notebook, from a Lambda that runs once per S3 event, from a serverless daily ETL, and increasingly from a Rust or Go service, all without forcing every reader through a JVM boot the size of a small game. The engineering trade-off does not live in "should we use table formats" — every serious 2026 lakehouse does — but in which library each workload uses and whether it needs Spark at all.

This guide is the senior-DE walkthrough you wished existed the first time an interviewer asked "how would you read a Delta table without Spark?", or "when is pyiceberg the right choice over a Trino cluster?", or "walk me through the JVM-versus-Rust trade-off for a Lambda that has to append one file per S3 event." It walks through the deltalake python package end-to-end — DeltaTable read + write, partitioned writes, MERGE / UPSERT, OPTIMIZE + VACUUM, schema evolution and time-travel — the PyIceberg client and its catalog integrations (Glue, Hive, REST, Nessie, Polaris, SQL), the Python-stack interop story via polars.scan_delta, scan_iceberg, DuckDB's iceberg_scan, Ray Data, Dask and Modin, the serverless story that finally kills the JVM cold-start tax, and the decision framework that tells you when iceberg without spark is fine and when a full Spark cluster is still the right answer. Each section pairs a teaching block with a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

PipeCode blog header for delta-rs & PyIceberg — bold white headline 'Delta-rs & PyIceberg' over a hero composition of two glyph medallions (a Rust gear + a Python snake ribbon) framing a central purple 'no JVM 2026' seal, on a dark gradient.

When you want hands-on reps immediately after reading, drill the SQL practice library →, rehearse aggregation patterns with the aggregation practice library →, and stretch the design axis with the design practice library →.

On this page

1. Why Rust-native table format libraries matter in 2026

The Delta / Iceberg reference implementations were Spark-first — Python paid the JVM tax on every read

The one-sentence invariant: the Delta Lake and Apache Iceberg reference implementations were both built on the JVM (Spark-native for Delta, Java + Spark for Iceberg), which meant every Python team that wanted to touch a lakehouse table paid a 2-8 second JVM cold-start, hundreds of megabytes of resident memory, and a giant fat-jar deployment on every invocation — Rust-native libraries (delta-rs for Delta, PyIceberg with Rust hot paths for Iceberg) collapse that tax to tens of milliseconds and let notebooks, Lambda functions, and small services read and write open-lakehouse tables at cost profiles the JVM stack cannot match. The pattern you pick in month one becomes the tax you fight to migrate away from in year two, because every downstream consumer that assumes "the reader is Spark" hard-codes latencies, memory footprints, and packaging conventions that the Rust-native path renders obsolete.

The four axes interviewers actually probe.

  • Cold-start cost. Spark's JVM boots in 2-8 seconds before it opens a single Parquet file. delta-rs and PyIceberg boot in 30-200 milliseconds. On a Lambda invoked once per S3 event, this is the difference between a hot function and a timeout. Interviewers open with this because it separates people who've deployed serverless data pipelines from those who've only run Spark clusters.
  • Memory footprint. A minimal PySpark process resident-sets at ~700 MB before it does anything useful. delta-rs in a Python process runs comfortably in a 256 MB Lambda. Every megabyte matters when your per-invocation cost model is measured in GB-seconds.
  • Read/write parity. Both libraries have reached the point where "can you read Delta / Iceberg from Python?" is unambiguously yes. Write parity is more nuanced — deltalake's MERGE has matured through 2025-2026 to cover the common upsert cases; PyIceberg's positional-delete story is still landing. Interviewers probe which operations you've actually shipped.
  • Catalog and ecosystem. Iceberg forces a catalog choice (Glue, Hive, REST, Nessie, Polaris, SQL); Delta historically used the transaction log alone but Unity Catalog and the Delta Sharing protocol have added federation layers. Naming the catalogs correctly and picking one per workload is a senior signal that separates "I read the docs" from "I've shipped this."

The 2026 reality — Rust-native has become the default for < 100 GB Python-first work.

  • delta-rs (the deltalake Python package). Version 0.15+ ships production-ready Delta reads and writes, MERGE / UPSERT, OPTIMIZE (bin-packing compaction), VACUUM (soft-delete of stale files), Z-ORDER hints, schema evolution, and time-travel. Interop is via Arrow — you hand deltalake an Arrow table or receive one back, and the rest of the Python stack (Pandas, Polars, DuckDB, Ray) chains from there. Reference maintainers: the Delta Lake project (Linux Foundation).
  • PyIceberg. Pure Python client with Rust in the hot paths (Arrow decoding, manifest parsing). Supports read, append, and overwrite writes; catalog federation across Glue, Hive Metastore, Iceberg REST, Nessie, Snowflake Polaris, and a SQL-backed catalog for local dev; schema evolution and partition evolution; time-travel via snapshot_id or as_of_timestamp. Reference maintainers: the Apache Iceberg project.
  • Polars, DuckDB, Ray Data — the consumers. Polars ships scan_delta and scan_iceberg for lazy frames; DuckDB's Iceberg extension exposes iceberg_scan; Ray Data reads Arrow-shaped output; all three go through delta-rs / PyIceberg under the hood. The Rust-native path has become the shared implementation the whole Python analytics ecosystem builds on.
  • Spark. Still the correct answer for multi-TB batch, complex joins across dozens of tables, and Databricks-native pipelines. But not the default answer for a Lambda that appends one file to a Delta table — that's deltalake.write_deltalake(...).

What interviewers listen for.

  • Do you name the specific packages — deltalake for delta-rs, pyiceberg for Iceberg — rather than saying "the Python Delta library"? — senior signal.
  • Do you cite the JVM cold-start (2-8 s) versus the Rust cold-start (30-200 ms) as the quantified trade-off? — senior signal.
  • Do you push back on "just use Spark" with the "what's the workload size?" question? — senior signal.
  • Do you know that PyIceberg is Python-with-Rust-hot-paths, not pure Rust like delta-rs, and can name the Arrow decode + manifest parse as the Rust-accelerated hot loops? — senior signal.
  • Do you describe the ecosystem as "table-format libraries + Arrow-native consumers" rather than "Spark alternatives"? — required answer.

Worked example — the JVM cold-start tax versus the Rust cold-start

Detailed explanation. The single most useful artifact for a Rust-native lakehouse interview is a memorised cold-start comparison table. Every senior discussion converges on this within the first ten minutes; having it in your head is what separates a fluent answer from a stumbling one. Walk through building the table for a hypothetical Lambda that reads a Delta table on each S3 event.

  • Workload. An AWS Lambda triggered by S3 PutObject events. On each invocation the function must read the latest orders Delta table, find rows for the newly-uploaded file's customer, and write a small enrichment record back.
  • Options. (a) PySpark + Delta Lake (JVM), (b) deltalake Python package (Rust), (c) Container Lambda with Spark preloaded, (d) provisioned-concurrency Lambda with PySpark.
  • Target. p99 invocation cost < 500 ms and < 512 MB memory.

Question. Build the cold-start / memory / packaging comparison and pick the option that fits the Lambda's constraints.

Input.

Option Cold-start Warm memory Package size Notes
PySpark + Delta 5-8 s 700-900 MB ~400 MB JVM boot dominates
deltalake (Rust) 50-200 ms 80-150 MB ~30 MB pip install; no JVM
Container Lambda + Spark 2-4 s 900 MB ~1.5 GB image ECR pull + warmup
Provisioned-concurrency PySpark 100 ms after warm 900 MB ~400 MB pay for warm concurrency 24/7

Code.

# Option B — deltalake (the winning fit for this Lambda)
from deltalake import DeltaTable
import pyarrow.compute as pc

def handler(event, context):
    # Cold-start budget: ~50-200 ms including import
    dt = DeltaTable("s3://acme-lake/orders")

    # Push-down the customer filter before materialisation
    key       = event["Records"][0]["s3"]["object"]["key"]
    customer  = key.split("/")[1]
    orders    = (
        dt.to_pyarrow_dataset()
          .filter(pc.field("customer_id") == customer)
          .to_table()
    )

    return {"rows": orders.num_rows, "cold_start_ms": context.get_remaining_time_in_millis()}
Enter fullscreen mode Exit fullscreen mode

Step-by-step explanation.

  1. Option A (PySpark + Delta) fails the p99 budget on the cold path. The 5-8 s JVM boot alone exceeds the 500 ms budget by an order of magnitude. Even if you accept an 8 s p99, the 700-900 MB memory footprint pushes you into the 1 GB Lambda tier at ~4x the per-ms cost.
  2. Option C (container Lambda with Spark preloaded) trims the boot to 2-4 s by baking the JVM into the image, but you still pay for a 1.5 GB image pull on cold start and the same 900 MB steady-state memory. The image size also inflates deployment time — every code change ships another 1.5 GB.
  3. Option D (provisioned concurrency for PySpark) makes the cold path fast (100 ms) at the cost of paying for warm concurrency 24/7. On a Lambda that runs a few thousand times per day, this is 10-50x the cost of on-demand deltalake.
  4. Option B (deltalake Rust package) is the fit. pip install deltalake produces a ~30 MB dependency, the process boots in 50-200 ms including the Delta table log parse, and the warm-state memory is 80-150 MB — comfortably under the 256 MB Lambda tier. Reading a single-partition slice of a moderately-sized Delta table completes in 100-300 ms.
  5. The choice is not "Spark or nothing" — it is "which library matches the invocation model." Long-running batch that reads a TB stays on Spark; a Lambda that reads a slice on every S3 event runs on deltalake. Both engines can read the same Delta table because the table format is the contract; the reader is just an implementation detail.

Output.

Constraint Winner Reasoning
p99 < 500 ms cold deltalake JVM boot alone breaks the budget for PySpark
Memory < 512 MB deltalake Rust process fits in 128-256 MB Lambda tier
Package size < 100 MB deltalake 30 MB pip wheel vs 400 MB PySpark deps
Cost per invocation deltalake on-demand, no provisioned concurrency needed
Multi-TB batch job PySpark different workload; different tool

Rule of thumb. Never pick a Delta or Iceberg reader based on "which one is trendy." Pick it based on (cold-start × memory × package size × workload cardinality) — the four axes. Write the table on a whiteboard first; the reader falls out of the constraints.

Worked example — what interviewers actually probe

Detailed explanation. The senior data-engineering Rust-native lakehouse interview has a predictable structure: the interviewer opens with an ambiguous question ("how would you read a Delta table from a Python service?"), then progressively narrows to test whether you know the trade-offs. Candidates who name the specific package in sentence one score highest; candidates who describe "we'd use PySpark" score lowest. Walk through the grading rubric.

  • Ambiguous opener. "How would you read Delta / Iceberg from a Python-first stack?" — invites you to name deltalake or pyiceberg specifically.
  • Follow-up 1. "What's the cold-start cost?" — probes the JVM-vs-Rust axis.
  • Follow-up 2. "What if we need MERGE / UPSERT?" — probes write-parity awareness.
  • Follow-up 3. "How does the catalog work?" — probes Iceberg catalog choice + Delta transaction-log semantics.
  • Follow-up 4. "When would you still reach for Spark?" — probes the escalation trigger.

Question. Draft a 5-minute senior answer that covers all four axes without waiting to be asked.

Input.

Interview signal Weak answer Senior answer
Package named "we'd use PySpark" "deltalake for Delta, pyiceberg for Iceberg — both pip-installable"
Cold-start "startup is fast" "50-200 ms Rust vs 5-8 s JVM — measurable in Lambda p99"
MERGE story "we'd use SQL" "delta-rs .merge(...).when_matched_update() is production-ready in 0.15+"
Catalog "the metastore" "Glue / REST / Hive / Nessie / Polaris / SQL — pick one per environment"
Escalation "always Spark" "reach for Spark when data > ~100 GB per job or multi-cluster concurrency required"

Code.

Senior Rust-native lakehouse answer template (5 minutes)
=========================================================

Minute 1 — name the packages up front
  "For any Python-first workload under ~100 GB, I default to
   `deltalake` for Delta tables and `pyiceberg` for Iceberg
   tables. Both are Rust-backed under the hood — delta-rs is a
   Rust crate with a Python wrapper; PyIceberg is Python with
   Rust hot loops for Arrow decode and manifest parsing."

Minute 2 — cold-start and memory
  "The point of Rust-native readers is the cost profile: 50-200 ms
   cold-start vs 5-8 s for a JVM boot, and 80-150 MB memory vs
   700-900 MB for a minimal PySpark process. That's the difference
   between a Lambda that runs in the 128 MB tier and one that has
   to sit in the 1 GB tier just to import Spark."

Minute 3 — read / write / MERGE parity
  "Reads are unambiguously ready — DeltaTable.to_pyarrow_dataset()
   and pyiceberg.table.scan() both push down predicates and columns.
   Writes: deltalake supports append, overwrite, partitioned writes,
   and MERGE (production-ready in 0.15+). PyIceberg supports append
   and overwrite; positional deletes are maturing."

Minute 4 — catalogs and interop
  "Iceberg needs a catalog — PyIceberg speaks Glue, Hive Metastore,
   Iceberg REST (Polaris, Tabular, self-hosted), Nessie, and a SQL
   catalog for local dev. Delta uses the transaction log; catalog
   layers like Unity Catalog federate on top. Downstream: Polars
   `scan_delta` / `scan_iceberg`, DuckDB `iceberg_scan`, Ray Data —
   all consume via Arrow."

Minute 5 — when to still reach for Spark
  "Reach for Spark when the job is multi-TB batch, needs the
   Catalyst optimiser on complex joins, or lives inside Databricks.
   Everything else — Lambda, notebooks, small services, streaming
   consumers below ~1000 events/sec — goes Rust-native. The two
   stacks read the same tables; that's the whole point of an open
   table format."
Enter fullscreen mode Exit fullscreen mode

Step-by-step explanation.

  1. Minute 1 is the crucial framing. Naming the specific package — deltalake or pyiceberg — signals you have hands-on experience, not just docs-reader knowledge. Weak candidates default to "we'd use Spark" because Spark is what they were taught.
  2. Minute 2 addresses the quantified trade-off. Saying "the JVM is slow" is a weak answer; saying "5-8 s cold-start" is a senior answer because it maps to a concrete cost model (Lambda p99, container-image pulls, provisioned concurrency).
  3. Minute 3 handles the read/write asymmetry. Reads are trivially at parity; writes are the axis where Rust-native still has gaps. Naming MERGE specifically for delta-rs and positional deletes for PyIceberg is the senior tell.
  4. Minute 4 is the catalog probe. PyIceberg without a catalog is not usable; naming the catalogs and picking one per environment shows you've deployed this, not just imported it. Delta's transaction-log-only story vs Iceberg's catalog-required story is the architectural fork.
  5. Minute 5 is the escalation trigger. Senior candidates never say "always Spark" or "never Spark" — they name the workload thresholds. ~100 GB per job, multi-TB batch, Catalyst on complex joins, Databricks lock-in — those are the escalation triggers.

Output.

Grading criterion Weak score Senior score
Names packages in minute 1 rare mandatory
Quantifies cold-start rare required
Names MERGE / positional deletes rare mandatory
Names 3+ catalogs rare senior signal
Names Spark escalation triggers rare senior signal

Rule of thumb. The senior Rust-native lakehouse answer is a 5-minute monologue that covers cold-start, memory, read/write parity, catalog choice, and escalation triggers without waiting for the follow-ups. Rehearse it once; deploy it every time.

Worked example — the "pick the reader" decision tree

Detailed explanation. Given a new lakehouse workload, the senior architect runs a 4-question decision tree in their head. Codifying the tree makes the interview answer reproducible: any interviewer can hand you a scenario and you can walk it out loud. Walk through the tree with three canonical scenarios: a Lambda-based enrichment, a nightly warehouse rollup, and a Databricks-native ETL.

  • Q1. Is the invocation model serverless / notebook / small service? → yes = Rust-native (delta-rs or pyiceberg); no = go to Q2.
  • Q2. Is the data volume per job > ~100 GB? → yes = Spark; no = go to Q3.
  • Q3. Do you need multi-cluster concurrent writes with heavy MERGE fan-out? → yes = Spark (Delta with UC) or Trino (Iceberg); no = go to Q4.
  • Q4. Is the codebase already Databricks-native? → yes = Spark (ecosystem lock-in wins); no = default to Rust-native.

Question. Walk the decision tree for the three scenarios and record the reader each ends up with.

Input.

Scenario Q1 (serverless?) Q2 (> 100 GB?) Q3 (multi-cluster MERGE?) Q4 (Databricks-native?)
Lambda enrichment yes — — —
Nightly warehouse rollup no yes no no
Databricks-native ETL no no no yes

Code.

# Decision-tree helper (illustrative)
def pick_reader(is_serverless: bool,
                 data_gb: float,
                 multi_cluster_merge: bool,
                 databricks_native: bool,
                 table_format: str) -> str:
    """Return the reader for a lakehouse workload."""
    if is_serverless:
        return "deltalake" if table_format == "delta" else "pyiceberg"

    if data_gb > 100:
        return "spark"

    if multi_cluster_merge:
        return "spark (delta) or trino (iceberg)"

    if databricks_native:
        return "spark"

    return "deltalake" if table_format == "delta" else "pyiceberg"


# Walk the three scenarios
print(pick_reader(True,  5.0,  False, False, "delta"))
# → 'deltalake'

print(pick_reader(False, 500.0, False, False, "iceberg"))
# → 'spark'

print(pick_reader(False, 40.0, False, True,  "delta"))
# → 'spark'
Enter fullscreen mode Exit fullscreen mode

Step-by-step explanation.

  1. Scenario 1 — Lambda triggered by S3 PutObject reads a Delta table slice on each event. Q1 short-circuits to deltalake. This is the canonical Rust-native win: the JVM cold-start would break Lambda's cost model, and the data-per-invocation is tiny.
  2. Scenario 2 — a nightly warehouse rollup that reads ~500 GB of Iceberg fact tables and joins to five dimension tables. Q1 = no, Q2 = yes → Spark. The Catalyst optimiser and multi-node execution earn their keep at this scale, and PyIceberg would leave you doing manual partition pushdown and predicate simplification.
  3. Scenario 3 — a Databricks-native pipeline that ships as a job cluster in a Databricks workspace. Q4 = yes → Spark. This is the "ecosystem wins" rule: even for a 40 GB job that deltalake could handle, the operational cost of running a non-Spark reader inside Databricks is higher than the JVM tax it would save.
  4. The three scenarios collectively cover the decision space: Rust-native for serverless, Spark for large batch, Spark for ecosystem lock-in. Everything else — most notebooks, most small services, most < 100 GB batch — defaults to Rust-native.
  5. The table_format argument in the helper is a reminder that both readers exist per format: deltalake for Delta, pyiceberg for Iceberg. You don't pick "delta-rs or pyiceberg" — you pick one per table format based on what the table is.

Output.

Scenario Reader Primary reason
Lambda enrichment deltalake serverless invocation model
Nightly warehouse rollup Spark > 100 GB batch
Databricks-native ETL Spark ecosystem lock-in

Rule of thumb. The four-question decision tree is a whiteboard-friendly answer. Practice walking it end-to-end so an interviewer can hand you any scenario and get a reader name in under 60 seconds.

Senior interview question on Rust-native table format libraries

A senior interviewer often opens with: "You inherit a Python data platform that runs 300 daily Lambda-based ETL jobs, most of which currently spin up a small EMR cluster just to read a Delta table, do a small transform, and write back. Leadership wants to cut compute cost and cold-start latency. Walk me through the migration to deltalake for the < 100 GB jobs, the escalation criteria that keep the rest on Spark, and the operational monitoring you'd add to detect any silent write-behavior regression."

Solution Using delta-rs for the serverless tier + Spark for the batch tier + Parquet-checksum drift monitoring 

# Step 1 — Lambda-side reader / writer using deltalake
from deltalake import DeltaTable, write_deltalake
import pyarrow as pa
import pyarrow.compute as pc

DELTA_PATH = "s3://acme-lake/orders"

def handler(event, context):
    # Cold path: 60-180 ms including DeltaTable log parse
    dt = DeltaTable(DELTA_PATH)

    # Filter push-down happens in the Rust scanner
    day = event["day"]
    orders = (
        dt.to_pyarrow_dataset()
          .filter((pc.field("order_date") == day) & (pc.field("status") == "pending"))
          .to_table()
    )

    # Small transform: enrich with computed column
    enriched = orders.append_column(
        "shipping_cost",
        pc.multiply(orders["total_cents"], pa.scalar(0.05))
    )

    # Write back as a new Delta version (append; partition by order_date)
    write_deltalake(
        DELTA_PATH,
        enriched,
        mode="append",
        partition_by=["order_date"],
    )

    return {"rows_written": enriched.num_rows}
Enter fullscreen mode Exit fullscreen mode 
# Step 2 — Spark stays for the > 100 GB jobs (batch rollup)
from pyspark.sql import SparkSession

spark = (SparkSession.builder
         .appName("nightly-rollup")
         .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension")
         .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog")
         .getOrCreate())

spark.sql("""
    MERGE INTO analytics.orders_daily t
    USING (
      SELECT order_date, customer_id, SUM(total_cents) AS revenue_cents
      FROM   raw.orders
      WHERE  order_date >= current_date - INTERVAL 7 DAYS
      GROUP  BY order_date, customer_id
    ) s
    ON  t.order_date = s.order_date AND t.customer_id = s.customer_id
    WHEN MATCHED     THEN UPDATE SET revenue_cents = s.revenue_cents
    WHEN NOT MATCHED THEN INSERT *
""")
Enter fullscreen mode Exit fullscreen mode 
# Step 3 — Parquet-checksum drift monitor (catches silent write regressions)
import boto3, hashlib, json

def compare_writers(sample_path_a: str, sample_path_b: str) -> dict:
    s3 = boto3.client("s3")
    def digest(path):
        buf = s3.get_object(Bucket=path.split("/")[2],
                            Key="/".join(path.split("/")[3:]))["Body"].read()
        return hashlib.sha256(buf).hexdigest()
    return {"a": digest(sample_path_a),
            "b": digest(sample_path_b),
            "match": digest(sample_path_a) == digest(sample_path_b)}
Enter fullscreen mode Exit fullscreen mode

Step-by-step trace.

Step Before (all-Spark on EMR) After (delta-rs + Spark)
Lambda cold-start 8-12 s (EMR bootstrap) 60-180 ms (delta-rs)
Lambda memory 900 MB - 1.5 GB 128-256 MB
Per-invocation cost $0.02-0.05 $0.0005-0.002
Batch job (> 100 GB) Spark on EMR Spark on EMR (unchanged)
Write engine parity check none Parquet-checksum sampler
Table format contract Delta 3.x Delta 3.x (unchanged)
Reader migration risk — monitored via sampler

After the migration, ~250 of the 300 daily jobs run on deltalake in the 256 MB Lambda tier for 60-180 ms per invocation; the remaining ~50 large batch jobs stay on Spark; the sampler compares Parquet checksums produced by delta-rs vs Spark on a nightly canary and alerts on drift.

Output:

Metric Before After
Lambda p99 cold-start 12 s 200 ms
Lambda memory tier 1.5 GB 256 MB
Monthly compute spend $18k $4k
Nightly batch jobs 300 50 (rest on Lambda)
Silent write regression risk none monitored checksum sampler daily

Why this works — concept by concept:

  • delta-rs (Rust) as the serverless reader — the Rust crate reads the Delta transaction log, plans a scan with predicate + column push-down, and returns Arrow tables. The Python wrapper (deltalake pip package) is a thin binding; no JVM enters the process.
  • Spark retained for > 100 GB batch — Rust-native readers are not universally superior. Spark's Catalyst optimiser and multi-node execution earn their keep on large joins and multi-TB scans. The right architecture is both, not one or the other.
  • Parquet-checksum sampler — a sanity monitor that compares files produced by delta-rs and Spark on a nightly canary. If either engine ever writes a file the other cannot read, the sampler catches it before downstream consumers do. The check is cheap (SHA-256 of a small sample) and shipped from day one.
  • Arrow as the interchange contract — both engines produce Arrow; downstream Polars / DuckDB / Ray all consume Arrow. The reader choice is invisible to consumers because Arrow is the contract.
  • Cost — 4x total compute cost reduction ($18k → $4k monthly); 100x cold-start latency reduction; 6x memory-tier reduction. The operational cost is the sampler (one Lambda / day) plus the mental model shift from "everything on Spark" to "reader per workload." Net O(1) per Lambda invocation vs O(cluster-bootstrap) for the old model.

SQL
Topic — sql
SQL practice on lakehouse and table-format problems

Practice →

Design Topic — design Design problems on Rust-native lakehouse readers

Practice →

2. delta-rs deep dive

delta-rs is a Rust crate with a Python wrapper — read, write, MERGE, OPTIMIZE, VACUUM Delta tables without a single line of Spark

The mental model in one line: delta-rs is a Rust implementation of the Delta Lake protocol (transaction log parsing, Parquet reading and writing, MERGE / UPSERT planning, OPTIMIZE / VACUUM maintenance, schema evolution, time-travel) shipped as the deltalake Python package (and as a bare Rust crate for Rust services), which lets Python code read and write Delta tables through Arrow with no JVM in the process and cold-start latencies measured in tens of milliseconds — every Delta operation the Databricks Spark connector supports has landed in deltalake through 2024-2026 with MERGE reaching production maturity in the 0.15+ line. Every senior Python-first data engineer has one job that would previously have needed Spark and now runs on deltalake in a Lambda.

Iconographic delta-rs diagram — a Rust crate card feeding a Python deltalake package, a DeltaTable card with version-history ribbons, a MERGE builder card and an OPTIMIZE+VACUUM utility strip.

The four operational axes for delta rust.

  • Reading. DeltaTable(path) parses the transaction log (JSON + checkpoint Parquet) and hands back a Rust-native scanner. to_pyarrow_dataset() returns a PyArrow dataset that supports predicate + column push-down; to_pandas() and to_pyarrow_table() materialise. Time-travel via load_as_version(n) or load_as_timestamp(ts) — the Delta log is fully replayable.
  • Writing. write_deltalake(path, arrow_table, mode="append" | "overwrite", partition_by=[...]) is the one-liner. Partitioned writes, schema evolution (with schema_mode="merge"), overwrite of specific partitions (partition_filters=[...]), and Z-ORDER (via OPTIMIZE) are all supported.
  • MERGE / UPSERT. dt.merge(source, predicate, source_alias, target_alias).when_matched_update(...).when_not_matched_insert(...).execute() is the production-ready UPSERT path. Delta 3.x compatibility means the transaction log stays interoperable with Spark readers.
  • OPTIMIZE + VACUUM. dt.optimize.compact() bin-packs small files into larger targets; dt.optimize.z_order(["col"]) clusters data by high-cardinality columns; dt.vacuum(retention_hours=168) soft-deletes files older than a week that are no longer referenced. All three run in-process, no cluster required.

The Delta transaction log — what deltalake reads and writes.

  • _delta_log/ folder. Sits next to your data files (typically s3://bucket/table/_delta_log/). Contains numbered JSON commit files (00000000000000000001.json, etc.) and periodic Parquet checkpoints (00000000000000000010.checkpoint.parquet).
  • Commit files. Each one records the operation (add / remove file, metadata change, protocol change) plus a commitInfo object with author + timestamp. deltalake reads these bottom-up to reconstruct the current table state.
  • Checkpoints. Written every 10 commits by default; a Parquet file containing the compacted state of all add / remove actions up to that commit. Loading a table with a fresh checkpoint is O(1) commits behind rather than O(all history).
  • Protocol versions. Delta 3.x uses minReaderVersion=3, minWriterVersion=7 for tables with modern features (column mapping, deletion vectors). deltalake 0.15+ supports these; older versions will refuse tables with features they don't understand.

Interop via Arrow — the contract that keeps everything simple.

  • Reads emit Arrow. dt.to_pyarrow_dataset() and dt.to_pyarrow_table() return PyArrow types; from there, Pandas / Polars / DuckDB / Ray Data / any Arrow-native consumer takes over. Zero-copy where possible.
  • Writes accept Arrow. write_deltalake(path, arrow_table, ...) accepts a pyarrow.Table (or Pandas / Polars via easy conversion). One writer signature covers everything.
  • Push-down. Predicate and column push-down happen in the Rust scanner before Arrow batches cross the FFI boundary, so filter selectivity translates directly to reduced I/O.

Common interview probes on deltalake python.

  • "How does deltalake read the Delta transaction log?" — required answer: it parses _delta_log/*.json + checkpoints in Rust, replays add / remove actions to reconstruct the current version.
  • "Can deltalake MERGE?" — required answer: yes, .merge(source, predicate).when_matched_update().when_not_matched_insert().execute() is production-ready in 0.15+.
  • "Is the write protocol interoperable with Spark?" — required answer: yes, Delta 3.x transaction log is a shared contract; a Spark reader can read a deltalake-written table and vice versa.
  • "How do you VACUUM safely?" — required answer: dt.vacuum(retention_hours=168) after checking that no reader will need older versions; default retention protects against readers with 7-day time-travel needs.

Worked example — reading a Delta table into Polars for a notebook analysis

Detailed explanation. The canonical Rust-native Delta read: a data-analyst notebook opens a partitioned orders Delta table on S3, filters to the last 30 days for a specific region, and joins to a small dimension frame. The whole pipeline runs in a Jupyter kernel with 1 GB of RAM and no cluster. Walk through the read.

  • Table. s3://acme-lake/orders, partitioned by order_date (daily) and region (US / EU / APAC).
  • Query. Last 30 days, region = 'US', project order_id, customer_id, total_cents, order_date.
  • Downstream. Polars DataFrame for group-by + join with customers frame.

Question. Write the notebook code that opens the Delta table, applies predicate + column push-down, and lands the result in Polars.

Input.

Parameter Value
Table path s3://acme-lake/orders
Partitioning order_date (daily), region
Filter last 30 days, region='US'
Projection 4 columns
Consumer Polars

Code.

# Notebook — read Delta into Polars, no Spark
from deltalake import DeltaTable
import pyarrow.compute as pc
import polars as pl
from datetime import date, timedelta

dt = DeltaTable("s3://acme-lake/orders")

# 1. Push-down predicate + column list into the Rust scanner
since = date.today() - timedelta(days=30)
scan = (
    dt.to_pyarrow_dataset()
      .filter((pc.field("order_date") >= pc.scalar(since)) &
              (pc.field("region")     == pc.scalar("US")))
      .to_table(columns=["order_id", "customer_id", "total_cents", "order_date"])
)

# 2. Zero-copy Arrow → Polars
orders = pl.from_arrow(scan)

# 3. Continue in Polars (group-by + join)
revenue = (
    orders.group_by("order_date")
          .agg(pl.col("total_cents").sum().alias("revenue_cents"))
          .sort("order_date")
)
print(revenue.head())
Enter fullscreen mode Exit fullscreen mode

Step-by-step explanation.

  1. DeltaTable("s3://acme-lake/orders") parses the transaction log via the Rust scanner. On a fresh table with a recent checkpoint, this is 30-80 ms — one Parquet checkpoint read plus a handful of JSON commits since the checkpoint.
  2. to_pyarrow_dataset() returns a PyArrow Dataset that carries the full add-file list. Applying .filter(...) and .to_table(columns=[...]) triggers the Rust scanner to (a) prune partitions that don't match order_date >= since and region == 'US', (b) open only the surviving Parquet files, (c) read only the requested columns.
  3. Because both filter columns are partition keys, partition pruning discards ~29/30 * 3 = ~29/90 of the files before any I/O. This is the critical optimisation: the Rust scanner uses the transaction log's file-level statistics (min / max per column) to skip whole files even for non-partition predicates.
  4. pl.from_arrow(scan) is a zero-copy conversion from PyArrow to Polars because both share the Arrow memory layout. The resulting Polars frame is ready for group-by / join / any Polars operation.
  5. The whole flow — open table, plan scan, execute read, land in Polars — completes in 200-800 ms for a modest US region slice, all inside a 1 GB notebook. Compare to a PySpark-in-a-notebook equivalent: 5-10 s just to boot Spark before any read happens.

Output.

shape: (30, 2)
┌────────────┬────────────────┐
│ order_date ┆ revenue_cents  │
├────────────┼────────────────┤
│ 2026-06-21 ┆ 12,485,300     │
│ 2026-06-22 ┆ 11,982,100     │
│ 2026-06-23 ┆ 13,401,050     │
│ …          ┆ …              │
│ 2026-07-20 ┆ 14,120,900     │
└────────────┴────────────────┘
Enter fullscreen mode Exit fullscreen mode

Rule of thumb. For any notebook or small-service Delta read, use DeltaTable(path).to_pyarrow_dataset().filter(...).to_table(columns=[...]) and hand the result to Polars via pl.from_arrow. Push down partition filters and column projections at the same call; the Rust scanner combines them into one optimised plan.

Worked example — partitioned write + MERGE / UPSERT from Python

Detailed explanation. The deltalake package's MERGE is the operation that unlocks CDC-style writes and slowly-changing-dimension pipelines without Spark. Walk through a canonical upsert: a Lambda receives a batch of updated customer records and merges them into a partitioned customers Delta table.

  • Target. s3://acme-lake/customers, partitioned by country.
  • Source. In-memory Arrow batch of ~500 changed customer rows.
  • Merge predicate. target.customer_id = source.customer_id.
  • On match. Update email, updated_at; keep created_at.
  • On no match. Insert full row.

Question. Write the Python function that performs the MERGE against a partitioned Delta table and returns the number of rows inserted vs updated.

Input.

Parameter Value
Target path s3://acme-lake/customers
Partition key country
Source Arrow batch (~500 rows)
Merge key customer_id
Update columns email, updated_at

Code.

# Lambda handler — upsert customers via deltalake MERGE
from deltalake import DeltaTable, write_deltalake
import pyarrow as pa
from datetime import datetime, timezone

TARGET = "s3://acme-lake/customers"

def upsert_customers(source_batch: pa.Table) -> dict[str, int]:
    """MERGE `source_batch` into the customers Delta table; return counts."""
    dt = DeltaTable(TARGET)

    result = (
        dt.merge(
            source        = source_batch,
            predicate     = "target.customer_id = source.customer_id",
            source_alias  = "source",
            target_alias  = "target",
        )
        .when_matched_update(
            updates = {
                "email":      "source.email",
                "updated_at": "source.updated_at",
            }
        )
        .when_not_matched_insert_all()
        .execute()
    )

    return {
        "rows_updated":  result["num_target_rows_updated"],
        "rows_inserted": result["num_target_rows_inserted"],
        "rows_scanned":  result["num_target_rows_scanned"],
    }


# One-time initial write — partitioned by country
def bootstrap_table(initial_batch: pa.Table) -> None:
    write_deltalake(
        TARGET,
        initial_batch,
        mode          = "overwrite",
        partition_by  = ["country"],
        schema_mode   = "overwrite",
    )
Enter fullscreen mode Exit fullscreen mode

Step-by-step explanation.

  1. DeltaTable(TARGET) opens the table — 30-80 ms including transaction-log parse. The Rust scanner also loads per-file statistics so the MERGE planner can prune Parquet files whose min/max customer_id range doesn't overlap the source batch.
  2. .merge(source, predicate, source_alias, target_alias) starts the builder. The predicate is the join expression; source_alias and target_alias disambiguate the two sides in the update / insert expressions.
  3. .when_matched_update(updates={...}) specifies the columns to overwrite when a source row matches an existing target row. .when_not_matched_insert_all() inserts the full source row (all columns) when no target row matches. You can chain multiple when_matched / when_not_matched clauses with additional predicates for SCD-2 patterns.
  4. .execute() runs the merge as a single Delta transaction: it reads only the Parquet files touched by matching keys, rewrites those files with the updated rows and newly-inserted rows, and appends a new transaction-log commit that removes the old files and adds the new ones. Readers on old versions still see consistent state; readers on the new version see the merged result.
  5. The result dict returns per-transaction counts — num_target_rows_updated, num_target_rows_inserted, num_target_rows_scanned, plus num_output_rows. Emit these to CloudWatch / Datadog to detect regressions (e.g. an unexpected update-vs-insert ratio signals a source-data drift).

Output.

Metric Value
rows_updated 342
rows_inserted 158
rows_scanned 500
Files rewritten 3 (out of ~40 partitions touched)
Wall time 380 ms
Delta commit version 42 → 43

Rule of thumb. For any deltalake MERGE, use .when_matched_update(...) with an explicit column list and .when_not_matched_insert_all() for the insert side; emit the return-dict counts to your monitoring stack; and pre-check the predicate has a highly-selective partition filter so the merge planner can prune files aggressively. Blind full-table merges are the one anti-pattern.

Worked example — OPTIMIZE + VACUUM in-process

Detailed explanation. After hundreds of small write_deltalake(..., mode='append') calls (e.g. one Lambda per S3 event), the table accumulates many tiny Parquet files — the classic "small-file problem." deltalake ships OPTIMIZE (bin-packing compaction) and VACUUM (soft-delete of stale files) so you can maintain the table from Python without launching Spark. Walk through the daily maintenance job.

  • Table. s3://acme-lake/orders, partitioned by order_date.
  • Target file size. 256 MB (delta-rs default is 268435456 bytes = 256 MiB).
  • Retention. 168 hours (7 days) for VACUUM — protects readers with a week of time-travel needs.

Question. Write the daily maintenance job that OPTIMIZEs the current day's partition and VACUUMs stale files older than the retention window.

Input.

Parameter Value
Target path s3://acme-lake/orders
Partition today's order_date
Target file size 256 MiB
VACUUM retention 168 h
Runner small ECS Fargate task (0.5 vCPU, 1 GB)

Code.

# Daily maintenance — runs on a small Fargate task, no Spark
from deltalake import DeltaTable
from datetime import date

TABLE = "s3://acme-lake/orders"

def daily_maintenance() -> dict:
    dt = DeltaTable(TABLE)

    # 1. OPTIMIZE — bin-pack today's partition into ~256 MiB files
    today   = date.today().isoformat()
    optim   = dt.optimize.compact(
        partition_filters = [("order_date", "=", today)],
        target_size       = 256 * 1024 * 1024,   # 256 MiB
    )

    # 2. Z-ORDER on a high-cardinality column for skipping downstream
    zord    = dt.optimize.z_order(
        columns           = ["customer_id"],
        partition_filters = [("order_date", "=", today)],
    )

    # 3. VACUUM — delete files no longer referenced, older than retention
    vac     = dt.vacuum(
        retention_hours   = 168,
        dry_run           = False,
        enforce_retention_duration = True,
    )

    return {
        "files_added_by_compact":     optim["numFilesAdded"],
        "files_removed_by_compact":   optim["numFilesRemoved"],
        "files_added_by_zorder":      zord["numFilesAdded"],
        "files_vacuumed":             len(vac),
    }
Enter fullscreen mode Exit fullscreen mode

Step-by-step explanation.

  1. dt.optimize.compact(partition_filters=..., target_size=...) scans the specified partition, groups small Parquet files whose combined size would fit under target_size, and rewrites them as fewer larger files. The transaction log records the swap atomically — a reader mid-scan either sees the old files or the new files, never both.
  2. partition_filters=[("order_date", "=", today)] scopes compaction to a single day. Compacting the whole table daily would be wasteful; scoping to today's partition keeps the job small and predictable.
  3. dt.optimize.z_order(columns=[...]) performs Z-ORDER clustering on the specified columns within each file. High-cardinality columns like customer_id benefit most — downstream point-lookup queries can skip whole files by leveraging Delta's per-file stats (min/max per column) that the Z-order write leaves in a tight range.
  4. dt.vacuum(retention_hours=168) soft-deletes Parquet files that are no longer referenced by any Delta commit newer than now - 168h. This is the storage-cost cleanup: OPTIMIZE dereferences old small files but they stay on S3 until VACUUM removes them. enforce_retention_duration=True rejects retention values shorter than the default safeguard.
  5. The whole maintenance job runs in a 0.5 vCPU / 1 GB Fargate task in 30 s - 2 min depending on daily volume. Compared to launching a Spark cluster for the same maintenance, this is a step-change in cost: pennies per day vs dollars.

Output.

Operation Files before Files after Bytes rewritten
OPTIMIZE compact 187 (avg 8 MB) 6 (avg 250 MB) 1.5 GB
Z-ORDER on customer_id 6 6 (re-sorted) 1.5 GB
VACUUM (168h retention) (187 stale) 0 stale freed 1.5 GB S3
Delta commit versions used v42 → v43 (compact) v43 → v44 (zorder) v44 (vacuum)
Total wall time 45 s

Rule of thumb. Ship an OPTIMIZE + VACUUM daily maintenance job the day you deploy deltalake writes. Scope OPTIMIZE with partition_filters so you never compact the entire table in one shot; run VACUUM with enforce_retention_duration=True at ≥ 168h retention unless you can prove no reader needs older time-travel. Skipping maintenance is the fastest way to turn a fast deltalake table into a slow one.

Senior interview question on delta-rs

A senior interviewer might ask: "You are building a Python-based ingestion service that receives ~5000 CDC events per minute for a customers table and needs to upsert them into a partitioned Delta table on S3. The consumers are a Spark warehouse job that reads nightly and a Trino cluster that reads ad-hoc. Walk me through the delta-rs design — the batching strategy, the MERGE call, the OPTIMIZE + VACUUM maintenance, and the interoperability guarantees with the Spark and Trino readers."

Solution Using batched delta-rs MERGE with per-minute micro-batches, hourly OPTIMIZE, and Delta 3.x protocol pinning 

# 1. Ingestion service — batches CDC events into per-minute micro-batches
import asyncio, boto3, json
import pyarrow as pa
from collections import defaultdict
from deltalake import DeltaTable
from datetime import datetime, timezone

TARGET = "s3://acme-lake/customers"
BATCH_INTERVAL_SEC = 60
MAX_BATCH_ROWS     = 20_000

class CustomerCDCIngester:
    def __init__(self) -> None:
        self.buffer: list[dict] = []
        self.dt = DeltaTable(TARGET)

    async def enqueue(self, event: dict) -> None:
        self.buffer.append(event)
        if len(self.buffer) >= MAX_BATCH_ROWS:
            await self._flush()

    async def _flush(self) -> None:
        if not self.buffer:
            return
        batch, self.buffer = self.buffer, []

        # Deduplicate by customer_id, keep latest updated_at
        latest: dict[int, dict] = {}
        for ev in batch:
            existing = latest.get(ev["customer_id"])
            if not existing or ev["updated_at"] > existing["updated_at"]:
                latest[ev["customer_id"]] = ev

        source = pa.Table.from_pylist(list(latest.values()))

        result = (
            self.dt.merge(
                source        = source,
                predicate     = "target.customer_id = source.customer_id",
                source_alias  = "source",
                target_alias  = "target",
            )
            .when_matched_update(updates={
                "email":      "source.email",
                "phone":      "source.phone",
                "updated_at": "source.updated_at",
            })
            .when_not_matched_insert_all()
            .execute()
        )
        print(json.dumps({
            "ts":       datetime.now(timezone.utc).isoformat(),
            "rows_in":  len(batch),
            "rows_dedupe": len(latest),
            **result,
        }))

    async def run(self) -> None:
        while True:
            await asyncio.sleep(BATCH_INTERVAL_SEC)
            await self._flush()
Enter fullscreen mode Exit fullscreen mode 
# 2. Hourly OPTIMIZE + Z-ORDER — runs on small Fargate task
from deltalake import DeltaTable

def hourly_maintenance() -> None:
    dt = DeltaTable(TARGET)
    dt.optimize.compact(target_size=256 * 1024 * 1024)
    dt.optimize.z_order(columns=["customer_id"])

# 3. Daily VACUUM — separate schedule, 168h retention
def daily_vacuum() -> None:
    dt = DeltaTable(TARGET)
    dt.vacuum(retention_hours=168, enforce_retention_duration=True)
Enter fullscreen mode Exit fullscreen mode 
# 4. Protocol pinning — ensure Delta 3.x compatibility with Spark + Trino
from deltalake import DeltaTable
dt = DeltaTable(TARGET)
protocol = dt.protocol()
assert protocol.min_reader_version <= 3
assert protocol.min_writer_version <= 7
Enter fullscreen mode Exit fullscreen mode

Step-by-step trace.

Concern Answer Reasoning
Ingest rate 5000 events/min fits comfortably in per-minute micro-batches
Batch dedupe in-memory by (customer_id, max updated_at) avoids MERGE ping-pong on same key
Write path delta-rs MERGE per batch atomic Delta commit
Compaction hourly OPTIMIZE + Z-ORDER keeps file sizes ~256 MiB
Retention 168 h VACUUM protects 7-day time-travel
Protocol Delta reader 3, writer 7 Spark + Trino read the same table

After deployment, the ingester writes one Delta commit per minute (~1440 commits/day), hourly OPTIMIZE rewrites 100+ small files into ~5 larger ones per partition, daily VACUUM frees dereferenced storage; Spark and Trino read the same table version-by-version without lock contention because Delta's optimistic concurrency handles the multi-reader case natively.

Output:

Metric Value
Ingest throughput 5000 events/min sustained
MERGE latency p99 800 ms per micro-batch
Files per partition (peak) 60
Files per partition (post-OPTIMIZE) ~5
S3 storage freed by daily VACUUM 3-8 GB
Spark reader compatibility full (Delta 3.x)
Trino reader compatibility full (Delta 3.x)
Cost per month (compute) ~$120 (ingest + maintenance)

Why this works — concept by concept:

  • delta-rs MERGE with per-minute micro-batches — batching amortises the transaction-log commit cost over ~20k events per commit rather than one per event; the Rust scanner reads only Parquet files whose min/max customer_id range overlaps the batch, so MERGE stays O(touched-files) not O(table).
  • In-memory dedupe by (key, max updated_at) — CDC streams routinely deliver multiple updates for the same row within one batch; folding them to the latest before the MERGE avoids "update then update again" wasted I/O and keeps the write path idempotent under at-least-once delivery.
  • Hourly OPTIMIZE + Z-ORDER — the small-file problem is the #1 delta-rs performance regression at scale. Compaction to 256 MiB targets keeps Parquet row-group sizes healthy; Z-ORDER on customer_id tightens per-file stats so downstream point-lookup queries can skip files.
  • Delta 3.x protocol pinning — Delta's protocol versions gate reader / writer feature compatibility. Pinning minReaderVersion <= 3 and minWriterVersion <= 7 guarantees Spark and Trino can read the same table; enabling higher-version features (deletion vectors, column mapping) would silently exclude older readers.
  • Cost — per-minute writes at ~800 ms p99, hourly maintenance at ~30 s per run, daily VACUUM at ~5 min; total compute cost ~$120/month vs ~$800/month for the same workload on a Spark cluster. Interoperability with Spark and Trino consumers is preserved because the Delta transaction log is a shared protocol. O(1) per commit on the ingest side; the readers pay their own scan costs.

Aggregation
Topic — aggregation
Aggregation problems on Delta and lakehouse tables

Practice →

SQL Topic — sql SQL practice on Delta MERGE and upsert patterns

Practice →

3. PyIceberg deep dive

pyiceberg is a Python client with Rust hot paths — read + write Iceberg tables against Glue, REST, Hive, Nessie, Polaris, or a local SQL catalog

The mental model in one line: PyIceberg is the Apache Iceberg project's Python client — pure Python for the surface API but with Rust hot loops for Arrow decode and manifest parsing — that speaks the Iceberg catalog protocol (Glue, Hive Metastore, Iceberg REST, Nessie, Snowflake Polaris, and a SQL-backed catalog for local dev), plans scans against snapshot metadata, executes them through PyArrow, and supports append + overwrite writes plus a maturing positional-delete story so Python-first teams can serve, mutate, and time-travel Iceberg tables without launching Spark or Trino. Every senior data engineer who has evaluated iceberg python in 2026 has settled on the same three catalog choices (Glue for AWS, REST for open lakehouse, SQL for local dev) and the same three write operations (append, overwrite, positional delete) as the production-ready subset.

Iconographic PyIceberg diagram — a Python client card connecting to four catalog cards (Glue, REST, Hive, SQL), a table.scan() card feeding Arrow / Polars / DuckDB, and a snapshot-history ribbon.

The four operational axes for iceberg python.

  • Catalog choice. Iceberg requires a catalog — unlike Delta, there is no "just point at the folder" option. PyIceberg's load_catalog(name, **props) speaks Glue, Hive Metastore, Iceberg REST, Nessie, Snowflake Polaris, and SQL (via sqlite:/// or postgresql:// for local dev). Pick one per environment; production is usually Glue or REST.
  • Reading. catalog.load_table("db.table") returns a Table object; table.scan(row_filter=..., selected_fields=(...), snapshot_id=...) plans a scan; .to_arrow(), .to_polars(), .to_duckdb(), .to_pandas() materialise. Predicate + column push-down are respected; partition + file-level statistics prune the file list before I/O.
  • Writing. table.append(df) and table.overwrite(df, overwrite_filter=...) are the two production-ready write paths. Positional deletes (row-level MERGE analogue) are maturing through 2026 releases. Schema evolution and partition evolution have a dedicated table.update_schema() / table.update_spec() API.
  • Time-travel. table.scan(snapshot_id=...) reads a historical snapshot; table.snapshots() lists the snapshot history; a snapshot ID identifies a point-in-time state precisely. as_of_timestamp semantics are supported via snapshot lookup.

Catalog config templates — the six real options.

  • AWS Glue. load_catalog("glue", **{"type": "glue", "s3.region": "us-east-1"}) — the AWS default; PyIceberg picks up the ambient credential chain (env vars, IAM role, ~/.aws/credentials).
  • Iceberg REST Catalog. load_catalog("rest", **{"type": "rest", "uri": "https://polaris.acme.internal/api/catalog", "credential": "client_id:client_secret", "scope": "PRINCIPAL_ROLE:ALL"}) — the open-protocol default; talks to Polaris, Tabular, LakeFS Iceberg, or any Iceberg-REST-Spec 1.6+ server.
  • Hive Metastore. load_catalog("hive", **{"type": "hive", "uri": "thrift://hive-metastore.internal:9083"}) — the legacy option; still ships in on-prem and older cloud deployments.
  • Nessie. load_catalog("nessie", **{"type": "nessie", "uri": "https://nessie.internal/api/v1"}) — the git-like branching option; strong on data-versioning use cases.
  • Snowflake Polaris. Uses the REST type with uri=https://<account>.snowflakecomputing.com/polaris/api/catalog/v1/<catalog>; supports SigV4 or OAuth2 authentication.
  • SQL (local dev). load_catalog("sql", **{"type": "sql", "uri": "sqlite:///:memory:", "warehouse": "file:///tmp/warehouse"}) — the "runs on my laptop" option; excellent for tests + tutorials.

Rust in the hot paths — the acceleration story.

  • Arrow decode. Parquet reading goes through pyarrow, which is C++/Arrow under the hood; PyIceberg calls into it with pre-computed offset / length pairs from the manifest scan.
  • Manifest parsing. Manifest files (Avro) list every data file in a snapshot with per-column stats. Parsing them at scale is the hot path Iceberg engines optimise; PyIceberg has a Rust reader for this in newer releases.
  • Predicate pruning. Row-filter → partition-filter simplification uses PyIceberg's expression tree in Python, but the resulting per-file skip decisions apply against Rust-decoded manifest stats.
  • Vectorised reads. Once the file list is planned, the actual data read is vectorised via PyArrow — batches sized to the row-group boundaries in the underlying Parquet.

Common interview probes on iceberg python.

  • "How does PyIceberg discover tables?" — required answer: through the catalog (Glue / REST / Hive / etc.); Iceberg requires a catalog unlike Delta's transaction-log-only mode.
  • "Can PyIceberg write?" — required answer: yes, .append() and .overwrite() are production-ready; positional deletes maturing.
  • "How does time-travel work?" — required answer: .scan(snapshot_id=...); the snapshot list is table.snapshots(); each snapshot is a durable point-in-time state.
  • "How does PyIceberg compare to using Trino for Iceberg?" — required answer: PyIceberg is single-process (great for notebooks + Lambda); Trino is distributed (great for interactive SQL and > 100 GB); they read the same tables.

Worked example — reading an Iceberg table from Glue into DuckDB

Detailed explanation. The canonical iceberg python read: a Lambda uses PyIceberg's Glue catalog integration to open a partitioned events Iceberg table, applies predicate + column push-down, and hands the result to DuckDB for SQL analysis — all in one Python process with no cluster. Walk through the flow.

  • Catalog. AWS Glue with the data_lake database.
  • Table. data_lake.events, partitioned by event_date (day-partitioned) and event_type.
  • Query. Yesterday's page_view events, project event_id, user_id, page_url, event_ts.
  • Consumer. DuckDB in-process for a group-by SQL query.

Question. Write the Lambda code that loads the table via Glue, scans with pushed-down filters, and runs a DuckDB SQL query on the result.

Input.

Parameter Value
Catalog AWS Glue
Database data_lake
Table events
Partitioning event_date, event_type
Filter yesterday, event_type='page_view'
Consumer DuckDB

Code.

# Lambda — read Iceberg from Glue into DuckDB, no Spark
from pyiceberg.catalog import load_catalog
from pyiceberg.expressions import And, EqualTo
from datetime import date, timedelta
import duckdb

# 1. Load the Glue catalog (uses ambient AWS credentials)
catalog = load_catalog(
    "glue",
    **{
        "type":      "glue",
        "s3.region": "us-east-1",
    },
)

# 2. Open the Iceberg table
table = catalog.load_table("data_lake.events")

# 3. Plan the scan with predicate + column push-down
yesterday = (date.today() - timedelta(days=1)).isoformat()

scan = table.scan(
    row_filter        = And(
        EqualTo("event_date", yesterday),
        EqualTo("event_type", "page_view"),
    ),
    selected_fields   = ("event_id", "user_id", "page_url", "event_ts"),
)

# 4. Materialise to PyArrow (Rust-accelerated decode)
arrow_table = scan.to_arrow()

# 5. Hand off to DuckDB for SQL
duck = duckdb.connect()
duck.register("events_slice", arrow_table)
top_pages = duck.execute("""
    SELECT   page_url,
             COUNT(*)        AS views,
             COUNT(DISTINCT user_id) AS uniques
    FROM     events_slice
    GROUP BY page_url
    ORDER BY views DESC
    LIMIT    10
""").fetchall()

print(top_pages)
Enter fullscreen mode Exit fullscreen mode

Step-by-step explanation.

  1. load_catalog("glue", **{"type": "glue", "s3.region": "us-east-1"}) initialises the Glue catalog client. PyIceberg reads AWS_REGION, AWS_ACCESS_KEY_ID, and the IAM role from the ambient credential chain — no explicit boto3 setup needed in a Lambda that already has an IAM role.
  2. catalog.load_table("data_lake.events") calls Glue's GetTable API and reads the metadata pointer (metadata_location from the Glue table properties), then fetches the Iceberg metadata JSON from S3. This gives PyIceberg the current snapshot + partition spec + schema. Cost: ~1 Glue API call + 1 S3 GET, typically 40-120 ms.
  3. table.scan(row_filter=..., selected_fields=...) builds a scan plan. The And(EqualTo("event_date", yesterday), EqualTo("event_type", "page_view")) filter is analysed against the partition spec — both filter columns are partition keys, so partition pruning trims the manifest read to just yesterday's page_view partition.
  4. .to_arrow() executes the scan: PyIceberg fetches the surviving manifest files (Avro), further prunes data files by per-column min/max stats, then reads only the required columns from the surviving Parquet files. The Arrow output is materialised in Python.
  5. duck.register("events_slice", arrow_table) gives DuckDB zero-copy access to the Arrow data (via Arrow C data interface). The GROUP BY + ORDER BY + LIMIT runs in DuckDB's vectorised execution engine, in the same Python process, in tens of milliseconds. Total end-to-end: 250-800 ms for a moderate day-partition read.

Output.

[('/home',      12480, 8210),
 ('/pricing',    8920, 6420),
 ('/blog',       6750, 5810),
 ('/docs',       5310, 4020),
 ('/product/a',  4180, 3220),
 ('/product/b',  3240, 2510),
 ('/checkout',   1920, 1520),
 ('/faq',        1610, 1280),
 ('/about',      1280,  970),
 ('/careers',     910,  740)]
Enter fullscreen mode Exit fullscreen mode

Rule of thumb. For any PyIceberg read in a Lambda or notebook, load the catalog once at module scope (not per-invocation), scope row_filter to partition columns whenever possible for maximum pruning, and hand off to Arrow / Polars / DuckDB via .to_arrow() / .to_polars() / .to_duckdb() rather than materialising to Pandas. The zero-copy Arrow interchange is the reason PyIceberg is fast.

Worked example — writing to Iceberg — append and overwrite

Detailed explanation. PyIceberg's .append(df) and .overwrite(df, overwrite_filter=...) are the two production-ready write paths. Walk through appending a batch of new order events, then overwriting a specific partition when a late-arriving correction lands.

  • Append. Batch of 5000 new order rows for today's partition.
  • Overwrite. Yesterday's partition needs to be rewritten because a correction batch arrived.
  • Schema. orders(order_id BIGINT, customer_id BIGINT, total_cents BIGINT, status STRING, order_date DATE).
  • Partition spec. order_date (day-partitioned).

Question. Write the two functions — append_orders and overwrite_partition — that use PyIceberg's write API against a Glue-catalogued Iceberg table.

Input.

Parameter Value
Table data_lake.orders
Partition order_date
Append batch Arrow table (5000 rows)
Overwrite filter EqualTo("order_date", yesterday)

Code.

# Iceberg writes via PyIceberg
from pyiceberg.catalog import load_catalog
from pyiceberg.expressions import EqualTo
from datetime import date, timedelta
import pyarrow as pa

catalog = load_catalog("glue", **{"type": "glue", "s3.region": "us-east-1"})

def append_orders(batch: pa.Table) -> str:
    """Append a batch of new orders; return the new snapshot ID."""
    table = catalog.load_table("data_lake.orders")

    # PyIceberg partitions the batch by the current partition spec
    # and writes one data file per (partition, batch) tuple
    table.append(batch)

    # Refresh + return the new snapshot ID
    table = catalog.load_table("data_lake.orders")
    return str(table.current_snapshot().snapshot_id)


def overwrite_partition(correction_batch: pa.Table, partition_date: date) -> str:
    """Overwrite all rows in `partition_date`'s partition with `correction_batch`."""
    table = catalog.load_table("data_lake.orders")

    table.overwrite(
        df                = correction_batch,
        overwrite_filter  = EqualTo("order_date", partition_date.isoformat()),
    )

    table = catalog.load_table("data_lake.orders")
    return str(table.current_snapshot().snapshot_id)
Enter fullscreen mode Exit fullscreen mode

Step-by-step explanation.

  1. catalog.load_table("data_lake.orders") reads the current Iceberg metadata JSON from S3 (via the Glue table's metadata_location), giving PyIceberg the schema + partition spec + current snapshot ID.
  2. table.append(batch) (a) validates the Arrow schema against the Iceberg schema, (b) partitions the batch by the current partition spec (order_date), (c) writes one Parquet data file per resulting partition, (d) writes a new manifest file listing the new data files, (e) writes a new manifest list, (f) writes a new metadata JSON with a new snapshot ID, (g) commits the pointer via the catalog (GlueUpdateTable). All seven steps are atomic — either the commit succeeds or the whole operation rolls back.
  3. table.overwrite(df, overwrite_filter=EqualTo("order_date", ...)) is the same flow with one addition: before writing, PyIceberg synthesises a "delete files matching the filter" operation, then adds the new files. In Iceberg terms this is a "replace" snapshot — the old data files for that partition are dereferenced (visible in older snapshots for time-travel; GC'd by expire_snapshots later); the new data files are added.
  4. The catalog commit (step g) is where multi-writer coordination happens. Iceberg uses optimistic concurrency: PyIceberg tells Glue "advance the metadata pointer from snapshot X to snapshot Y." If another writer beat you to it, Glue rejects the commit; PyIceberg retries with a fresh base snapshot. Two writers cannot silently overwrite each other.
  5. The returned snapshot ID is the durable identifier for the new state. Downstream consumers that want reproducible reads can .scan(snapshot_id=...) against it — a permanent, immutable pointer to exactly this write.

Output.

Operation Files added Files dereferenced New snapshot ID Wall time
append_orders (5000 rows) 1-3 (per partition touched) 0 8213456789012345678 450 ms
overwrite_partition (7000 correction rows) 2 (rewritten) 4 (old yesterday files) 8213456789012345679 620 ms

Rule of thumb. For PyIceberg writes, use .append() for insert-only paths and .overwrite(overwrite_filter=...) for scoped rewrites (partition corrections, late-arriving data reprocessing). Never call .overwrite() without a filter unless you intend to replace the whole table — the filter-less call is the closest thing to DROP TABLE + CREATE + INSERT.

Worked example — schema evolution + partition evolution

Detailed explanation. One of Iceberg's headline features is schema evolution and partition evolution as first-class operations — you can add columns, drop columns, rename columns, and change the partition spec on a live table without rewriting data. PyIceberg exposes this via .update_schema() and .update_spec(). Walk through adding a region column, then evolving the partition spec to include it.

  • Starting schema. orders(order_id, customer_id, total_cents, status, order_date).
  • Change 1. Add region STRING column.
  • Change 2. Evolve partition spec from [order_date] to [order_date, bucket(region, 4)].

Question. Perform both evolutions and show how PyIceberg preserves reader compatibility.

Input.

Parameter Value
Table data_lake.orders
New column region STRING NULL
New partition spec [order_date, bucket(region, 4)]
Existing partition spec [order_date]

Code.

# Schema evolution + partition evolution via PyIceberg
from pyiceberg.catalog import load_catalog
from pyiceberg.schema import Schema
from pyiceberg.types import StringType, NestedField
from pyiceberg.transforms import BucketTransform, IdentityTransform

catalog = load_catalog("glue", **{"type": "glue", "s3.region": "us-east-1"})
table   = catalog.load_table("data_lake.orders")

# 1. Add the `region` column (schema evolution)
with table.update_schema() as upd:
    upd.add_column("region", StringType(), required=False)

# 2. Evolve the partition spec to add bucket(region, 4)
with table.update_spec() as upd:
    upd.add_field("region", BucketTransform(4), "region_bucket")

# 3. Verify the new schema + spec
table = catalog.load_table("data_lake.orders")
print("Schema fields:", [f.name for f in table.schema().fields])
print("Partition spec:", table.spec())
Enter fullscreen mode Exit fullscreen mode

Step-by-step explanation.

  1. table.update_schema() returns a context-manager builder. .add_column("region", StringType(), required=False) stages a nullable column add; on exit the context manager commits a new metadata JSON with the extended schema. Existing data files remain untouched — their region column reads as NULL for old rows.
  2. Iceberg tracks columns by field ID, not by name, so region gets a fresh ID (e.g. 6) that never conflicts with existing IDs. This is why rename + drop + re-add operations are safe: the field ID is the durable identity.
  3. table.update_spec() evolves the partition spec. .add_field("region", BucketTransform(4), "region_bucket") adds a new partition dimension: hash-bucket the region column into 4 buckets. The new spec becomes the current spec; old spec versions are retained for reading historical files.
  4. New data written after the spec change is partitioned by the new spec ([order_date, region_bucket]). Old data still lives in the old spec ([order_date]). Iceberg's scan planner handles the mixed-spec case natively — you don't have to rewrite the old data.
  5. Readers get consistent semantics: SELECT * FROM data_lake.orders WHERE region = 'US' prunes both old-spec files (via file-level stats on region — which are NULL for old files, so all old files match and are read) and new-spec files (via partition pruning on region_bucket matching the hash of 'US'). Correctness is preserved; only new writes get the speedup.

Output.

Schema fields: ['order_id', 'customer_id', 'total_cents', 'status', 'order_date', 'region']
Partition spec: [
  1000: order_date: identity(order_date)
  1001: region_bucket: bucket[4](region)
]
Enter fullscreen mode Exit fullscreen mode

Rule of thumb. Iceberg's schema evolution and partition evolution are the reason to pick Iceberg over Delta if you expect frequent structural changes. table.update_schema() for column adds / drops / renames; table.update_spec() for repartitioning; both are atomic metadata operations that never rewrite existing data. Old writers using the old spec continue to work; readers see the union of specs transparently.

Senior interview question on PyIceberg

A senior interviewer might ask: "You need to build a Python microservice that appends events to a Glue-catalogued Iceberg table at 200 events/sec, exposes a read API that scans the table with predicate push-down for a downstream dashboard, and supports point-in-time queries via snapshot ID. Walk me through the PyIceberg design, the batching strategy, the catalog choice, the read API, and the failure semantics on Glue commit conflicts."

Solution Using PyIceberg with a 30-second micro-batch appender, snapshot-parameterised read API, and Glue commit-retry on conflict 

# 1. Micro-batch appender — 30-second flush interval
import asyncio, time, os
from collections import deque
import pyarrow as pa
from pyiceberg.catalog import load_catalog

CATALOG = load_catalog("glue", **{"type": "glue", "s3.region": os.environ["AWS_REGION"]})
TABLE_NAME = "data_lake.events"
FLUSH_SEC  = 30
MAX_BATCH  = 10_000

class EventAppender:
    def __init__(self) -> None:
        self.buf: deque[dict] = deque()
        self.table = CATALOG.load_table(TABLE_NAME)

    async def enqueue(self, event: dict) -> None:
        self.buf.append(event)
        if len(self.buf) >= MAX_BATCH:
            await self._flush()

    async def _flush(self) -> None:
        if not self.buf:
            return
        rows = list(self.buf)
        self.buf.clear()
        batch = pa.Table.from_pylist(rows)

        for attempt in range(5):
            try:
                self.table.append(batch)
                self.table = CATALOG.load_table(TABLE_NAME)  # refresh snapshot
                break
            except Exception as exc:
                if "concurrent" in str(exc).lower() and attempt < 4:
                    await asyncio.sleep(0.5 * (2 ** attempt))  # exp backoff
                    self.table = CATALOG.load_table(TABLE_NAME)
                    continue
                raise

    async def run_flusher(self) -> None:
        while True:
            await asyncio.sleep(FLUSH_SEC)
            await self._flush()
Enter fullscreen mode Exit fullscreen mode 
# 2. Read API — accepts optional snapshot_id for point-in-time reads
from fastapi import FastAPI, HTTPException
from pyiceberg.expressions import EqualTo, And

app = FastAPI()

@app.get("/events/{event_type}")
async def read_events(event_type: str,
                      day: str,
                      snapshot_id: int | None = None) -> dict:
    table = CATALOG.load_table(TABLE_NAME)

    scan = table.scan(
        row_filter        = And(
            EqualTo("event_type", event_type),
            EqualTo("event_date", day),
        ),
        selected_fields   = ("event_id", "user_id", "page_url", "event_ts"),
        snapshot_id       = snapshot_id,      # None = latest; int = historical
    )
    arrow = scan.to_arrow()

    return {
        "count":       arrow.num_rows,
        "snapshot_id": snapshot_id or table.current_snapshot().snapshot_id,
        "rows":        arrow.slice(0, 100).to_pylist(),   # cap payload
    }
Enter fullscreen mode Exit fullscreen mode 
# 3. Snapshot listing API — enables clients to time-travel
@app.get("/snapshots")
async def list_snapshots(limit: int = 20) -> list[dict]:
    table = CATALOG.load_table(TABLE_NAME)
    snaps = table.snapshots()[-limit:]
    return [
        {
            "snapshot_id": s.snapshot_id,
            "committed_at": s.timestamp_ms,
            "operation": s.summary.get("operation"),
        }
        for s in snaps
    ]
Enter fullscreen mode Exit fullscreen mode

Step-by-step trace.

Concern Answer Reasoning
Ingest rate 200 events/sec fits in per-30-second micro-batches
Batch size up to 10k rows one Iceberg commit per batch
Catalog AWS Glue AWS-native; no extra infra
Commit conflict exponential backoff retry Glue optimistic-lock rejection
Read API .scan(row_filter, snapshot_id) predicate + snapshot pushed to Rust scanner
Time-travel optional snapshot_id param serves point-in-time queries
Snapshot listing table.snapshots() enables client-side time-travel selection

After deployment, the appender writes ~2 Iceberg commits/minute (~2900/day); the read API serves 250-800 ms p99 reads with predicate push-down; the snapshot-parameterised query path lets clients pin a snapshot ID for reproducible dashboards; Glue commit conflicts (rare, ~1% under 5 concurrent writers) resolve on the first backoff retry.

Output:

Metric Value
Sustained append rate 200 events/sec
Micro-batch flush interval 30 s
Rows per commit (avg) 6000
Commit conflict rate ~1% (single-writer service)
Read API p99 latency 800 ms
Snapshot-history depth ~30 days (before expire_snapshots)
Cost per month ~$40 (Fargate + Glue API calls + S3 requests)

Why this works — concept by concept:

  • PyIceberg micro-batch appender — batching to 30 seconds amortises the Iceberg commit cost (metadata JSON + manifest list + manifest + Glue update) across ~6000 events per commit rather than one per event; the commit sequence is O(1) per batch not O(events).
  • Glue optimistic-lock retry — Iceberg's compare-and-swap on the current snapshot pointer means concurrent writers can conflict; PyIceberg surfaces this as an exception that the appender catches and retries with exponential backoff, always re-reading the base snapshot before the next attempt.
  • Snapshot-parameterised read API — .scan(snapshot_id=...) reads a historical snapshot; the snapshot ID is a durable, immutable pointer to a point-in-time state, so downstream dashboards can pin a snapshot for reproducibility without racing with new writes.
  • Predicate + column push-down — And(EqualTo(...), EqualTo(...)) is analysed against the partition spec so partition pruning kicks in; the surviving files are further pruned by per-column min/max stats before I/O. The Rust decode paths keep the actual read fast once file selection is done.
  • Cost — Fargate service ~$30/month, Glue API calls ~$5/month, S3 requests + storage ~$5/month; total ~$40/month vs ~$400/month for the same workload on a Spark cluster with an equivalent Iceberg writer. Reader latency stays interactive because Rust-accelerated Arrow decode dominates the scan phase. O(1) per batch on the write side; O(matching-files) on reads.

Design
Topic — design
Design problems on Iceberg-native Python services

Practice →

SQL Topic — sql SQL practice on Iceberg time-travel and snapshot queries

Practice →

4. Interop with the Python data stack

Polars, DuckDB, Ray Data, Lambda — the whole Python analytics stack now speaks Delta and Iceberg via delta-rs and PyIceberg

The mental model in one line: the Rust-native table format libraries are the shared implementation the entire Python analytics ecosystem now builds on — Polars ships scan_delta and scan_iceberg for lazy frames, DuckDB's Iceberg extension exposes iceberg_scan() for zero-copy Arrow reads, Ray Data / Dask / Modin all read via Arrow, and Lambda / Cloud Run functions finally have viable open-lakehouse readers because the 5-second JVM cold-start that made serverless untenable is gone — meaning your reader choice per workload is now genuinely orthogonal to the table format, and the "engine per workload" architecture that was aspirational in 2022 has become the 2026 default. Every senior Python-first data platform in 2026 is a mesh of Rust-backed readers on top of one or two table formats.

Iconographic Python-stack interop diagram — a central table-format hub linking to Polars, DuckDB, Ray Data, and a serverless Lambda card, with a local MinIO dev card at the bottom.

The four operational axes for polars delta + duckdb iceberg + Ray Data + serverless.

  • Polars. pl.scan_delta("s3://...") and pl.scan_iceberg(catalog_table) return LazyFrame objects that push filter / column pruning back into delta-rs / PyIceberg. Polars is the fastest single-node DataFrame engine in 2026 and pairs perfectly with these readers for < 100 GB analytics.
  • DuckDB. duckdb.execute("SELECT ... FROM delta_scan('s3://...')") and the Iceberg extension's iceberg_scan(...) expose table formats as first-class SQL sources. Zero-copy Arrow interchange means the file scan is done by delta-rs / PyIceberg while DuckDB does the SQL execution.
  • Ray Data / Dask / Modin. All three consume Arrow batches; delta-rs' to_pyarrow_dataset() and PyIceberg's .to_arrow() slot straight into ray.data.from_arrow(...), dask.dataframe.from_pandas(pyarrow.Table.to_pandas()), and Modin's Arrow adapter.
  • Serverless (Lambda / Cloud Run). The 30 MB deltalake wheel and the ~40 MB pyiceberg wheel both fit in a Lambda layer or a slim container image; boot time in the 128-256 MB tier is 60-300 ms; per-invocation cost is 10-50x cheaper than PySpark for the same table read.

Polars integration — scan_delta + scan_iceberg for lazy frames.

  • pl.scan_delta. Uses delta-rs under the hood. Returns a LazyFrame that supports the full Polars DSL (filter, select, group_by, join). Push-down happens at .collect() time — Polars translates the plan into predicate + column projection for delta-rs.
  • pl.scan_iceberg. Uses PyIceberg. Takes an IcebergTable object (loaded via pyiceberg.catalog.load_catalog(...).load_table(...)) and returns a LazyFrame. Predicate push-down + partition pruning are respected.
  • Streaming collect. .collect(streaming=True) on a lazy plan tells Polars to spill intermediate results to disk, letting you process datasets larger than RAM. Combined with delta-rs / PyIceberg's file-level pruning, this is how Python-first analytics scales to hundreds of GB without a cluster.
  • Sink patterns. .sink_parquet(...), .sink_delta(...) (via delta-rs write path), and .sink_csv(...) complete the read-transform-write loop entirely in Polars + delta-rs.

DuckDB integration — Iceberg + Delta extensions.

  • Delta. INSTALL delta; LOAD delta; then SELECT * FROM delta_scan('s3://acme-lake/orders'). Uses delta-rs under the hood.
  • Iceberg. INSTALL iceberg; LOAD iceberg; then SELECT * FROM iceberg_scan('s3://acme-lake/warehouse/db/orders/metadata/00042.metadata.json'). Direct-metadata scan; catalog integration via attach() in newer versions.
  • Attach a catalog. ATTACH 'iceberg:...' AS lake (TYPE ICEBERG, ...) (rolling out through 2026) lets you SELECT ... FROM lake.db.table with catalog discovery.
  • Cross-engine. The same table can be read by duckdb, polars, and deltalake/pyiceberg in one Python script — they all agree on the underlying file format and produce compatible Arrow.

Ray Data / Dask / Modin — Arrow as the interchange.

  • Ray Data. ray.data.from_arrow(dt.to_pyarrow_table()) (or read as pyarrow.dataset and stream). Ray's block layout is Arrow-native; delta-rs / PyIceberg output slots in without copy.
  • Dask. dd.from_pandas(dt.to_pyarrow_dataset().to_table().to_pandas()) is the pragmatic path; native dask.dataframe.read_delta and read_iceberg variants exist via community extensions.
  • Modin. mpd.DataFrame(dt.to_pyarrow_table().to_pandas()) — Modin drops in as a Pandas replacement; combined with delta-rs, gives distributed Pandas on top of Delta with no Spark.

Serverless — Lambda / Cloud Run / Cloud Functions.

  • Package size. deltalake ~30 MB wheel; pyiceberg ~40 MB wheel; pyarrow ~90 MB. Fits comfortably in a Lambda layer (250 MB uncompressed limit) or a 500 MB container image.
  • Cold-start budget. 60-300 ms in the 256 MB Lambda tier including import + first table open. Warm invocations under 50 ms including scan planning.
  • Local dev parity. Point at MinIO or S3-Local for local S3 emulation; use LocalStack for Glue emulation; PyIceberg's SQL catalog for local table metadata. The whole pipeline runs on a laptop with no cloud dependency.
  • When it doesn't work. Multi-TB scans, cross-region reads without VPC endpoints (network overhead dominates), or workloads that need Spark's Catalyst optimiser on 20-way joins. Escalate to Spark then, not for smaller work.

Common interview probes on interop.

  • "Which Python DataFrame library reads Delta / Iceberg natively?" — required answer: Polars via scan_delta / scan_iceberg; DuckDB via the Delta / Iceberg extensions; both use delta-rs / PyIceberg under the hood.
  • "Can I run Iceberg reads in Lambda?" — required answer: yes; PyIceberg + pyarrow fit in a Lambda layer; cold-start 60-300 ms.
  • "How do I develop against Iceberg locally without AWS?" — required answer: PyIceberg SQL catalog + MinIO / S3-Local + LocalStack Glue.
  • "When does Spark still win?" — required answer: > ~100 GB per job, complex joins across many tables, or Databricks-native pipelines.

Worked example — Polars lazy pipeline reading Delta and Iceberg together

Detailed explanation. A common analytics workload joins a Delta fact table (streaming source) with an Iceberg dimension table (batch source). Polars' lazy engine lets you plan the join across both formats and only materialise the final result. Walk through the pipeline.

  • Fact. Delta table s3://acme-lake/orders (updated every minute by a streaming ingester).
  • Dim. Iceberg table data_lake.customers (updated nightly).
  • Query. Last day's orders enriched with customer country + segment.

Question. Write a Polars lazy pipeline that scans both tables, joins them, aggregates by country + segment, and materialises to a small result.

Input.

Parameter Value
Fact Delta, s3://acme-lake/orders
Dim Iceberg, data_lake.customers
Join key customer_id
Filter order_date = yesterday
Aggregation SUM total_cents by country, segment

Code.

# Polars lazy pipeline over Delta + Iceberg
import polars as pl
from pyiceberg.catalog import load_catalog
from datetime import date, timedelta

catalog       = load_catalog("glue", **{"type": "glue", "s3.region": "us-east-1"})
customers_tbl = catalog.load_table("data_lake.customers")

yesterday = (date.today() - timedelta(days=1)).isoformat()

# 1. Lazy Delta scan with predicate push-down
orders_lf = pl.scan_delta("s3://acme-lake/orders").filter(
    pl.col("order_date") == yesterday
).select(["order_id", "customer_id", "total_cents"])

# 2. Lazy Iceberg scan — full customer dim
customers_lf = pl.scan_iceberg(customers_tbl).select(
    ["customer_id", "country", "segment"]
)

# 3. Lazy join + group-by
result_lf = (
    orders_lf.join(customers_lf, on="customer_id", how="inner")
             .group_by(["country", "segment"])
             .agg([
                 pl.col("total_cents").sum().alias("revenue_cents"),
                 pl.count().alias("order_count"),
             ])
             .sort("revenue_cents", descending=True)
)

# 4. Materialise — Polars pushes filters + columns back into delta-rs and pyiceberg
result = result_lf.collect(streaming=True)
print(result.head(20))
Enter fullscreen mode Exit fullscreen mode

Step-by-step explanation.

  1. pl.scan_delta("s3://...") returns a LazyFrame — no I/O yet. The .filter() and .select() chain builds a Polars plan node; when the plan collects, Polars translates them into delta-rs' predicate + column push-down.
  2. pl.scan_iceberg(customers_tbl) does the same for the Iceberg dimension. PyIceberg's manifest-level pruning kicks in as soon as filters are pushed — for the dimension we don't filter, so the whole table is planned, but only the customer_id, country, segment columns are read.
  3. The .join(customers_lf, on="customer_id", how="inner") is planned lazily. Polars' plan optimiser sees that only three columns from customers are needed and only three columns from orders are needed, and it pushes both projections back to the respective scans.
  4. .group_by([...]).agg([...]) and .sort() add plan nodes. The whole DAG is optimised holistically — filter push-down, projection push-down, and (in newer Polars) predicate pushdown across joins.
  5. .collect(streaming=True) tells Polars to execute the plan with a streaming engine that spills intermediates to disk. The fact-table scan streams through Polars' hash-join builder; the small dim is loaded fully. Total time for a moderate day's orders: 1-4 seconds on a laptop, no cluster, no Spark.

Output.

shape: (18, 4)
┌─────────┬──────────┬───────────────┬─────────────┐
│ country ┆ segment  ┆ revenue_cents ┆ order_count │
├─────────┼──────────┼───────────────┼─────────────┤
│ US      ┆ premium  ┆ 8,215,400     ┆ 4,120       │
│ US      ┆ standard ┆ 7,801,200     ┆ 6,340       │
│ DE      ┆ premium  ┆ 3,201,850     ┆ 1,910       │
│ UK      ┆ standard ┆ 2,981,050     ┆ 2,110       │
│ …       ┆ …        ┆ …             ┆ …           │
└─────────┴──────────┴───────────────┴─────────────┘
Enter fullscreen mode Exit fullscreen mode

Rule of thumb. For any Python-first analytics that joins across table formats, use Polars' scan_delta + scan_iceberg and rely on the lazy plan optimiser to push filters + projections back into the readers. .collect(streaming=True) handles datasets larger than RAM; anything you can express in Polars runs in one process without Spark.

Worked example — Lambda pipeline with delta-rs + zero JVM

Detailed explanation. The archetypal Rust-native serverless pipeline: a Lambda triggered by S3 PutObject reads a Parquet file, joins to a Delta lookup table, and appends the enriched rows to another Delta table — all in a 256 MB Lambda tier at 60-180 ms cold-start. Walk through the deployment.

  • Trigger. S3 PutObject on s3://acme-events/raw/.
  • Lookup. s3://acme-lake/customer_lookup (Delta).
  • Sink. s3://acme-lake/enriched_events (Delta, partitioned by event_date).
  • Package. deltalake + pyarrow + polars in a Lambda layer.

Question. Write the Lambda handler and the deployment package spec.

Input.

Component Value
Runtime Python 3.12
Memory 256 MB
Layer deltalake + pyarrow + polars (~180 MB unzipped)
Timeout 30 s

Code.

# lambda_handler.py — no JVM, no Spark, no cluster
from deltalake import DeltaTable, write_deltalake
import pyarrow.parquet as pq
import polars as pl
import boto3, urllib.parse

s3 = boto3.client("s3")

LOOKUP_TBL = DeltaTable("s3://acme-lake/customer_lookup")  # module-scope; reused across warm invocations
SINK       = "s3://acme-lake/enriched_events"

# Materialise the lookup once per warm container
_lookup_df = pl.from_arrow(LOOKUP_TBL.to_pyarrow_table())

def handler(event, context):
    # 1. Read the newly-uploaded Parquet
    rec       = event["Records"][0]["s3"]
    bucket    = rec["bucket"]["name"]
    key       = urllib.parse.unquote_plus(rec["object"]["key"])
    obj       = s3.get_object(Bucket=bucket, Key=key)
    events_arrow = pq.read_table(obj["Body"])
    events_df    = pl.from_arrow(events_arrow)

    # 2. Enrich via Polars join with the cached lookup
    enriched = events_df.join(_lookup_df, on="customer_id", how="left")

    # 3. Append to the sink Delta table
    write_deltalake(
        SINK,
        enriched.to_arrow(),
        mode          = "append",
        partition_by  = ["event_date"],
    )

    return {"rows": enriched.height, "cold": context.get_remaining_time_in_millis()}
Enter fullscreen mode Exit fullscreen mode 
# serverless.yml (illustrative)
service: enrich-events

provider:
  name: aws
  runtime: python3.12
  region: us-east-1
  memorySize: 256
  timeout: 30

functions:
  enrich:
    handler: lambda_handler.handler
    layers:
      - {Ref: DeltaRustLayer}
    events:
      - s3:
          bucket: acme-events
          event: s3:ObjectCreated:*
          rules:
            - prefix: raw/
            - suffix: .parquet

layers:
  DeltaRustLayer:
    path: layers/delta-rust
    compatibleRuntimes: [python3.12]
Enter fullscreen mode Exit fullscreen mode

Step-by-step explanation.

  1. The Lambda layer bundles deltalake, pyarrow, and polars (~180 MB uncompressed, within the 250 MB Lambda layer limit). No JVM, no Spark, no fat jars. pip install --target layers/delta-rust/python builds the layer contents.
  2. LOOKUP_TBL = DeltaTable(...) runs at module scope — this happens once per warm container, not per invocation. On cold start, the transaction-log parse is 40-100 ms; on warm invocations, the table object is already loaded.
  3. _lookup_df = pl.from_arrow(LOOKUP_TBL.to_pyarrow_table()) materialises the small lookup once (also at module scope). This is the "cache dim in memory" pattern — the lookup fits in RAM, so each invocation joins against the in-memory Polars frame rather than re-reading Delta.
  4. Per invocation: pq.read_table(obj["Body"]) reads the triggering Parquet from S3 into Arrow (50-200 ms depending on size). events_df.join(_lookup_df, ...) runs in Polars in-memory (10-100 ms). write_deltalake(...) appends to the sink Delta table (150-400 ms including one transaction-log commit).
  5. Total warm-invocation latency: 200-700 ms. Cold-start: +60-180 ms for module init. Memory: 100-200 MB. Cost per invocation: ~$0.0006 at 256 MB × 700 ms. Compared to the equivalent PySpark-on-EMR pipeline: 5-15 s per invocation, 900 MB memory, $0.02-0.05 per invocation.

Output.

Metric Value
Cold-start 60-180 ms
Warm invocation 200-700 ms
Memory used 100-200 MB
Package size 180 MB (layer, uncompressed)
Cost per invocation ~$0.0006
JVM cold-start 0 (no JVM)

Rule of thumb. For any Lambda that touches a Delta or Iceberg table, cache module-scope references to DeltaTable / Catalog / small dim frames; keep the layer at ≤ 200 MB uncompressed; and stay in the 128-256 MB memory tier unless the join size forces otherwise. The JVM cold-start you're avoiding is the whole point.

Worked example — local dev with MinIO + PyIceberg SQL catalog

Detailed explanation. The single biggest velocity improvement Rust-native readers unlock is offline local dev — a laptop can host MinIO (S3 API), a PyIceberg SQL catalog (SQLite for metadata), and iterate on the actual table format without a cloud dependency. Walk through the local stack.

  • Storage. MinIO container on port 9000, exposing an S3 API at http://localhost:9000.
  • Catalog. PyIceberg SQL catalog with sqlite:///./catalog.db for metadata.
  • Warehouse. s3://warehouse/ on MinIO.
  • Test. Create table, write, read, time-travel — all offline.

Question. Write the docker-compose and the PyIceberg local-dev script.

Input.

Component Value
MinIO endpoint http://localhost:9000
Catalog PyIceberg SQL with SQLite
Warehouse s3://warehouse/ on MinIO
Table test.orders

Code.

# docker-compose.yml — local MinIO
services:
  minio:
    image: minio/minio:latest
    ports:
      - "9000:9000"
      - "9001:9001"
    environment:
      MINIO_ROOT_USER:     admin
      MINIO_ROOT_PASSWORD: adminadmin
    command: server /data --console-address ":9001"
Enter fullscreen mode Exit fullscreen mode 
# local_iceberg.py — end-to-end local Iceberg dev
from pyiceberg.catalog.sql import SqlCatalog
from pyiceberg.schema import Schema
from pyiceberg.types import NestedField, IntegerType, StringType, TimestampType
import pyarrow as pa

# 1. SQL catalog with SQLite metadata + MinIO warehouse
catalog = SqlCatalog(
    "test",
    **{
        "uri":       "sqlite:///./catalog.db",
        "warehouse": "s3://warehouse/",
        "s3.endpoint":          "http://localhost:9000",
        "s3.access-key-id":     "admin",
        "s3.secret-access-key": "adminadmin",
    },
)

# 2. Create namespace + table
catalog.create_namespace_if_not_exists("test")

schema = Schema(
    NestedField(1, "order_id",    IntegerType(), required=True),
    NestedField(2, "customer_id", IntegerType(), required=True),
    NestedField(3, "amount",      IntegerType(), required=True),
    NestedField(4, "created_at",  TimestampType(), required=True),
)
if ("test", "orders") not in catalog.list_tables("test"):
    catalog.create_table("test.orders", schema=schema)

table = catalog.load_table("test.orders")

# 3. Append some rows
batch = pa.Table.from_pylist([
    {"order_id": 1, "customer_id": 10, "amount": 100, "created_at": pa.scalar(1720000000, type=pa.timestamp("us"))},
    {"order_id": 2, "customer_id": 11, "amount": 250, "created_at": pa.scalar(1720000100, type=pa.timestamp("us"))},
])
table.append(batch)

# 4. Read back with predicate push-down
result = table.scan(row_filter="amount > 100").to_arrow()
print(result)

# 5. Time-travel to previous snapshot
snaps = table.snapshots()
print("Snapshots:", [(s.snapshot_id, s.timestamp_ms) for s in snaps])
Enter fullscreen mode Exit fullscreen mode

Step-by-step explanation.

  1. SqlCatalog("test", uri="sqlite:///./catalog.db", warehouse="s3://warehouse/", s3.endpoint="http://localhost:9000", ...) wires PyIceberg to a SQLite-backed catalog (metadata pointer storage) and a MinIO-backed warehouse (data + metadata files). No AWS credentials or Glue setup required.
  2. catalog.create_namespace_if_not_exists(...) + catalog.create_table(...) is the DDL surface. The SQLite catalog stores (namespace, table_name) → metadata_location; the metadata JSON lives in MinIO alongside the data.
  3. table.append(batch) writes exactly like the AWS-side flow: create Parquet data files, manifest, manifest list, metadata JSON, update the pointer in SQLite. The MinIO S3 emulation handles the byte-level compatibility.
  4. .scan(row_filter="amount > 100").to_arrow() runs the full PyIceberg scanner locally: predicate push-down against the manifest stats, file-level pruning, Arrow read. No cloud round-trip.
  5. table.snapshots() lists the snapshot history exactly as it would in production. Time-travel via .scan(snapshot_id=...) works identically. The local stack is byte-for-byte compatible with the AWS deployment — you can test schema evolution, MERGE (when it lands), and multi-writer scenarios entirely offline.

Output.

pyarrow.Table
order_id:    int32
customer_id: int32
amount:      int32
created_at:  timestamp[us]
----
order_id:    [[2]]
customer_id: [[11]]
amount:      [[250]]
created_at:  [[2024-07-03 09:48:20.000000]]

Snapshots: [(6238729154213546789, 1720000100000)]
Enter fullscreen mode Exit fullscreen mode

Rule of thumb. For any Iceberg (or Delta) development, run MinIO + SQL catalog locally and iterate against the actual format. The test-loop speed gain over "deploy-to-dev-AWS-and-see" is 10-100x, and the byte-level compatibility means anything that works locally works in production.

Senior interview question on interop

A senior interviewer might ask: "Design a Python-first analytics platform for a team of 50 data scientists who need to query Delta tables (from a Databricks pipeline) and Iceberg tables (from a Flink pipeline) from Jupyter notebooks, ad-hoc DuckDB SQL, and occasional Ray Data jobs. The platform must run in the cost tier of a single Kubernetes cluster (no Databricks / Trino cluster budgeting) with the option to escalate to Spark for the rare > 100 GB job. Walk me through the stack, the reader choices per workload, and the failure modes you'd guard against."

Solution Using Polars + DuckDB + Ray Data on delta-rs and PyIceberg, with a shared Kubernetes cluster and Spark-on-Kubernetes escalation 

# 1. Notebook base image — every notebook ships with delta-rs + PyIceberg + Polars + DuckDB
# Dockerfile
FROM python:3.12-slim
RUN pip install --no-cache-dir \
    deltalake==0.15.* \
    pyiceberg[glue]==0.7.* \
    polars==1.0.* \
    duckdb==1.0.* \
    ray[data]==2.30.* \
    jupyterlab
Enter fullscreen mode Exit fullscreen mode 
# 2. Reader helper — hides catalog details behind a single API
from deltalake import DeltaTable
from pyiceberg.catalog import load_catalog
import polars as pl

_iceberg_catalog = load_catalog("glue", **{"type": "glue", "s3.region": "us-east-1"})

def read_table(uri: str, *, format: str = "auto") -> pl.LazyFrame:
    """Return a Polars LazyFrame for a Delta or Iceberg table URI."""
    if format == "auto":
        format = "iceberg" if uri.startswith("iceberg:") else "delta"

    if format == "delta":
        # e.g. read_table("s3://acme-lake/orders")
        return pl.scan_delta(uri)
    elif format == "iceberg":
        # e.g. read_table("iceberg:data_lake.orders")
        table_name = uri.removeprefix("iceberg:")
        iceberg_table = _iceberg_catalog.load_table(table_name)
        return pl.scan_iceberg(iceberg_table)
    else:
        raise ValueError(f"unknown format {format!r}")
Enter fullscreen mode Exit fullscreen mode 
# 3. Ray Data escalation — for parallel scans across many partitions
import ray
from deltalake import DeltaTable

def parallel_delta_scan(delta_path: str, partition_col: str, values: list) -> ray.data.Dataset:
    """Fan-out a Delta scan across Ray workers, one per partition value."""
    ray.init(address="ray://ray-head:10001")

    @ray.remote
    def scan_partition(v):
        import pyarrow.compute as pc
        dt = DeltaTable(delta_path)
        return (dt.to_pyarrow_dataset()
                  .filter(pc.field(partition_col) == v)
                  .to_table())

    tables = ray.get([scan_partition.remote(v) for v in values])
    return ray.data.from_arrow(tables)
Enter fullscreen mode Exit fullscreen mode 
# 4. Spark escalation — for > 100 GB batch (rare)
# spark-on-kubernetes submit
spark-submit --master k8s://... \
             --conf spark.executor.instances=20 \
             --conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension \
             big_batch_job.py
Enter fullscreen mode Exit fullscreen mode

Step-by-step trace.

Layer Component Purpose
Notebook deltalake + pyiceberg + polars + duckdb in one image uniform reader stack
Reader helper read_table(uri) -> LazyFrame hides Delta / Iceberg / catalog details
SQL DuckDB with iceberg + delta extensions ad-hoc SQL over both formats
Parallelism Ray Data fan-out per partition scale beyond one notebook
Escalation Spark-on-Kubernetes for > 100 GB rare path; same tables
Local dev MinIO + SQL catalog offline iteration

After deployment, notebook users open Delta and Iceberg tables via read_table("s3://...") or read_table("iceberg:data_lake.orders"), both returning Polars lazy frames; DuckDB SQL queries hit the same tables via extensions; Ray Data fan-out handles jobs up to ~50 GB; the ~2% of jobs exceeding 100 GB submit to Spark-on-Kubernetes for the same session's notebook to pick up.

Output:

Metric Value
Notebook cold-start (Delta + Iceberg + Polars ready) ~1.5 s
Typical read latency (< 10 GB slice) 300 ms - 3 s
Ray Data fan-out (10-50 GB) 20-90 s
Spark escalation (> 100 GB) 5-30 min
Kubernetes cluster monthly cost $2500 (shared)
Comparison: dedicated Databricks + Trino ~$18k/month
JVM cold-start (steady state) 0 (all Rust)

Why this works — concept by concept:

  • Uniform Rust-native reader stack — one Docker image contains deltalake, pyiceberg, polars, duckdb, and ray; all reads route through Rust-backed libraries; JVM is only ever launched for the rare Spark escalation.
  • Reader helper as the format abstraction — read_table(uri) hides the Delta-vs-Iceberg-vs-catalog decision behind a single call. Notebook users work in Polars LazyFrames regardless of source format; changing the source table format later is a one-line change.
  • DuckDB extensions for SQL — ad-hoc SQL over both Delta and Iceberg via INSTALL delta; INSTALL iceberg; puts a familiar analytics interface on top of the same Rust reader stack. The Arrow interchange keeps it zero-copy.
  • Ray Data fan-out — for the 10-50 GB slice that outgrows one notebook, parallel_delta_scan distributes per-partition scans across Ray workers. The same delta-rs library runs in each worker; no reader swap needed.
  • Spark-on-Kubernetes escalation — the > 100 GB path stays on Spark because Catalyst + multi-node execution earn their keep there. Sharing the Kubernetes cluster means no dedicated Spark cluster budget; jobs run only when needed.
  • Cost — one shared Kubernetes cluster (~$2500/month) serves 50 data scientists; comparison to a dedicated Databricks + Trino deployment (~$18k/month) is a 7x savings. Reader choice per workload keeps latency interactive at the low end and escalates only when justified. O(RAM) per notebook; O(cluster-nodes) for the rare escalation.

SQL
Topic — sql
SQL practice on cross-format lakehouse queries

Practice →

Design Topic — design Design problems on Python-first analytics platforms

Practice →

5. Rust-native vs Spark decision + interview signals

Pick delta-rs / PyIceberg for < 100 GB Python-first workloads; keep Spark for multi-TB batch, complex joins, and Databricks lock-in

The mental model in one line: delta-rs and PyIceberg are the default Delta / Iceberg readers for anything under ~100 GB per job, serverless invocation models (Lambda, Cloud Run, Cloud Functions), notebook-interactive analytics, and small streaming consumers — but Spark still owns multi-TB batch jobs, complex joins across many tables where the Catalyst optimiser earns its keep, and Databricks-native pipelines where ecosystem lock-in dominates the technical trade-off — and the senior architect's job is to name the workload thresholds that flip the decision rather than dogmatically picking one tool for everything. Every senior Python-first data platform in 2026 is a mesh of both.

Iconographic decision matrix diagram — a 2-column comparison card with Rust-native vs Spark columns rated on data-size / concurrency / catalog / ecosystem axes, and a 4-question decision-tree strip at the bottom.

The four decision axes.

  • Data size per job. < 100 GB → Rust-native comfortably; 100 GB - 1 TB → Rust-native possible with Ray Data fan-out or streaming Polars; > 1 TB → Spark's shuffle + Catalyst optimiser earn their keep. The threshold is fuzzy but the direction is unambiguous.
  • Concurrency profile. Serverless / notebook / small service (one-writer-one-reader) → Rust-native; multi-cluster concurrent writes with high MERGE fan-out → Spark or Trino (for their optimistic-concurrency runtime + retry story at scale).
  • Catalog integration. Iceberg REST / Glue / Nessie / Polaris → PyIceberg is the reference client; Databricks-native + Unity Catalog → Spark is the reference client. Cross-catalog federation (Trino / Presto) → Trino, not Rust-native.
  • Ecosystem lock-in. Databricks-workspace pipelines → Spark (operational cost of running non-Spark inside Databricks exceeds the JVM tax it would save); Snowflake-adjacent pipelines → often PyIceberg via Snowflake Open Catalog (Polaris).

The rust table formats cost model.

  • Cold-start. Rust: 30-200 ms. JVM: 2-8 s. Difference: 1-2 orders of magnitude.
  • Warm memory. Rust: 80-200 MB. Spark: 700 MB - 2 GB. Difference: 4-10x.
  • Per-invocation cost. For a Lambda that reads a table slice, Rust-native is 10-50x cheaper than PySpark-in-Lambda; for a multi-TB batch job, Spark is 2-5x cheaper than Rust-native (because Rust-native scales less well beyond one node).
  • Break-even. Empirically, ~80-120 GB per job is where per-node throughput advantages of Spark start dominating; below that, single-process Rust-native wins on total wall time + cost.

When to still reach for Spark — the honest list.

  • Multi-TB batch. A nightly job that scans 5 TB of Iceberg and joins across 20 dimension tables. Spark's Catalyst optimiser + broadcast + shuffle machinery are unmatched here.
  • Complex joins across many tables. Anything with 6+ tables in a single query where join reordering and cost-based optimisation matter. Polars' optimiser is impressive but not on Catalyst's level for these.
  • Databricks-native pipelines. If the pipeline ships as a Databricks job in a Databricks workspace, use Spark. Operational cost dominates.
  • Streaming with structured-streaming semantics. Spark Structured Streaming's exactly-once + stateful aggregations at high volume. Rust-native alternatives exist (Redpanda + Materialize, custom Rust services) but they're not "just replace Spark."
  • Advanced Delta features not yet in delta-rs. Deletion vectors + full DML MERGE with complex predicates + Change Data Feed reads at scale. Improving quarter over quarter, but Spark still leads on the bleeding edge.

Interview signals — what separates fluent from stumbling.

  • Names workload thresholds numerically. "< 100 GB", "60-180 ms cold-start", "256 MB Lambda tier" — quantified thresholds are senior signal.
  • Names both tools' sweet spots. Not "delta-rs is better" nor "Spark is better" — "delta-rs for X, Spark for Y."
  • Describes the same tables read by both. The insight that Delta / Iceberg are contracts, not engines, is what unlocks the "engine per workload" architecture.
  • Names catalog choice as orthogonal. Iceberg with Glue can be read by Spark or PyIceberg; Delta with Unity Catalog can be read by Spark or delta-rs (with UC support in newer releases).
  • Points to serverless as the killer app. Lambda + Cloud Functions + Cloud Run pipelines were operationally untenable with the JVM; Rust-native reader stacks are what made them viable.

Worked example — the workload-threshold cost model

Detailed explanation. The single most useful artifact for a "Rust-native vs Spark" interview is a numeric cost model calibrated to your workload. Walk through building one for three canonical job sizes: 5 GB, 50 GB, 500 GB.

  • Job A. 5 GB read + light transform + append (typical Lambda job).
  • Job B. 50 GB read + 3-way join + rollup (typical Ray Data job).
  • Job C. 500 GB read + 10-way join + wide rollup (typical Spark batch).

Question. Compute wall time + memory + cost for each job on both Rust-native and Spark; identify the break-even.

Input.

Job Data Reader options
A (5 GB) 40 partitions × 128 MB Lambda 256 MB + deltalake OR PySpark on EMR
B (50 GB) 400 partitions × 128 MB Fargate 4 vCPU + Polars + Ray OR PySpark on EMR
C (500 GB) 4000 partitions × 128 MB Ray cluster + deltalake OR PySpark on EMR

Code.

# Cost model — rough per-job estimates
def cost_rust_native(gb: float, mem_mb: float, wall_sec: float, node_cost_per_hr: float) -> float:
    return (wall_sec / 3600.0) * node_cost_per_hr

def cost_spark(gb: float, executor_count: int, executor_cost_per_hr: float, wall_sec: float) -> float:
    # Include ~15% coordinator overhead
    return (wall_sec / 3600.0) * executor_count * executor_cost_per_hr * 1.15

# Job A — 5 GB in a Lambda
print("A rust-native:", cost_rust_native(5, 256, 3.0, 0.20))          # ~$0.00017
print("A spark:      ", cost_spark(5, 2, 0.60, 60.0))                 # ~$0.023 (EMR + JVM boot)

# Job B — 50 GB Fargate + Ray
print("B rust-native:", cost_rust_native(50, 8000, 60.0, 1.20))       # ~$0.020
print("B spark:      ", cost_spark(50, 4, 0.60, 60.0))                # ~$0.046

# Job C — 500 GB
print("C rust-native:", cost_rust_native(500, 32000, 900.0, 6.00))    # ~$1.50 (Ray cluster)
print("C spark:      ", cost_spark(500, 10, 0.60, 300.0),)            # ~$0.575 (Spark faster on multi-TB)
Enter fullscreen mode Exit fullscreen mode

Step-by-step explanation.

  1. Job A (5 GB) on Rust-native fits in a 256 MB Lambda at 3 s wall time; per-invocation cost ~$0.00017. Same job on Spark needs EMR (or a container Lambda with Spark preloaded) at 60 s wall time and ~$0.023 per invocation. Rust-native is ~130x cheaper here.
  2. Job B (50 GB) on Rust-native (Fargate 4 vCPU + Polars + Ray fan-out) takes ~60 s at ~$0.020. Same job on Spark (4 executor EMR) takes ~60 s at ~$0.046. Rust-native still wins on cost, tied on wall time. This is the "middle" zone where either could work.
  3. Job C (500 GB) on Rust-native pushes into a Ray cluster (multi-node) with ~900 s wall time at ~$1.50. Same job on Spark (10 executors) is ~300 s at ~$0.575. Spark starts winning around 100-200 GB where multi-node scale-out dominates.
  4. The break-even is where per-node scale-out efficiency crosses. Empirically it lands at 80-120 GB per job for typical 3-5 vCPU nodes. Below → Rust-native. Above → Spark.
  5. The cost model assumes commodity pricing; your numbers will differ. The shape of the curve — Rust-native dominant at small sizes, Spark dominant at large — is the transferable insight.

Output.

Job Rust-native (wall/cost) Spark (wall/cost) Winner
A — 5 GB 3 s / $0.00017 60 s / $0.023 Rust-native (135x)
B — 50 GB 60 s / $0.020 60 s / $0.046 Rust-native (2.3x)
C — 500 GB 900 s / $1.50 300 s / $0.575 Spark (2.6x)
Break-even ~100 GB per job

Rule of thumb. The 100 GB per-job threshold is a good default; recalibrate against your actual node prices + data patterns. Below the threshold, run everything on Rust-native. Above the threshold, escalate to Spark. In the transition zone (50-200 GB), let workload characteristics — join complexity, partition fan-out, ecosystem — break the tie.

Worked example — interview probe on catalog choice + reader

Detailed explanation. A common senior interview question conflates catalog choice with reader choice. Walk through separating them cleanly for three scenarios.

  • Scenario 1. Iceberg on Glue, read from a Lambda. Catalog: Glue. Reader: PyIceberg.
  • Scenario 2. Delta on Unity Catalog, read from Databricks. Catalog: Unity Catalog. Reader: Spark (ecosystem).
  • Scenario 3. Iceberg on Polaris (REST), read from Trino for ad-hoc + Lambda for enrichment. Catalog: Polaris. Readers: Trino for interactive; PyIceberg for enrichment.

Question. For each scenario, name the catalog, the reader, and justify why they're orthogonal choices.

Input.

Scenario Format Catalog Reader
Iceberg on Glue + Lambda Iceberg Glue PyIceberg
Delta on UC + Databricks Delta Unity Catalog Spark
Iceberg on Polaris + mixed Iceberg Polaris (REST) Trino + PyIceberg

Code.

# Scenario 1 — Iceberg + Glue + PyIceberg (Lambda)
from pyiceberg.catalog import load_catalog
c1 = load_catalog("glue", **{"type": "glue", "s3.region": "us-east-1"})
t1 = c1.load_table("data_lake.orders")

# Scenario 2 — Delta + Unity Catalog + Spark (Databricks notebook)
# %sql
# SELECT * FROM main.data_lake.orders WHERE order_date = current_date()

# Scenario 3 — Iceberg + Polaris + PyIceberg (Lambda) OR Trino (interactive)
c3 = load_catalog(
    "polaris",
    **{
        "type":       "rest",
        "uri":        "https://polaris.acme.internal/api/catalog",
        "credential": "client_id:client_secret",
        "scope":      "PRINCIPAL_ROLE:ALL",
    },
)
t3 = c3.load_table("warehouse.orders")
Enter fullscreen mode Exit fullscreen mode

Step-by-step explanation.

  1. Scenario 1: Iceberg on Glue. The catalog is Glue (an AWS-native metadata store); the reader is PyIceberg (a Python client). They're orthogonal — you could also read the same Glue-catalogued table from Spark, Trino, or Snowflake without changing anything about the catalog.
  2. Scenario 2: Delta on Unity Catalog. UC is Databricks-native but open-source; Spark (with the UC connector) is the reference reader. delta-rs is adding UC support through 2026 but Spark is the operationally safest choice inside a Databricks workspace because of ecosystem integration.
  3. Scenario 3: Iceberg on Polaris (Snowflake's open catalog, ASF-hosted). Polaris speaks the Iceberg REST Catalog Spec, which PyIceberg speaks natively. Trino also speaks Iceberg REST via its Iceberg connector. Two different readers, one shared catalog, one shared table format.
  4. The pattern in all three cases: catalog choice is driven by the organisational / ecosystem factors (AWS-native, Databricks-native, open-lakehouse-native); reader choice is driven by the workload factors (invocation model, data size, latency).
  5. Senior interviewers probe this to see whether you can separate the two. Weak candidates conflate them ("we'd use Databricks" — that's an ecosystem statement, not a catalog choice); senior candidates name catalog + reader independently and justify each.

Output.

Scenario Catalog choice reason Reader choice reason
Iceberg + Glue + Lambda AWS-native metadata store PyIceberg for serverless Python
Delta + UC + Databricks Databricks-workspace integration Spark for ecosystem lock-in
Iceberg + Polaris + mixed open protocol, ASF-hosted PyIceberg for Lambda; Trino for interactive SQL

Rule of thumb. Never let a single interview answer conflate catalog and reader. Name each explicitly. Catalog answers organisational questions (where does metadata live? who governs it?); reader answers workload questions (how big is the job? what's the invocation model?). The two are orthogonal by design — that's the whole point of the Iceberg REST spec and the Delta transaction-log protocol.

Worked example — when Rust-native fails and you must escalate

Detailed explanation. Not every workload fits Rust-native. Walk through three concrete failure modes and the correct escalation path.

  • Failure 1. MERGE on Delta with complex UPDATE-SET expressions across 6 columns and a 20-condition WHEN MATCHED clause. delta-rs' MERGE is production-ready but the DSL doesn't cover this level of expression yet (as of 0.15). Escalate to Spark.
  • Failure 2. 8-way join across large fact + 7 dimensions with skewed keys. Polars can handle it but the plan optimisation and skew handling of Spark's Catalyst + adaptive query execution (AQE) win on wall time.
  • Failure 3. Delta Change Data Feed (CDF) read at scale — 100 GB of change events across a week. delta-rs CDF read exists but Spark's structured-streaming + CDF reader is more battle-tested.

Question. For each failure mode, define the trigger for escalation + the correct receiving stack.

Input.

Failure Trigger Escalate to
Complex MERGE > 4 UPDATE-SET columns OR nested predicates Spark on Databricks / EMR
Skewed multi-way join 5+ tables OR known skewed keys Spark with AQE
CDF read at scale > 10 GB CDF window per read Spark Structured Streaming + CDF

Code.

# Escalation trigger — code-level assertion in a Rust-native pipeline
def maybe_escalate(merge_columns: int,
                   join_tables: int,
                   cdf_gb_window: float) -> str | None:
    if merge_columns > 4:
        return "spark:merge-complexity"
    if join_tables >= 5:
        return "spark:multi-join"
    if cdf_gb_window > 10:
        return "spark:cdf-window"
    return None

# Usage in a pipeline
escalation = maybe_escalate(merge_columns=6, join_tables=2, cdf_gb_window=0)
if escalation:
    submit_to_spark_cluster(job_spec, reason=escalation)
else:
    run_on_deltalake(job_spec)
Enter fullscreen mode Exit fullscreen mode

Step-by-step explanation.

  1. The escalation trigger is quantified — a numeric threshold on MERGE column count, join arity, CDF window size. This turns the "should I use Spark?" question into a code-level assertion rather than a subjective judgement.
  2. When the trigger fires, the pipeline routes to a Spark cluster (Databricks job, Kubernetes Spark submit, EMR step). The receiving Spark job uses the same table format (Delta / Iceberg) — no data migration required.
  3. Complex MERGE (failure 1) escalates because delta-rs' MERGE builder DSL doesn't yet cover complex UPDATE-SET expressions with computed column values from multiple source columns. Rather than fight the DSL, ship to Spark's MERGE INTO ... USING ... SQL syntax.
  4. Multi-way skewed joins (failure 2) escalate because Polars' single-node optimiser cannot match Spark's Catalyst + AQE for skew-adaptive shuffle. If your join has 5+ tables or known skewed keys (one customer with 40% of orders), Spark's cost-based optimisation + skew join handling wins.
  5. Large CDF reads (failure 3) escalate because delta-rs CDF is newer and less battle-tested. For a > 10 GB window of change events, Spark Structured Streaming's CDF reader is the safer bet. This will change as delta-rs matures; keep re-evaluating quarterly.

Output.

Failure Rust-native attempt Escalated to Spark Time cost of escalation
Complex MERGE (6 cols, 20 conds) fails DSL support Databricks SQL MERGE 5-10 min extra (cluster boot)
8-way skewed join Polars runs but 15 min p99 Spark AQE, 3 min p99 net savings
100 GB CDF read delta-rs CDF, 25 min Structured Streaming CDF, 8 min net savings

Rule of thumb. Codify escalation triggers numerically — MERGE column count, join arity, CDF window size — and route to Spark when a threshold trips. The right architecture is Rust-native by default with named escalation criteria, not "always Rust" or "always Spark."

Senior interview question on Rust-native vs Spark decision

A senior interviewer might ask: "You're the tech lead for a 60-person data platform team spending $250k/year on Databricks + EMR. Leadership wants to cut cost by 40% without disrupting the dashboards or breaking any consumers. Walk me through an audit of workloads that would move from Spark to Rust-native (delta-rs / PyIceberg + Polars + Ray), the ones that must stay on Spark, the migration risk, and the monitoring that would catch any silent regression."

Solution Using a workload-audit + phased migration + Parquet-checksum canary + per-job cost dashboard 

# 1. Workload audit script — classifies each job by size + join arity
import boto3, json
from collections import defaultdict

emr = boto3.client("emr")
jobs = emr.list_steps(ClusterId="j-XXXXXXXX")["Steps"]

def classify(job) -> str:
    gb        = job["metrics"]["input_gb"]
    joins     = job["metrics"]["join_count"]
    merge_cx  = job["metrics"].get("merge_update_columns", 0)

    if gb < 20 and joins <= 3 and merge_cx <= 4:
        return "rust-native-easy"
    if gb < 100 and joins <= 4 and merge_cx <= 5:
        return "rust-native-with-ray"
    if gb > 500 or joins >= 6 or merge_cx > 6:
        return "keep-on-spark"
    return "candidate-migration"

buckets = defaultdict(list)
for j in jobs:
    buckets[classify(j)].append(j["Id"])

for cls, ids in buckets.items():
    print(cls, len(ids))
Enter fullscreen mode Exit fullscreen mode 
# 2. Phased migration — one job per week; keep Spark job as canary for 4 weeks
def canary_migration(job_id: str) -> None:
    rust_result  = run_rust_native(job_id)
    spark_result = run_spark(job_id)

    # Byte-level Parquet checksum comparison on output
    if not compare_parquet_files(rust_result.output_uri, spark_result.output_uri):
        alert_pagerduty(job_id, "checksum mismatch")
        return

    # After 4 weeks of clean canary runs, retire the Spark version
    schedule_retirement(job_id, after_days=28)
Enter fullscreen mode Exit fullscreen mode 
# 3. Per-job cost dashboard — daily rollup
import polars as pl

def cost_rollup(start_date: str, end_date: str) -> pl.DataFrame:
    dt = DeltaTable("s3://acme-ops/job_costs")
    frame = pl.from_arrow(
        dt.to_pyarrow_dataset()
          .filter(pc.field("day") >= start_date)
          .to_table()
    )
    return (
        frame.group_by(["day", "engine"])
             .agg([
                 pl.col("cost_usd").sum().alias("cost"),
                 pl.col("wall_sec").sum().alias("wall"),
                 pl.count().alias("jobs"),
             ])
             .sort("day")
    )
Enter fullscreen mode Exit fullscreen mode

Step-by-step trace.

Phase Action Owner Success signal
Audit classify all EMR + Databricks jobs by (gb, joins, merge complexity) platform lead list of migration candidates
Phase 1 (weeks 1-4) migrate rust-native-easy jobs; keep Spark as canary platform team checksum match
Phase 2 (weeks 5-12) migrate rust-native-with-ray jobs; Ray cluster on shared K8s platform team checksum match
Phase 3 (weeks 13-16) retire canary Spark jobs; downsize EMR fleet platform team cost drop visible
Phase 4 (ongoing) keep keep-on-spark on Spark; monitor for delta-rs feature maturity platform team quarterly re-evaluation
Monitoring Parquet-checksum canary + per-job cost dashboard on-call zero silent regressions

After the migration, ~65% of jobs land on Rust-native (Lambda / Fargate + delta-rs / PyIceberg + Polars); ~15% run on shared K8s Ray Data fan-out; ~20% stay on Spark; the per-job cost dashboard shows ~45% total cost reduction; the checksum canary caught 3 silent regressions over 6 months (all delta-rs bugs fixed in library upgrades).

Output:

Metric Before After
Monthly cost $20.8k $11.5k (45% cut)
Jobs on Spark 100% 20%
p99 job latency (small jobs) 5-15 min 1-3 min
Silent regressions caught (none checked) 3 / 6 months
Canary retirement — after 4 clean weeks
Team productivity baseline ~30% faster iteration cycle

Why this works — concept by concept:

  • Workload audit as the first step — classifying every job by (gb, joins, merge complexity) turns "should we migrate?" into a data-driven question. The three buckets (easy / with-ray / keep-on-spark) match the readers' sweet spots.
  • Phased migration with parallel canary — running the same job on both engines for 4 weeks catches silent write regressions before consumers notice. The Parquet-checksum comparison is cheap (SHA-256 of small samples) and definitive.
  • Per-job cost dashboard on Delta itself — storing per-job cost + wall + engine in a Delta table and querying via Polars closes the loop: the platform observes itself with the same tools it's promoting. This is the "eating your own dog food" senior signal.
  • Named escalation criteria — jobs classified as keep-on-spark stay on Spark by design, not by omission. The classification triggers (> 500 GB, 6+ joins, complex MERGE) are the escalation contract; jobs are re-evaluated quarterly as delta-rs matures.
  • Cost — 45% monthly compute cost cut ($20.8k → $11.5k) with zero silent regressions; iteration cycle time improved ~30% because small jobs run in seconds instead of minutes. Migration risk was bounded by the canary + checksum monitor. O(1) per-job cost model; O(portfolio) audit cadence.

Design
Topic — design
Design problems on lakehouse engine migration + cost tuning

Practice →

Aggregation
Topic — aggregation
Aggregation problems on cost / latency dashboards

Practice →

Cheat sheet — delta-rs & PyIceberg recipes

  • Which library when. deltalake (delta-rs Python wrapper) for any Delta Lake table where you don't have a Spark cluster handy — Lambda, notebook, Fargate, small ECS service, local dev. pyiceberg for any Apache Iceberg table with a Glue / REST / Hive / Nessie / Polaris / SQL catalog and a Python-first workload. Both are Rust-backed under the hood; both interoperate with Spark / Trino readers on the same underlying tables. Escalate to Spark when the job crosses ~100 GB, needs Catalyst-level join optimisation, or lives inside Databricks.
  • deltalake read one-liner. from deltalake import DeltaTable; DeltaTable("s3://bucket/table").to_pyarrow_dataset().filter(pc.field("day") == "2026-07-21").to_table(columns=["id", "amount"]) — one line covers table open + predicate + column push-down. Hand off to Polars via pl.from_arrow(...) or to DuckDB via duck.register(...).
  • deltalake write one-liner. write_deltalake("s3://bucket/table", arrow_table, mode="append", partition_by=["day"]) for append; mode="overwrite" + partition_filters=[...] for scoped overwrite. Schema evolution via schema_mode="merge". All in one Python process.
  • deltalake MERGE / UPSERT template. dt.merge(source, "target.id = source.id", "source", "target").when_matched_update(updates={"col": "source.col"}).when_not_matched_insert_all().execute() — production-ready in 0.15+. Returns per-transaction counts (num_target_rows_updated, num_target_rows_inserted) for monitoring. Escalate to Spark for complex UPDATE-SET with 5+ columns or nested predicates.
  • deltalake OPTIMIZE + VACUUM daily maintenance. dt.optimize.compact(partition_filters=[("day", "=", today)], target_size=256 * 1024 * 1024); dt.optimize.z_order(columns=["customer_id"]); dt.vacuum(retention_hours=168, enforce_retention_duration=True). Ship the maintenance job the day you ship writes; the small-file problem hits any append-heavy table within days.
  • PyIceberg catalog config templates. Glue: load_catalog("glue", type="glue", s3.region="us-east-1"). REST (Polaris / Tabular): load_catalog("rest", type="rest", uri="https://polaris.acme.internal/api/catalog", credential="id:secret", scope="PRINCIPAL_ROLE:ALL"). Hive: load_catalog("hive", type="hive", uri="thrift://hms:9083"). SQL local dev: load_catalog("sql", type="sql", uri="sqlite:///./cat.db", warehouse="s3://warehouse/", s3.endpoint="http://localhost:9000", s3.access-key-id="admin", s3.secret-access-key="adminadmin").
  • PyIceberg read + push-down template. catalog.load_table("db.table").scan(row_filter=And(EqualTo("day", "2026-07-21"), EqualTo("region", "US")), selected_fields=("id", "amount", "day")).to_arrow() — predicate + column push-down against partition + file-level stats. .to_polars(), .to_duckdb(), .to_pandas() for direct consumer handoff.
  • PyIceberg write. table.append(arrow_batch) for insert; table.overwrite(arrow_batch, overwrite_filter=EqualTo("day", "2026-07-21")) for scoped rewrite. table.update_schema() for schema evolution; table.update_spec() for partition evolution. All atomic at the catalog commit level.
  • Polars scan_delta + scan_iceberg pattern. pl.scan_delta("s3://...").filter(...).select([...]) and pl.scan_iceberg(iceberg_table).filter(...).select([...]) return LazyFrames. .collect(streaming=True) executes with filter + projection push-down + spill-to-disk for larger-than-RAM plans. The "one lazy plan across both formats" pattern is the Python-first analytics superpower.
  • DuckDB Iceberg + Delta extensions. INSTALL iceberg; LOAD iceberg; SELECT * FROM iceberg_scan('s3://.../metadata/00042.metadata.json') for direct-metadata Iceberg scan. INSTALL delta; LOAD delta; SELECT * FROM delta_scan('s3://bucket/table') for Delta. Zero-copy Arrow interchange means the scanner reads once and DuckDB executes on the same buffers.
  • Lambda deployment recipe. Package deltalake + pyarrow + polars in a Lambda layer (~180 MB uncompressed, under the 250 MB layer limit). Cache DeltaTable(...) / catalog.load_table(...) at module scope so warm invocations skip transaction-log parse. Run in the 256 MB tier for < 100 MB slices; escalate to 1 GB tier for larger. Cold-start 60-300 ms; warm 50-300 ms.
  • Local dev with MinIO + SQL catalog. docker-compose up minio on port 9000; SqlCatalog("test", uri="sqlite:///./cat.db", warehouse="s3://warehouse/", s3.endpoint="http://localhost:9000", ...). Byte-level compatible with AWS deployment. Iterate schema evolution + MERGE + partition changes offline; deploy the same code to production with an env-var-swapped catalog config.
  • Rust-native vs Spark decision matrix. < 20 GB → Rust-native (Lambda); 20-100 GB → Rust-native (Fargate + Polars streaming or Ray Data); 100-500 GB → Rust-native with Ray fan-out or Spark; > 500 GB → Spark. 6+ way joins with skewed keys → Spark (Catalyst + AQE). Databricks-native pipelines → Spark. Everything else → Rust-native by default with named escalation triggers.
  • Interoperability guarantees. Delta 3.x transaction log is a shared contract — a table written by deltalake is readable by Spark and vice versa. Iceberg REST Catalog Spec + Iceberg v2 spec are shared contracts — a table written by PyIceberg is readable by Spark / Trino / Snowflake. The reader is an implementation detail; the table format is the contract. Test cross-engine reads on any table you migrate.
  • Migration risk monitoring. Ship a Parquet-checksum canary the day you route the first job to Rust-native. Compare byte-for-byte outputs from Rust-native vs Spark on a 1% sample daily. Alert on drift. Retire the Spark canary after 4 consecutive weeks of clean matches. This is the only reliable way to catch silent write-behaviour regressions during a phased migration.
  • When to still reach for Spark. > 500 GB batch, 6+ way joins with skewed keys, complex MERGE (5+ UPDATE-SET columns or nested predicates), Databricks-native pipelines, structured streaming with exactly-once + stateful aggregations at high volume, or bleeding-edge Delta features (deletion vectors, CDF at scale) that delta-rs hasn't fully absorbed yet. Everything else defaults to Rust-native.
  • Cost model rule of thumb. Below ~100 GB per job, Rust-native is 3-100x cheaper than Spark on cost + wall time. Above ~200 GB, Spark is 2-5x cheaper because per-node scale-out efficiency dominates. The 100-200 GB "middle" zone is where you use workload characteristics (join complexity, invocation model, ecosystem) to break the tie. Recalibrate your break-even quarterly as delta-rs / PyIceberg mature.

Frequently asked questions

What is delta-rs in one sentence?

delta-rs is a Rust implementation of the Delta Lake protocol (transaction-log parsing, Parquet reading and writing, MERGE / UPSERT planning, OPTIMIZE and VACUUM maintenance, schema evolution, time-travel) shipped as the pip-installable deltalake Python package (and as a bare Rust crate for Rust services), which lets Python code read and write Delta Lake tables through Arrow with no JVM in the process and cold-start latencies measured in tens of milliseconds instead of the multi-second JVM boot Spark requires. The package has matured through 2024-2026 to the point where MERGE, OPTIMIZE (bin-packing compaction), Z-ORDER, VACUUM (soft-delete of stale files), schema evolution, and time-travel are all production-ready; the transaction log stays byte-compatible with Delta 3.x so Spark readers can read tables written by deltalake and vice versa. Every senior Python-first data engineer in 2026 has at least one Lambda or notebook job that used to need PySpark and now runs on deltalake in a few hundred milliseconds at a fraction of the cost.

delta-rs vs Spark for reading Delta tables — when do I pick each?

Default to delta-rs (via the deltalake Python package) for any Delta read where the job is under ~100 GB, the invocation model is serverless / notebook / small service, and the ecosystem is not Databricks-native. Cold-start is 30-200 ms vs Spark's 2-8 s JVM boot; warm memory is 80-200 MB vs Spark's 700 MB - 2 GB; per-invocation cost for a Lambda that reads a table slice is 10-50x cheaper than PySpark-in-Lambda; and the same transaction log means a Spark reader can consume the same table without reformatting. Pick Spark for multi-TB batch scans, 6+ way joins with skewed keys (Catalyst + AQE earn their keep here), complex MERGE with 5+ UPDATE-SET columns or nested predicates, Databricks-native pipelines (ecosystem lock-in wins), or bleeding-edge Delta features that delta-rs hasn't fully absorbed yet. Never pick Spark for a Lambda that reads one partition on each S3 event — the JVM cold-start alone breaks the p99 budget. The two engines read the same tables; picking per workload is the modern architecture.

What is PyIceberg and how does it differ from delta-rs?

PyIceberg is the Apache Iceberg project's Python client — pure Python for the surface API with Rust hot loops for Arrow decode and manifest parsing — that speaks the Iceberg catalog protocol (Glue, Hive Metastore, Iceberg REST, Nessie, Snowflake Polaris, and SQL for local dev), plans scans against Iceberg's snapshot + manifest metadata, executes them through PyArrow, and supports append + overwrite writes plus a maturing positional-delete story. The key structural difference from delta-rs is that Iceberg requires a catalog (there is no "just point at the folder" mode like Delta's transaction-log-only option), so PyIceberg's load_catalog(...) is the first call in any script. Reading is otherwise nearly identical in shape: catalog.load_table("db.table").scan(row_filter=..., selected_fields=...).to_arrow() mirrors DeltaTable(...).to_pyarrow_dataset().filter(...).to_table(...). Writes are ahead on Delta (delta-rs has production MERGE; PyIceberg still maturing on positional deletes), catalog federation is ahead on Iceberg (six real catalog options vs Delta's transaction-log + Unity Catalog). Both compose with Polars, DuckDB, Ray Data, and serverless Lambda through Arrow interchange.

Can I use delta-rs and PyIceberg in AWS Lambda?

Yes — this is one of the flagship use cases for both libraries. deltalake ships as a ~30 MB pip wheel, pyiceberg as a ~40 MB wheel, and both work with pyarrow (~90 MB) in a Lambda layer well under the 250 MB uncompressed layer limit. Cold-start in the 256 MB Lambda memory tier is 60-300 ms including import and first table open; warm invocations are 50-300 ms including scan planning. Compare this to PySpark in Lambda: 5-8 s JVM boot before any read, 700-900 MB memory floor pushing you into the 1 GB tier, and a fat-jar deployment that adds minutes to every function update. Cache DeltaTable(...) and catalog.load_table(...) at module scope so warm invocations skip transaction-log parse. For Iceberg, use the Glue catalog for AWS-native deployments (load_catalog("glue", type="glue", s3.region="us-east-1") picks up the ambient IAM role automatically). For Delta, point at the S3 path directly. Serverless Lakehouse pipelines that were operationally untenable with the JVM are now the sweet spot for Rust-native readers.

Do delta-rs and PyIceberg support MERGE / UPSERT?

Yes for delta-rs, production-ready in the 0.15+ line: dt.merge(source, "target.id = source.id", "source", "target").when_matched_update(updates={"col": "source.col"}).when_not_matched_insert_all().execute() is the canonical UPSERT surface. It returns per-transaction counts (num_target_rows_updated, num_target_rows_inserted, num_target_rows_scanned) that you should emit to monitoring to catch source-data drift. Complex MERGE with 5+ UPDATE-SET columns and nested predicates is the current escalation trigger to Spark — the delta-rs DSL is expanding quarterly but Spark's SQL MERGE INTO ... USING ... still covers more shape variance. For PyIceberg, the equivalent operation is positional deletes + append, which is maturing through 2026 releases. Today the production-ready write paths are .append(arrow_batch) and .overwrite(arrow_batch, overwrite_filter=EqualTo(...)) — the latter gives you scoped rewrites (partition-level or predicate-based) that cover the "upsert a whole partition" pattern. For row-level UPSERT on Iceberg, escalate to Spark or a MERGE-capable engine like Trino, or wait for PyIceberg's positional-delete API to fully land.

When should I still reach for Spark instead of delta-rs or PyIceberg?

Reach for Spark when any of five triggers fire. First, data size: > 500 GB per job (multi-TB batch is Spark's home turf); the 100-500 GB middle zone is negotiable based on other factors. Second, join complexity: 6+ way joins with skewed keys where Spark's Catalyst optimiser + adaptive query execution (AQE) beat Polars' single-node optimiser by wall time and cost. Third, complex MERGE: 5+ UPDATE-SET columns, nested predicates, or SCD-2 patterns that delta-rs' MERGE builder DSL doesn't yet cover cleanly; escalate to Spark's SQL MERGE. Fourth, ecosystem lock-in: Databricks-native pipelines where the operational cost of running a non-Spark reader inside a Databricks workspace exceeds the JVM tax it would save. Fifth, streaming with stateful semantics: Spark Structured Streaming's exactly-once + stateful aggregations at high volume — Rust-native alternatives exist but they're not "just replace Spark." For everything else — Lambda, notebook, small service, < 100 GB batch, Ray Data fan-out up to ~50 GB — default to Rust-native. Codify the triggers numerically in your platform so "should I use Spark?" becomes an assertion, not a subjective call.

Practice on PipeCode

  • Drill the SQL practice library → for the lakehouse-shaped SQL, upsert, and predicate-pushdown problems senior interviewers love.
  • Rehearse aggregation patterns with the aggregation practice library → for the group-by-heavy analytics workloads that live on Delta / Iceberg dashboards.
  • Sharpen join intuition with the joins practice library → for the multi-table join scenarios that separate Rust-native (Polars) from Spark (Catalyst) territory.
  • Stretch the design axis with the design practice library → for lakehouse-platform scenarios: reader-per-workload choices, JVM-vs-Rust cost models, and phased engine migrations.
  • Stack the prerequisites against PipeCode's broader 450+ data-engineering catalogue to anchor the Rust-vs-Spark decision matrix against real graded inputs.

Lock in delta-rs muscle memory

Docs explain APIs. PipeCode drills explain the decision — when `deltalake` fits in a 256 MB Lambda, when PyIceberg beats a Trino cluster for a notebook, when the JVM cold-start finally justifies escalating to Spark, when the same table is read by three engines without a migration. Pipecode.ai is Leetcode for Data Engineering — reader-first practice tuned for the production trade-offs senior data engineers actually face.

Practice SQL problems →
Practice design problems →

Top comments (0)