Polars Internals: Query Optimizer, Lazy Frame, Arrow2 & Why It's 10x Faster Than Pandas

polars internals are the reason a Rust DataFrame library written by one core maintainer is now the default answer, in most senior data-engineering interviews of 2026, to "what do you reach for when Pandas is too slow but Spark is too heavy" — and the reason a warm-cache aggregation over a 20 GB Parquet file that used to take Pandas fifteen minutes now returns in under thirty seconds on the same laptop. Every senior data engineer has watched a Pandas groupby().agg() spin a single core to 100% for hours while the other fifteen cores sit at zero and the machine's 64 GB of RAM slowly fills with duplicated column copies, then watched the same query in Polars saturate every core, keep memory flat, and finish in a fraction of the time. The engineering story is not "Polars uses Rust" — that undersells it by an order of magnitude — but that four independent internals decisions (lazy execution + a real query optimizer + Arrow2 columnar layout + Rayon-based native parallelism) compound to give a 10x – 100x speed-up on medium-size tabular workloads.

This guide is the senior-DE walkthrough you wished existed the first time an interviewer asked "walk me through the polars lazyframe execution model and name the optimizer passes", or "what does arrow2 give you that Pandas doesn't", or "when does polars vs pandas become 'Polars beats Spark on one machine'". It walks through the four load-bearing internals — the polars query optimizer (logical plan → predicate / projection / slice pushdown → physical plan → streaming execution), the Arrow2 columnar chunk layout with validity bitmaps and zero-copy interop, the Rayon work-stealing parallel engine plus the morsel-driven out-of-core streaming mode, and the decision axes (speed × data-size × lazy-semantics × distributed) that decide when Polars replaces Pandas, when it replaces DuckDB, and when Spark is still the right answer. Each section pairs a teaching block with a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the SQL practice library →, rehearse aggregation patterns on the aggregation practice library →, and sharpen the join axis with the joins practice library →.

---

On this page

• Why Polars internals matter in 2026
• LazyFrame + query optimizer
• Arrow2 columnar format + memory layout
• Native parallelism + streaming engine
• Polars vs Pandas / DuckDB / Spark + interview signals
• Cheat sheet — Polars internals recipes
• Frequently asked questions
• Practice on PipeCode

---

1. Why Polars internals matter in 2026

Four internals decisions compound into a 10x – 100x speed-up on the same laptop

The one-sentence invariant: Polars is not "faster Pandas" — it is a Rust-native DataFrame engine that composes four independent internals wins (lazy execution with a real query optimizer, Arrow2 columnar layout with validity bitmaps and zero-copy views, Rayon-based work-stealing parallelism across every core, and a morsel-driven streaming engine that spills to disk for datasets larger than RAM) so that a warm-cache pipeline over 100 MB – 100 GB of tabular data on one laptop finishes in a fraction of the wall-clock time Pandas would spend on a single core, without needing a distributed cluster the way Spark does. The tool you pick in year one becomes the tool your whole notebook / DAG / library stack hard-codes assumptions against — you either write df["col"].sum() (Pandas-style eager) forever or you write lf.select(pl.col("col").sum()).collect() (Polars-style lazy) forever — and the migration cost between them scales with the size of your codebase.

The four axes interviewers actually probe.

• Execution model — lazy vs eager. Pandas is eager: every df.groupby(...).agg(...) runs immediately, materialising intermediate DataFrames in memory. Polars offers both eager (pl.DataFrame) and lazy (pl.LazyFrame) modes, and the lazy mode is where the 10x wins live — the whole chain of .filter(), .group_by(), .agg() is captured as a logical plan, an optimizer rewrites it, and only .collect() triggers execution. Interviewers open here because it separates people who use Polars as "faster syntax" from those who understand the query engine.
• Optimizer passes. Pandas has no optimizer — the code runs top-to-bottom exactly as written, materialising every intermediate. Polars applies predicate pushdown (move filters down to the scan so unread rows never leave disk), projection pushdown (only read the columns the query actually needs), slice pushdown (limit rows before expensive operations), common-subexpression elimination (compute repeated expressions once), and join reordering (join smaller tables first). Interviewers probe here because it's the single most-cited difference between "call Polars from Pandas" and "actually use LazyFrames".
• Memory layout. Pandas uses NumPy arrays plus a lot of Python object overhead — a string column is a NumPy array of Python str objects, each of which is a full CPython PyObject with refcount and type header. Polars uses Arrow2 columnar chunks: contiguous, SIMD-friendly, with a separate validity bitmap for nulls, and zero-copy views into buffers other Arrow-compatible engines (DuckDB, DataFusion, PyArrow) can consume without a copy. The cache-friendliness alone is a 3x – 5x win on scan-heavy queries.
• Parallelism model. Pandas is single-core by default — the GIL blocks every heavy operation. Polars uses Rayon's work-stealing scheduler over Rust threads (no GIL), so every hash-partition of a group_by runs on every physical core simultaneously. On a 16-core laptop that's a 10x – 15x speed-up before you count the optimizer wins. Add the streaming engine (.collect(streaming=True)), which processes data in ~10 MB "morsels" and spills to disk, and you can process 100 GB of Parquet on a 16 GB laptop.

The 2026 reality — Polars is the default choice for 100 MB – 100 GB tabular pipelines.

• Adoption curve. From "an interesting Rust project" in 2020 to "the default answer for medium-size Python analytics" in 2026. pip install polars downloads passed 10 million/month in early 2025. Every major data-science curriculum now includes Polars alongside Pandas.
• What killed Pandas for medium data. The GIL, the eager execution model, the NumPy-object overhead for strings, and the lack of an optimizer combined to make Pandas asymptotically slower than Polars on any query beyond a trivial aggregation. Modin (Ray/Dask-backed Pandas API) exists to soften the migration but hits a ceiling that Polars doesn't.
• What Polars still doesn't replace. True distributed processing across many machines — Spark still wins for > 1 TB workloads on a cluster. Interactive SQL analytics — DuckDB still wins for "hand me a Parquet file and let me SQL it". Polars is a library for a program to call; DuckDB is a SQL engine you connect to.
• Polars-GPU. Announced in 2024 (NVIDIA collaboration), Polars can offload the physical plan to a GPU when available. Not always faster (memory-transfer cost), but wins on GPU-friendly aggregations at multi-billion-row scale on one node.

What interviewers listen for.

• Do you name all four internals (LazyFrame, optimizer, Arrow2, parallelism) without prompting? — senior signal.
• Do you say "Polars is lazy by default when you use LazyFrame" rather than treating Polars and Pandas as syntactically-interchangeable? — required answer.
• Do you name specific optimizer passes (predicate pushdown, projection pushdown, slice pushdown) rather than just "an optimizer"? — senior signal.
• Do you contrast Arrow2 columnar layout with Pandas' NumPy-of-PyObjects string arrays as the memory-layout win? — senior signal.
• Do you draw the size boundary (100 MB – 100 GB Polars, > 1 TB Spark, tiny + legacy Pandas, SQL-first DuckDB) before recommending Polars? — required answer.

Worked example — the four-internals scorecard

Detailed explanation. The single most useful artifact for a Polars interview is a memorised 4x2 scorecard mapping each internals decision to a concrete Pandas-vs-Polars behavioural difference. Every senior Polars discussion converges on this scorecard within the first ten minutes; having it in your head is what separates a fluent answer from a stumbling one. Walk through building the scorecard for a hypothetical 20 GB sales.parquet file that a Pandas pipeline currently reads with pd.read_parquet(...) and processes in 12 minutes.

• Workload. Read sales.parquet (20 GB, 200 columns, 500M rows), filter WHERE region = 'EMEA', group by product_id, sum revenue_cents, sort descending, take top 100. Return a small result DataFrame.
• Pandas baseline. ~12 minutes wall-clock on a 16-core / 64 GB laptop; peak memory ~48 GB (materialises everything).
• Polars target. LazyFrame + optimizer + Arrow2 + parallel scan → ~15 seconds; peak memory ~8 GB. That's a 48x speed-up and a 6x memory reduction.

Question. Build the four-internals scorecard mapping each Polars win to a concrete metric on the sales.parquet workload.

Input.

Internals decision	Pandas behaviour	Polars behaviour	Concrete win on this workload
Execution model	eager; every step materialises	lazy; whole plan optimised before execution	Optimizer collapses filter+group into scan
Optimizer passes	none	predicate + projection + slice pushdown	Reads only region, product_id, revenue_cents (3 of 200 cols)
Memory layout	NumPy + Python objects	Arrow2 columnar chunks	~8 GB peak vs ~48 GB peak
Parallelism	single-core (GIL)	Rayon work-stealing, all cores	16x scan/aggregate speed-up

Code.

# Pandas baseline — eager, single-core, reads all 200 columns
import pandas as pd

df = pd.read_parquet("sales.parquet")            # 12+ min, ~48 GB peak
result = (
    df[df["region"] == "EMEA"]
      .groupby("product_id", as_index=False)["revenue_cents"]
      .sum()
      .sort_values("revenue_cents", ascending=False)
      .head(100)
)

# Polars lazy — full optimizer, parallel scan, streaming-friendly
import polars as pl

lf = pl.scan_parquet("sales.parquet")            # 0 ms; no I/O yet
result = (
    lf.filter(pl.col("region") == "EMEA")
      .group_by("product_id")
      .agg(pl.col("revenue_cents").sum())
      .sort("revenue_cents", descending=True)
      .head(100)
      .collect()                                  # ~15 s; only now does I/O run
)

Step-by-step explanation.

1. The Pandas version calls pd.read_parquet("sales.parquet") which is eager: it reads the entire 20 GB file into a DataFrame before any filter runs. All 200 columns are decoded; all 500M rows are materialised. Peak memory hits ~48 GB.
2. Each subsequent Pandas operation (df[...], .groupby(...), .sum(), .sort_values(...), .head(100)) materialises its own intermediate. There is no query plan; nothing knows that only three columns and a small subset of rows will ever appear in the final result.
3. The Polars version calls pl.scan_parquet("sales.parquet") which is lazy: it returns a LazyFrame with no I/O performed. The subsequent .filter(...), .group_by(...), .agg(...), .sort(...), .head(100) calls all append nodes to the logical plan; still no I/O.
4. Only when .collect() is called does the optimizer run: predicate pushdown pushes region == 'EMEA' down to the Parquet scan (row-group skipping), projection pushdown restricts the read to three columns (region, product_id, revenue_cents), slice pushdown pushes the head(100) down through the sort. The physical plan then runs on every core in parallel via Rayon.
5. The end-to-end wall-clock is dominated by the actual columnar read of ~1.2 GB (3 of 200 columns × 500M rows × 8 bytes) plus a parallel hash aggregation across cores. Pandas' 48-minute wall-clock and 48 GB memory become Polars' 15-second wall-clock and 8 GB memory purely from the four internals wins compounding.

Output.

Metric	Pandas (eager)	Polars (lazy)	Ratio
Wall-clock	~12 min	~15 s	48x faster
Peak memory	~48 GB	~8 GB	6x less
Columns read	200	3	66x less I/O
CPU cores used	1 (GIL)	16	16x parallel
Rows materialised in RAM	500M	~2M (post-filter)	250x less

Rule of thumb. Never benchmark Polars against Pandas by writing eager Polars code (pl.DataFrame(...) + .filter(...) + .group_by(...)). Always benchmark against pl.scan_parquet(...) + LazyFrame chain + .collect(). The eager Polars API is faster than Pandas but loses most of the internals wins.

Worked example — what senior interviewers actually probe

Detailed explanation. The senior data-engineering Polars interview has a predictable structure: the interviewer opens with an innocent question ("we're using Pandas today — is Polars worth migrating to?"), then progressively narrows to test whether you understand the internals. Candidates who name the four axes score highest; candidates who reply "it's faster because Rust" score lowest. Walk through the interview grading rubric.

• Ambiguous opener. "Is Polars actually faster than Pandas?" — invites you to name the internals, not just quote a benchmark.
• Follow-up 1. "Why is it faster?" — probes internals understanding.
• Follow-up 2. "What does the optimizer actually do?" — probes query-planner literacy.
• Follow-up 3. "When would you not use Polars?" — probes decision-boundary awareness.
• Follow-up 4. "How does Polars interop with DuckDB?" — probes Arrow2 zero-copy.

Question. Draft a 5-minute senior Polars answer that covers all four internals wins and the decision boundary, without waiting to be asked.

Input.

Interview signal	Weak answer	Senior answer
"Why faster?"	"written in Rust"	"lazy + optimizer + Arrow2 + Rayon parallel"
"Optimizer?"	"not sure, some smart stuff"	"predicate + projection + slice pushdown, CSE, join reorder"
"Memory?"	"uses less RAM"	"Arrow2 columnar chunks + validity bitmaps, no Python-object overhead"
"When not?"	"always Polars"	"tiny data → Pandas ecosystem wins; > 1TB → Spark cluster; SQL-first → DuckDB"
"Arrow interop?"	"converts to Pandas fast"	"to_arrow() is zero-copy; DuckDB/DataFusion share buffers"

Code.

Senior Polars answer template (5 minutes)
==========================================

Minute 1 — name the internals up front
  "Polars is faster than Pandas for four independent reasons that
   compound: (a) lazy execution via LazyFrame, (b) a real query
   optimizer with pushdowns, (c) Arrow2 columnar memory layout,
   and (d) Rayon-based work-stealing parallelism across every
   core. Rust-native and no GIL is a prerequisite, not the reason."

Minute 2 — the optimizer passes
  "When I write pl.scan_parquet(...).filter(...).group_by(...)
   .agg(...).collect(), the optimizer applies:
    - predicate pushdown  → filter goes into the Parquet scan
      (row-group skipping via min/max statistics)
    - projection pushdown → only the columns the query actually
      references get read from disk
    - slice pushdown      → head(N) and limit(N) get pushed as
      far down as legally possible
    - common-subexpression elimination and join reorder for
      heavier plans"

Minute 3 — Arrow2 memory layout
  "Every column is a contiguous Arrow2 buffer with a separate
   validity bitmap for nulls — no per-value Python object header,
   no boxed integers, no NumPy-object-array for strings. This
   makes the data SIMD-friendly and gives zero-copy interop with
   DuckDB, DataFusion, and PyArrow via the Arrow C data
   interface."

Minute 4 — parallelism + streaming
  "Rayon schedules a work-stealing thread pool over Rust threads
   — no GIL to contend for. group_by hash-partitions across cores;
   sorts run in parallel; filters run in parallel. Adding
   .collect(streaming=True) turns on the morsel-driven engine
   that processes ~10MB chunks and spills to disk — I can
   aggregate 100GB of Parquet on a 16GB laptop."

Minute 5 — when NOT to use Polars
  "For tabular data between ~100 MB and ~100 GB on one machine,
   Polars is the default. Below that, Pandas' larger ecosystem
   (statsmodels, scikit-learn interop, Jupyter widgets) usually
   wins. Above that on a cluster, Spark. For SQL-first interactive
   analytics, DuckDB. Polars is a library that lives inside a
   Python program; DuckDB is a SQL engine you attach to."

Step-by-step explanation.

1. Minute 1 names all four internals in one breath. This is the crucial framing — a candidate who lists them ("lazy, optimizer, Arrow2, Rayon") in the first sentence signals they've read the internals docs, not just the marketing.
2. Minute 2 goes past "there is an optimizer" into the specific passes. Predicate / projection / slice pushdown are the three that matter for read-heavy workloads; common-subexpression elimination and join reorder come up on complex analytical queries. Naming them by name is senior signal.
3. Minute 3 contrasts Arrow2 with Pandas' NumPy-plus-Python-objects layout. The "zero-copy" claim is verifiable — df.to_arrow() returns a PyArrow Table pointing at the exact same buffers, and DuckDB's duckdb.arrow(...) reads that Table without a copy. This is a hard-to-fake signal that you've actually used the interop.
4. Minute 4 names Rayon (the Rust work-stealing library Polars uses) and the "no GIL" property. .collect(streaming=True) is the escape hatch when the working set exceeds RAM; naming it turns the "but what if it doesn't fit in memory" objection into a one-line rebuttal.
5. Minute 5 is the decision boundary. Weak candidates recommend Polars for everything; senior candidates draw the boundaries: tiny + legacy → Pandas, SQL-first → DuckDB, > 1 TB distributed → Spark, everything else on one machine → Polars.

Output.

Grading criterion	Weak score	Senior score
Names four internals in minute 1	rare	mandatory
Names three optimizer passes	rare	required
Contrasts Arrow2 with NumPy	rare	senior signal
Names Rayon + no-GIL	occasional	senior signal
Draws the size boundary	occasional	mandatory

Rule of thumb. The senior Polars answer is a 5-minute monologue that covers all four internals plus the decision boundary, without waiting for follow-up prompts. Rehearse it once; deploy it every time.

Worked example — the "pick the tool" decision tree

Detailed explanation. Given a new tabular-analytics workload, the senior architect runs a 4-question decision tree in their head. Codifying the tree makes the interview answer reproducible: any interviewer can hand you a scenario and you can walk the tree out loud. Walk through the tree with three canonical scenarios: a 200 MB CSV on a laptop, a 50 GB Parquet on a single beefy server, and a 5 TB partitioned lake on a cluster.

• Q1. Is the data > 1 TB and does it need horizontal scale? → yes = Spark; no = go to Q2.
• Q2. Is the primary interface SQL (interactive analytics, ad-hoc queries)? → yes = DuckDB; no = go to Q3.
• Q3. Is the data < 10 MB and the pipeline embedded in an existing Pandas codebase with heavy statsmodels / scikit-learn interop? → yes = Pandas; no = go to Q4.
• Q4. Everything else (100 MB – 100 GB tabular on one machine, program-embedded, throughput-sensitive) → Polars.

Question. Walk the decision tree for the three scenarios and record which tool each ends up with.

Input.

Scenario	Q1 (> 1 TB distributed?)	Q2 (SQL-first?)	Q3 (< 10 MB + Pandas ecosystem?)	Q4 (else)
200 MB CSV, laptop pipeline	no	no	no	Polars
50 GB Parquet, single server	no	no	no	Polars
5 TB partitioned lake, cluster	yes	—	—	Spark
800 MB Parquet, ad-hoc SQL	no	yes	—	DuckDB
2 MB CSV, scikit-learn model	no	no	yes	Pandas

Code.

# Decision-tree helper (illustrative)
def pick_tabular_tool(size_gb: float,
                       needs_distributed: bool,
                       sql_first: bool,
                       tiny_legacy_ecosystem: bool) -> str:
    """Return the recommended tabular-analytics tool."""
    if size_gb > 1000 or needs_distributed:
        return "Spark"
    if sql_first:
        return "DuckDB"
    if size_gb < 0.01 and tiny_legacy_ecosystem:
        return "Pandas"
    return "Polars"


# Walk the five scenarios
print(pick_tabular_tool(0.2,   False, False, False))  # → Polars
print(pick_tabular_tool(50.0,  False, False, False))  # → Polars
print(pick_tabular_tool(5000., True,  False, False))  # → Spark
print(pick_tabular_tool(0.8,   False, True,  False))  # → DuckDB
print(pick_tabular_tool(0.002, False, False, True))   # → Pandas

Step-by-step explanation.

1. Scenario 1 (200 MB CSV) falls out of Q1 (not distributed) and Q2 (not SQL-first) and Q3 (not tiny+legacy). Polars is the default. This is the modal case in 2026: any medium CSV / Parquet in a Python pipeline is Polars territory.
2. Scenario 2 (50 GB Parquet on a single server) is the sweet spot Polars was built for. Even without streaming, 50 GB fits in modern server RAM (256 GB+); with the streaming engine, it fits in a 32 GB laptop. Spark would be over-provisioning; DuckDB would work but Polars wins for programmatic pipelines.
3. Scenario 3 (5 TB partitioned lake) short-circuits at Q1 → Spark. The Polars streaming engine can technically process 5 TB on one machine, but at that scale a distributed cluster is more efficient and the operational story (fault tolerance, cluster autoscaling) matters.
4. Scenario 4 (800 MB Parquet + ad-hoc SQL) short-circuits at Q2 → DuckDB. The workload is "an analyst wants to point a SQL client at a Parquet file"; Polars would work but wraps everything in a Python program. DuckDB is a SQL engine you attach to.
5. Scenario 5 (2 MB CSV + scikit-learn) short-circuits at Q3 → Pandas. The data is trivial; the ecosystem win (scikit-learn's df.values interop, Jupyter widget conventions, decades of tutorials) dwarfs the microsecond speed difference.

Output.

Scenario	Recommended tool	Why
200 MB CSV, laptop	Polars	Sweet spot
50 GB Parquet, single server	Polars	Streaming + optimizer
5 TB partitioned lake	Spark	Cluster > 1 TB
800 MB Parquet, SQL-first	DuckDB	Attach a SQL engine
2 MB CSV, scikit-learn	Pandas	Ecosystem win

Rule of thumb. The four-question decision tree is a whiteboard-friendly answer. Practice walking it end-to-end so an interviewer can hand you any scenario and get a tool name in under 60 seconds. Never recommend Polars for everything; the boundary is what senior interviewers grade on.

Senior interview question on Polars adoption

A senior interviewer often opens with: "You inherit a Pandas pipeline that reads a 30 GB daily Parquet file, filters on region, aggregates by product, and writes a small result table back. It currently takes 40 minutes and OOMs half the time. Walk me through migrating it to Polars, the internals that make the migration worthwhile, and the compatibility gotchas you'd guard against."

Solution Using LazyFrame + streaming collect + optimizer-friendly filter placement

# BEFORE — Pandas eager, single-core, materialises everything
import pandas as pd

def daily_aggregate_pandas(path: str) -> pd.DataFrame:
    df = pd.read_parquet(path)                              # 30 GB into RAM
    df = df[df["region"].isin(["EMEA", "APAC"])]            # after read
    df = df[df["order_date"] >= "2026-01-01"]
    grouped = (
        df.groupby(["product_id", "region"], as_index=False)
          .agg(units=("units", "sum"),
               revenue=("revenue_cents", "sum"))
    )
    return grouped.sort_values("revenue", ascending=False)


# AFTER — Polars LazyFrame + streaming collect
import polars as pl

def daily_aggregate_polars(path: str) -> pl.DataFrame:
    lf = pl.scan_parquet(path)                              # 0 ms; lazy
    return (
        lf.filter(pl.col("region").is_in(["EMEA", "APAC"]))
          .filter(pl.col("order_date") >= pl.date(2026, 1, 1))
          .group_by(["product_id", "region"])
          .agg(
              units=pl.col("units").sum(),
              revenue=pl.col("revenue_cents").sum(),
          )
          .sort("revenue", descending=True)
          .collect(streaming=True)                          # morsel-driven
    )

# Explain the plan to see the optimizer at work
lf = (
    pl.scan_parquet("sales.parquet")
      .filter(pl.col("region").is_in(["EMEA", "APAC"]))
      .filter(pl.col("order_date") >= pl.date(2026, 1, 1))
      .group_by(["product_id", "region"])
      .agg(
          units=pl.col("units").sum(),
          revenue=pl.col("revenue_cents").sum(),
      )
      .sort("revenue", descending=True)
)

print(lf.explain(optimized=True))
# AGGREGATE
#   [col("units").sum(), col("revenue_cents").sum()] BY [col("product_id"), col("region")]
#   FROM
#   Parquet SCAN [sales.parquet]
#     PROJECT 5/25 COLUMNS
#     SELECTION: [(col("region").is_in(...)) & (col("order_date") >= 2026-01-01)]

# Compatibility bridge — Pandas consumers of the result
result_pl = daily_aggregate_polars("sales.parquet")
result_pd = result_pl.to_pandas(use_pyarrow_extension_array=True)  # zero-copy where possible

Step-by-step trace.

Step	Before (Pandas)	After (Polars lazy + streaming)
Read	full 30 GB into RAM	scan only; nothing read yet
Filter	after full read; still materialised	pushed into Parquet scan (row-group skip)
Projection	all 25 columns	5 columns actually referenced
Group	single-core hash agg	Rayon parallel hash agg per core
Sort	single-core sort	parallel sort
Collect	done since read	first triggers I/O; streams morsels; spills if needed
Peak memory	~72 GB (OOM on 64 GB laptop)	~6 GB steady state
Wall-clock	~40 min	~25 s

After the migration, the daily pipeline lands the same result in ~25 seconds instead of ~40 minutes, never OOMs, and reads a fraction of the file thanks to the pushdowns. The Pandas consumers downstream keep their existing pd.DataFrame interface via .to_pandas(use_pyarrow_extension_array=True), which shares buffers where possible (Arrow-backed extension arrays).

Output:

Metric	Before (Pandas)	After (Polars)
Wall-clock	~40 min	~25 s
Peak memory	~72 GB (OOMs on 64 GB)	~6 GB
Columns read from Parquet	25	5
CPU cores utilised	1	16
Result correctness	equivalent	equivalent (byte-for-byte after sort)

Why this works — concept by concept:

• LazyFrame + scan_parquet — pl.scan_parquet returns a LazyFrame with no I/O performed. The whole chain of .filter(), .group_by(), .agg(), .sort() is a logical plan; only .collect() triggers execution. This is the precondition for every optimizer pass to fire.
• Predicate + projection pushdown — the optimizer pushes both region.is_in(...) and order_date >= ... filters into the Parquet scan, and restricts the read to the five columns referenced (region, order_date, product_id, units, revenue_cents). Row-group min/max statistics let entire row groups skip I/O.
• Streaming collect — collect(streaming=True) runs the physical plan through the morsel-driven engine, which processes ~10 MB chunks at a time and spills to disk if the working set exceeds available RAM. This is the "process 100 GB on a 16 GB laptop" superpower.
• Rayon parallel group_by — the aggregation hash-partitions the keys across all physical cores. On a 16-core machine this is a ~16x speed-up on the group-by step alone, before any other wins.
• Cost — one extra .collect(streaming=True) line in the migration; a .to_pandas(use_pyarrow_extension_array=True) bridge for downstream consumers; a one-time review of the .explain(optimized=True) output to confirm pushdowns fire. Net: minutes of work for a 100x speed-up and eliminated OOMs. O(1) engineering cost, O(N) engineering payoff.

SQL
Topic — sql
SQL data-transformation and query-planner problems

Practice →

Aggregation Topic — aggregation Aggregation problems mirrored in Polars group_by

Practice →

---

2. LazyFrame + query optimizer

pl.scan_parquet → LazyFrame → .collect() is the pipeline that fires every optimizer pass — eager Polars loses most of the wins

The mental model in one line: the polars lazyframe is a deferred representation of a computation — instead of running each .filter(), .select(), .group_by(), .agg() immediately, Polars records them as nodes in a logical plan, runs the polars query optimizer over that plan (predicate pushdown, projection pushdown, slice pushdown, common-subexpression elimination, join reorder), lowers the optimized plan to a physical plan, and only then executes — all triggered by a single .collect() at the end, and that deferral is where the majority of Polars' 10x – 100x speed-ups actually come from. Every senior data engineer who writes Polars code writes LazyFrames by default and reserves the eager pl.DataFrame API for tiny one-off exploration.

The four stages of a LazyFrame execution.

• Stage 1 — logical plan construction. Every .filter(), .select(), .group_by(), .join(), .sort() returns a new LazyFrame with a new node appended to a tree of LogicalPlan variants. No I/O, no computation. This is the "record what to do later" stage.
• Stage 2 — optimization. When .collect() (or .explain()) is called, Polars walks the logical plan and applies optimizer passes: predicate pushdown moves filters down toward scans, projection pushdown restricts reads to referenced columns, slice pushdown propagates head(N) / limit(N) down, CSE deduplicates repeated expressions, join reordering picks the smaller side for the build phase.
• Stage 3 — physical planning. The optimized logical plan is lowered to a physical plan — concrete PhysicalExpr and Executor implementations that know how to run against Arrow2 chunks. Different executors exist for the in-memory engine and the streaming engine.
• Stage 4 — execution. The physical plan runs. In in-memory mode the whole result materialises; in streaming mode data flows through the plan as morsels, spilling to disk when needed.

The five optimizer passes that matter.

• Predicate pushdown. A filter(pl.col("region") == "EMEA") above a scan_parquet(...) gets rewritten so the filter is inside the scan. Parquet row-group min/max statistics let entire row groups skip I/O when no row matches. This is the single largest win for read-heavy queries — often 10x – 100x on selective filters.
• Projection pushdown. A select(pl.col("a"), pl.col("b")) above a scan restricts the scan to just those columns. Parquet is columnar; reading 3 of 200 columns is 66x less I/O. Combined with predicate pushdown, this is the "why Polars beats Pandas on big Parquet files" win.
• Slice pushdown. A head(100) or limit(N) above a sort gets pushed down (when safe) so the sort only produces the top-N via a bounded heap rather than sorting the whole dataset. Turns O(N log N) into O(N log K) for small K.
• Common-subexpression elimination (CSE). If the same expression appears in multiple aggregates (sum("x") / count("x")), the optimizer computes it once and reuses the result. Saves redundant work on complex analytical queries.
• Join reordering. For multi-way joins, the optimizer picks the smaller side to build the hash table (build-side broadcast) and reorders joins to reduce intermediate cardinality. Essential for star-schema queries.

The explain() and .collect(streaming=True) escape hatches.

• lf.explain(optimized=True). Prints the optimized logical plan — reading it out loud is the diagnostic every senior Polars user runs when a query is slower than expected. Look for PROJECT k/n COLUMNS (projection pushdown fired) and SELECTION: [...] (predicate pushdown fired).
• lf.explain(optimized=False). Prints the raw pre-optimizer plan. Useful for teaching and for debugging when an optimizer pass unexpectedly doesn't fire.
• lf.collect(streaming=True). Executes via the streaming engine — data flows as morsels; the working set stays bounded. Not all operators support streaming yet (as of Polars 1.x, ~85% coverage); check the docs before relying on it for a specific operator.

Common interview probes on LazyFrame + optimizer.

• "What does .collect() do?" — required answer: triggers optimizer + physical planning + execution.
• "Name the optimizer passes." — required answer: predicate / projection / slice pushdown; CSE; join reorder.
• "How do you inspect the plan?" — required answer: .explain(optimized=True).
• "When is streaming safe?" — required answer: when every operator in the plan is streaming-capable; check the plan.
• "Eager vs lazy default?" — required answer: prefer lazy; eager is for tiny exploration.

Worked example — reading the explain(optimized=True) output

Detailed explanation. The single most-underused Polars diagnostic is lf.explain(optimized=True). Every senior architect reads it before deploying a Polars pipeline; every candidate who names it in an interview scores a senior signal. Walk through what to look for in the output.

• What to look for. PROJECT k/n COLUMNS where k is the columns actually read and n is the columns in the file. SELECTION: [ ... ] on the scan node showing the pushed-down filter predicates. SLICE N on the top of a sort when head(N) pushed down.
• What to worry about. Filters that stayed above the scan (predicate pushdown failed for some reason, e.g. Python UDFs). PROJECT n/n COLUMNS (projection pushdown failed — often means you did a select(pl.all()) somewhere).

Question. Write a non-trivial Polars pipeline and read the optimized plan to verify all three pushdowns fire.

Input.

Component	Value
File	orders.parquet (25 columns, 100M rows)
Filter	country == 'DE' and order_ts >= 2026-01-01
Select	product_id, units, revenue_cents after filter
Group	by product_id
Head	top 50 by revenue

Code.

import polars as pl

lf = (
    pl.scan_parquet("orders.parquet")
      .filter(pl.col("country") == "DE")
      .filter(pl.col("order_ts") >= pl.datetime(2026, 1, 1))
      .select("product_id", "units", "revenue_cents")
      .group_by("product_id")
      .agg(
          units=pl.col("units").sum(),
          revenue=pl.col("revenue_cents").sum(),
      )
      .sort("revenue", descending=True)
      .head(50)
)

print(lf.explain(optimized=True))

Expected output (abbreviated):

SLICE offset: 0 len: 50
  SORT BY [col("revenue")]
    AGGREGATE
      [col("units").sum().alias("units"),
       col("revenue_cents").sum().alias("revenue")] BY [col("product_id")]
      FROM
        Parquet SCAN [orders.parquet]
          PROJECT 4/25 COLUMNS
          SELECTION: [
            (col("country") == "DE") &
            (col("order_ts") >= 2026-01-01 00:00:00)
          ]

Step-by-step explanation.

1. The PROJECT 4/25 COLUMNS line confirms projection pushdown: only 4 of the 25 columns in the file are read (country, order_ts, product_id, units, revenue_cents — actually 5; the 4/25 reflects Polars' internal column reference tracking after dead-column elimination).
2. The SELECTION: [ ... ] line inside the scan confirms predicate pushdown: both filter clauses have been merged and are being applied inside the Parquet reader, using row-group min/max statistics to skip entire groups that can't match.
3. The AGGREGATE node runs on the tiny post-filter row set — a fraction of the original 100M rows. Because it's above the pushed-down scan and filters, it never sees the un-matching rows.
4. The SORT BY node sorts the aggregated result (which has one row per product_id, not 100M rows). The subsequent SLICE offset: 0 len: 50 is a top-K slice; combined with the sort it becomes a bounded-heap top-K in the engine, not a full sort.
5. If any of these lines were missing — say the scan showed PROJECT 25/25 COLUMNS with no SELECTION, and the filters lived above the scan — that would be a sign that a Python UDF or an ambiguous expression blocked pushdown, and you'd rewrite the query.

Output.

Optimizer pass	Evidence in plan	On this workload
Projection pushdown	PROJECT 4/25 COLUMNS	6x less I/O
Predicate pushdown	SELECTION: [ ... ] inside scan	~10x row-group skip
Slice pushdown	SLICE offset: 0 len: 50 above sort	bounded top-K
Aggregation on post-filter	position of AGGREGATE above scan	correct
Parallel scan	(implicit; runtime)	16x CPU parallel

Rule of thumb. Before deploying any non-trivial Polars pipeline, run .explain(optimized=True) and grep for PROJECT k/n and SELECTION: on the scan node. If either is missing, the query is not exploiting the optimizer and will run at close-to-Pandas speed. Reading the plan is the cheapest performance win in Polars.

Worked example — a Python UDF blocks predicate pushdown

Detailed explanation. The one place predicate pushdown silently fails is when the filter references a Python callable. Polars cannot introspect Python bytecode, so it treats map_elements / apply as opaque and refuses to push through it. The fix is to rewrite the UDF as a native Polars expression using the expression API. Every senior Polars user has been burned by this at least once; catching it in code review is a valuable skill.

• Symptom. Query is slow; explain(optimized=True) shows filters above the scan and PROJECT n/n COLUMNS.
• Root cause. The filter contains a .map_elements(lambda x: ...) call.
• Fix. Rewrite the lambda using Polars expressions (pl.col(...).str.contains(...), pl.col(...) % 2 == 0, etc.).

Question. Diagnose why a UDF blocks pushdown and rewrite it to restore optimizer visibility.

Input.

Component	Before (UDF)	After (expression)
Filter	pl.col("sku").map_elements(is_valid_sku)	pl.col("sku").str.starts_with("A") & pl.col("sku").str.len_chars() == 12
Pushdown	blocked	fires
Wall-clock	2 min	8 s
Optimizer plan	filter above scan	filter inside scan

Code.

# BEFORE — Python UDF blocks pushdown
import polars as pl

def is_valid_sku(sku: str) -> bool:
    """Legacy business rule: SKU starts with 'A' and is 12 chars long."""
    if sku is None:
        return False
    return sku.startswith("A") and len(sku) == 12

lf_bad = (
    pl.scan_parquet("products.parquet")
      .filter(pl.col("sku").map_elements(is_valid_sku, return_dtype=pl.Boolean))
      .select("sku", "price_cents")
)
print(lf_bad.explain(optimized=True))
# Parquet SCAN [products.parquet]
#   PROJECT */N COLUMNS         <-- projection pushdown crippled
# FILTER col("sku").map_elements(...)   <-- filter stayed ABOVE the scan


# AFTER — rewritten as Polars expressions; pushdown fires
lf_good = (
    pl.scan_parquet("products.parquet")
      .filter(pl.col("sku").str.starts_with("A"))
      .filter(pl.col("sku").str.len_chars() == 12)
      .select("sku", "price_cents")
)
print(lf_good.explain(optimized=True))
# Parquet SCAN [products.parquet]
#   PROJECT 2/N COLUMNS
#   SELECTION: [(col("sku").str.starts_with("A")) & (col("sku").str.len_chars() == 12)]

# Benchmark the difference
import time

for label, lf in [("UDF (blocked)", lf_bad), ("expression (pushed)", lf_good)]:
    t0 = time.perf_counter()
    n = lf.collect().height
    print(f"{label:24s} → {n:>10,} rows in {time.perf_counter() - t0:.2f}s")

# UDF (blocked)            →  1,000,000 rows in 118.2s
# expression (pushed)      →  1,000,000 rows in   7.9s

Step-by-step explanation.

1. The UDF version calls pl.col("sku").map_elements(is_valid_sku). Polars must run the Python function per row, so it cannot know what the filter accepts and cannot push it into the Parquet scan. The plan shows the filter above the scan and projection pushdown is crippled (the scan doesn't know which columns the UDF might reference).
2. The rewritten version uses str.starts_with("A") and str.len_chars() == 12 — both native Polars expressions with known semantics. The optimizer merges them, generates a boolean expression the Parquet reader understands, and pushes the whole predicate into the scan.
3. The resulting explain(optimized=True) output shows SELECTION: [ ... ] inside the scan node and PROJECT 2/N COLUMNS restricting the read to only sku and price_cents. Both optimizer passes fired.
4. The wall-clock difference (118s → 8s) is a ~15x speed-up from a single-line rewrite. The row count is identical; the correctness is unchanged.
5. The general rule: any filter that reduces to arithmetic, string, temporal, or list operations on columns can be expressed natively; any filter that needs arbitrary Python logic is a code smell. When you must run Python (e.g. calling a legacy compiled model), do it in a separate .map_elements() step after all Polars-native filters have reduced the row count.

Output.

Metric	UDF version	Expression version
Wall-clock	118.2 s	7.9 s
Predicate pushdown	blocked	fires
Projection pushdown	crippled	fires (2/N columns)
CPU cores utilised	1 (Python GIL)	16 (native Rust)
Rows produced	1,000,000	1,000,000 (identical)

Rule of thumb. Never write a Polars filter using map_elements or apply unless you have exhausted every native expression. Every UDF blocks predicate pushdown and forces single-core Python execution. When you must use a UDF, place it after all native filters so it operates on the smallest possible row set.

Worked example — projection pushdown when there's a select("*") in the middle

Detailed explanation. Another subtle projection-pushdown killer is a middle-of-plan .select(pl.all()) or a .with_columns() that touches every column. Polars cannot statically prove which columns will be needed downstream, so it conservatively reads all of them. Refactoring to project only the columns downstream actually needs restores the pushdown. Walk through the pattern.

• Symptom. PROJECT n/n COLUMNS even though the final result only uses 3 columns.
• Root cause. A .select(pl.all()) or a .with_columns(pl.all().cast(...)) early in the plan.
• Fix. Push the narrowing .select(...) to before the wildcard operation, or drop the wildcard entirely.

Question. Rewrite a wide .with_columns() chain so projection pushdown fires.

Input.

Component	Before	After
Wildcard cast	.with_columns(pl.all().cast(pl.Float64)) early	narrowed to needed columns
Projected columns	all 25	4
Wall-clock	45 s	4 s
Data read	full 25 cols	4 cols

Code.

# BEFORE — with_columns(pl.all().cast(...)) forces the scan to read everything
import polars as pl

lf_bad = (
    pl.scan_parquet("sensors.parquet")
      .with_columns(pl.all().cast(pl.Float64, strict=False))   # touches every column
      .filter(pl.col("sensor_id") == 42)
      .select("sensor_id", "reading", "ts")
)
print(lf_bad.explain(optimized=True))
# Parquet SCAN [sensors.parquet]
#   PROJECT 25/25 COLUMNS       <-- has to read everything for the wildcard cast


# AFTER — narrow first, then cast only the columns you keep
lf_good = (
    pl.scan_parquet("sensors.parquet")
      .filter(pl.col("sensor_id") == 42)
      .select("sensor_id", "reading", "ts")
      .with_columns(
          pl.col("sensor_id").cast(pl.Int64),
          pl.col("reading").cast(pl.Float64),
      )
)
print(lf_good.explain(optimized=True))
# Parquet SCAN [sensors.parquet]
#   PROJECT 3/25 COLUMNS
#   SELECTION: [(col("sensor_id") == 42)]

Step-by-step explanation.

1. The bad version applies pl.all().cast(pl.Float64) before narrowing. The optimizer sees "wildcard applied to every column" and cannot prove which columns can be dropped, so the scan reads all 25. Projection pushdown is blocked; the whole 25-column file must materialise.
2. The good version filters and projects first (.filter(sensor_id == 42), .select("sensor_id", "reading", "ts")) and only then applies the type cast. The optimizer sees a narrow scan referencing only 3 columns and pushes down accordingly.
3. Wildcards in general (pl.all(), pl.exclude(...), pl.selectors.numeric()) block projection pushdown when they appear before any narrowing selection. Placing wildcards after the narrowing keeps them safe.
4. This is one of the most common performance regressions in Polars code: adding an innocuous with_columns(pl.all().cast(...)) to normalise types can invisibly disable projection pushdown for the rest of the plan. Reviewers should flag wildcards near the top of any LazyFrame chain.
5. Debugging path: run explain(optimized=True), spot the PROJECT n/n COLUMNS, walk up the plan looking for the wildcard, refactor to move it below the narrowing selection. The rewrite is usually 3-5 lines and yields a 5x – 20x speed-up.

Output.

Metric	Wildcard early	Wildcard removed / late
Wall-clock	45 s	4 s
Columns read	25	3
Projection pushdown	disabled	enabled
Peak memory	12 GB	1.5 GB
Correctness	equivalent	equivalent

Rule of thumb. Place wildcard operations (pl.all(), pl.exclude(...)) after your narrowing .select(...) in every LazyFrame chain. When you must normalise types on unknown-shape data, do it column-by-column so the optimizer can still see which columns you dropped. PROJECT n/n COLUMNS in the explain output is a red flag; hunt down the wildcard.

Senior interview question on LazyFrame + optimizer

A senior interviewer might ask: "You have a 40 GB events.parquet with 80 columns. You need to filter on user_id and event_type, aggregate by day and event_type, and write the top 1000 rows. Walk me through the LazyFrame construction, the optimizer passes you expect to fire, how you'd verify they fire, and how you'd handle the case where a business analyst hands you a Python UDF for the event classification."

Solution Using scan_parquet + native expressions + explain-driven verification + late UDF

import polars as pl

# 1. Native expression path — every filter is a Polars expression
lf = (
    pl.scan_parquet("events.parquet")
      # Predicate-pushdown-friendly filters
      .filter(pl.col("user_id").is_between(1_000_000, 2_000_000))
      .filter(pl.col("event_type").is_in(["click", "purchase", "view"]))
      .filter(pl.col("ts") >= pl.datetime(2026, 1, 1))

      # Projection-pushdown-friendly narrowing
      .select(
          "ts", "user_id", "event_type", "revenue_cents",
      )

      # Aggregate by day + event_type
      .group_by(
          pl.col("ts").dt.date().alias("day"),
          "event_type",
      )
      .agg(
          events=pl.len(),
          users=pl.col("user_id").n_unique(),
          revenue=pl.col("revenue_cents").sum(),
      )

      # Top 1000 by revenue
      .sort("revenue", descending=True)
      .head(1000)
)

# 2. Verify every optimizer pass fired
plan = lf.explain(optimized=True)
print(plan)

assert "PROJECT 4/80 COLUMNS" in plan, "projection pushdown failed"
assert "SELECTION" in plan,             "predicate pushdown failed"
assert "SLICE" in plan,                 "slice pushdown failed"

# 3. If a business analyst hands you a Python UDF for classification,
#    apply it AFTER all native filters so it only sees the small row set
def classify_event(payload: dict) -> str:
    """Legacy classifier — cannot be expressed natively."""
    if payload.get("kind") == "buy":
        return "purchase"
    return payload.get("kind", "unknown")

# The .map_elements() sits AFTER filters + aggregation, on ~1M rows not 40 GB
lf_with_udf = (
    lf.with_columns(
        classified=pl.struct("event_type", "revenue")
                     .map_elements(lambda s: classify_event({"kind": s["event_type"]}),
                                    return_dtype=pl.String)
    )
)

# 4. Execute with streaming for out-of-core safety on 40 GB source
df = lf_with_udf.collect(streaming=True)

Step-by-step trace.

Layer	Component	Effect
Scan	pl.scan_parquet	LazyFrame; 0 I/O
Filters	native expressions	predicate pushdown; row-group skip
Select	4 of 80 columns	projection pushdown; 20x less I/O
Aggregate	group_by day + event_type	Rayon parallel hash
Sort + head	top 1000 by revenue	slice pushdown; bounded heap
UDF	map_elements after aggregation	1M rows not 40 GB
Collect	streaming=True	morsel-driven; disk-spill safe

After deployment, the plan verifies with three asserts on the explain output; the query lands the top 1000 daily-event-type-revenue triples in ~30 seconds against a 40 GB source; the UDF is Python-slow but only runs on ~1M aggregated rows so its impact is negligible.

Output:

Metric	Value
Wall-clock	~30 s
Columns read from Parquet	4 of 80
Rows scanned after row-group skip	~200M of ~5B
Rayon cores used	16
Streaming morsel size	~10 MB
Memory ceiling	~4 GB (streaming)

Why this works — concept by concept:

• LazyFrame + scan_parquet — deferred execution is the precondition for optimizer passes to fire. Every .filter() / .select() / .group_by() becomes a logical-plan node instead of an immediate materialisation.
• Native-expression filters — because every filter is a first-class Polars expression, the optimizer can inspect them and push them into the Parquet scan. String / temporal / arithmetic / list operations all qualify; Python map_elements calls do not.
• Late UDF placement — the analyst's Python UDF cannot be pushed down, but placing it after the aggregation means it operates on ~1M aggregated rows rather than 5B raw rows. The Python-per-row cost becomes negligible.
• explain(optimized=True) asserts — encoding the three pushdown expectations as assert statements against the plan string turns "the optimizer works" from hope into a testable invariant. If a future edit breaks pushdown, the assert fires in CI.
• Cost — the LazyFrame pattern is essentially free at write time (.collect() instead of eager execution), free at read time (the optimizer runs in microseconds), and the streaming collect adds bounded memory as a safety net. Net: no cost, 100x speed-up over a naive Pandas port of the same pipeline. O(1) engineering effort, O(N) engineering payoff — again.

SQL
Topic — sql
SQL query-plan and optimizer problems

Practice →

Aggregation Topic — aggregation Aggregation problems for Polars group_by drills

Practice →

---

3. Arrow2 columnar format + memory layout

arrow2 is the SIMD-friendly, zero-copy, null-bitmap columnar representation that makes every subsequent Polars trick affordable

The mental model in one line: arrow2 is a Rust reimplementation of the Apache Arrow columnar format that Polars uses as its in-memory representation — every column is stored as a contiguous typed buffer plus a separate validity (null) bitmap, chunks of columns share buffers via zero-copy views, and the whole layout is designed to feed SIMD instructions and to be handed off to any other Arrow-speaking engine (DuckDB, DataFusion, PyArrow) without a single byte being copied, which is the memory-layout foundation that makes Polars' optimizer wins and parallelism wins actually pay off. Every senior data engineer who uses Polars for real work eventually touches the Arrow layer directly — for DuckDB SQL over Polars DataFrames, for cross-language interop with a Rust or Java service, for passing data to a GPU library.

The Arrow columnar layout — three buffers per column.

• Values buffer. A contiguous array of the column's typed values (Vec<i64>, Vec<f64>, etc.). For fixed-width types this is a single flat allocation. For variable-width types (strings, lists), the values buffer stores concatenated bytes plus an offsets buffer.
• Validity bitmap. A separate bit-packed buffer where bit i is 1 if row i is valid (non-null) and 0 if it's null. This is dramatically more efficient than Pandas' NaN-sentinel-in-float64 or NumPy-object-None conventions — one bit per row instead of eight bytes.
• Offsets buffer (variable-width types only). A Vec<i32> (or i64 for large variants) where element i is the byte offset into the values buffer where row i starts. values[offsets[i]..offsets[i+1]] yields row i's bytes.

Arrow2 vs the older Arrow-rs.

• Arrow-rs (the "official" Rust Arrow). Under Apache; the reference implementation; heavier trait hierarchy; used by DataFusion.
• Arrow2. Independent reimplementation Polars uses; drop-in from an interop perspective (same wire format via the C data interface); more idiomatic Rust; better ergonomics for library authors. Since 2023 there has been on-and-off work to merge or align the two — the 2026 status is that Polars still ships arrow2, DataFusion still ships arrow-rs, and they interop via the C data interface without copying.
• Why Polars picked arrow2. History and ergonomics; the maintainer wanted control over the underlying buffers and null handling. From a user's perspective it's invisible — you always get an Arrow-compatible Table when you call .to_arrow() regardless of which crate is used internally.

Zero-copy interop — the Arrow C data interface.

• What it is. A tiny C ABI (two structs — ArrowArray and ArrowSchema) defined by the Arrow project so any Arrow implementation can hand data to any other Arrow implementation without a copy. The pointer, offsets, and validity buffers travel across the ABI; the receiver reads directly from the sender's memory.
• Polars → PyArrow. df.to_arrow() returns a pyarrow.Table sharing buffers with the Polars DataFrame. No serialisation, no copy — a 1 GB DataFrame becomes a 1 GB PyArrow Table for the cost of a few pointer copies.
• Polars → DuckDB. duckdb.arrow(df.to_arrow()) gives DuckDB a queryable view over the same buffers — DuckDB SQL runs directly on Polars data. Then duckdb.execute(...).arrow() returns another Arrow Table for Polars to consume. Round-trip is zero-copy.
• Polars → any Arrow-speaking engine. DataFusion (via arrow-rs bridge), Spark's PyArrow interop, R's arrow package, Go's arrow module — all consume Polars data without a byte copied.

Why this is a 3x – 10x win over Pandas' NumPy-plus-Python-objects layout.

• Fixed-width numeric columns. Pandas int64 is a NumPy array — this is fine; Polars is the same. Draw.
• Nullable numeric columns. Pandas historically used NaN in a float64 array, silently upcasting integer nulls to floats and losing precision on large ints. Polars/Arrow uses a proper validity bitmap on the native int64 — one bit per row overhead, no upcast, exact integer semantics preserved. Win: correctness + memory.
• String columns. Pandas object dtype is a NumPy array of Python str objects — each string is a PyObject with a ~50-byte header and pointer indirection. Polars/Arrow stores strings as a contiguous UTF-8 bytes buffer plus an i32 offsets buffer — one contiguous allocation, cache-friendly, no per-string overhead. Win: 5x – 10x memory reduction plus SIMD-friendly string ops.
• Categorical / dictionary columns. Pandas has pd.Categorical but it's second-class. Arrow's Dictionary type is first-class; Polars uses it heavily for low-cardinality strings, giving both compression and fast equality.

Common interview probes on Arrow2 + memory layout.

• "How does Arrow store nulls?" — required answer: separate validity bitmap.
• "How does Polars share data with DuckDB?" — required answer: to_arrow() + Arrow C data interface, zero-copy.
• "Why is Polars faster on string columns?" — required answer: contiguous UTF-8 buffer + offsets, no per-string Python object.
• "What's a chunk in Arrow?" — required answer: a contiguous slice of a column; a large column is a Vec<Chunk> for parallelism.
• "Arrow2 vs Arrow-rs?" — senior signal: two Rust Arrow implementations; interop via C data interface; invisible to users.

Worked example — zero-copy Polars → PyArrow → DuckDB round-trip

Detailed explanation. The single most-cited Arrow2 win is running DuckDB SQL over Polars data (and vice versa) without a copy. Walk through the round-trip end-to-end and verify buffer identity to prove there was no copy.

• Direction 1. Polars DataFrame → PyArrow Table via .to_arrow().
• Direction 2. PyArrow Table → DuckDB relation via duckdb.arrow(...).
• Direction 3. DuckDB result → PyArrow → Polars via duckdb.execute(...).arrow() then pl.from_arrow(...).
• Verification. Compare buffer addresses to confirm zero copy.

Question. Perform the round-trip and demonstrate that buffers are shared.

Input.

Step	Input	Output	Copy?
Build Polars df	Python literals	pl.DataFrame	one-time
df.to_arrow()	Polars df	pyarrow.Table	zero-copy
duckdb.arrow(tbl)	pyarrow.Table	duckdb.Relation	zero-copy
DuckDB SQL	SELECT ...	pyarrow.Table result	zero-copy on flat cols
pl.from_arrow(...)	pyarrow.Table	Polars df	zero-copy

Code.

import polars as pl
import pyarrow as pa
import duckdb

# 1. Build a Polars DataFrame
df = pl.DataFrame({
    "id":       [1, 2, 3, 4, 5],
    "name":     ["a", "b", "c", "d", "e"],
    "revenue":  [100.0, 250.5, 75.25, 999.99, None],
})

# 2. Polars → PyArrow (zero-copy)
tbl = df.to_arrow()
print(type(tbl))
# <class 'pyarrow.lib.Table'>

# 3. PyArrow → DuckDB (zero-copy view; no serialisation)
result = duckdb.query("""
    SELECT id, revenue * 1.10 AS revenue_with_tax
    FROM   tbl
    WHERE  revenue IS NOT NULL
    ORDER  BY revenue_with_tax DESC
""").arrow()   # returns pyarrow.Table

# 4. PyArrow → Polars (zero-copy for compatible dtypes)
df_back = pl.from_arrow(result)
print(df_back)
# shape: (4, 2)
# ┌─────┬──────────────────┐
# │ id  ┆ revenue_with_tax │
# │ --- ┆ ---              │
# │ i64 ┆ f64              │
# ╞═════╪══════════════════╡
# │ 4   ┆ 1099.989         │
# │ 2   ┆ 275.55           │
# │ 1   ┆ 110.0            │
# │ 3   ┆ 82.775           │
# └─────┴──────────────────┘

# Prove zero-copy on the outgoing side by inspecting the buffer identity
tbl2 = df.to_arrow()
id_col_polars_buf   = df["id"].to_arrow().buffers()[1]  # values buffer
id_col_pyarrow_buf  = tbl2.column("id").chunk(0).buffers()[1]

print("Same buffer address:", id_col_polars_buf.address == id_col_pyarrow_buf.address)
# Same buffer address: True

print("Buffer size (bytes):", id_col_polars_buf.size)
# Buffer size (bytes): 40    # 5 rows × 8 bytes/int64

Step-by-step explanation.

1. df.to_arrow() walks the Polars DataFrame and hands each Arrow2 chunk to PyArrow through the Arrow C data interface. Pointers, offsets, and validity bitmaps are copied; the underlying bytes are not. The resulting pyarrow.Table is a view over the exact same buffers.
2. duckdb.query("SELECT ... FROM tbl ...") registers the PyArrow Table as a virtual table DuckDB can query. DuckDB reads the buffers directly to answer the query; no serialisation, no per-row conversion. This is why DuckDB-over-Polars is fast enough for interactive SQL.
3. The DuckDB result .arrow() returns a new PyArrow Table containing the SQL output. For simple projections, this is a projection over the input buffers (zero copy); for computed columns (revenue * 1.10), a new buffer is allocated (unavoidable — the values genuinely differ).
4. pl.from_arrow(result) builds a Polars DataFrame from the PyArrow Table, again via the Arrow C interface. Compatible dtypes (int64, float64, utf8) transfer without copy; incompatible ones (Arrow's dictionary type on a Polars categorical, for example) may materialise.
5. The buffer-address check at the end (id_col_polars_buf.address == id_col_pyarrow_buf.address) is the empirical proof: the exact same memory region is exposed by both engines. This is what "zero-copy interop" actually means at the level of bytes.

Output.

Round-trip step	Result	Copy behaviour
Polars → PyArrow (to_arrow)	pyarrow.Table	zero-copy on flat cols
PyArrow → DuckDB (duckdb.arrow)	duckdb Relation	zero-copy view
DuckDB compute + project	pyarrow.Table	zero-copy for identity cols; new alloc for computed
PyArrow → Polars (from_arrow)	pl.DataFrame	zero-copy for compatible dtypes

Rule of thumb. For any workflow that combines Polars for pipeline transformations with DuckDB for ad-hoc SQL exploration, use df.to_arrow() as the handoff. Never .to_pandas() in the middle — that forces a full serialisation through NumPy-plus-Python-objects and defeats every zero-copy win. Round-trip via Arrow, never via Pandas.

Worked example — the string-column memory win vs Pandas object dtype

Detailed explanation. The single most dramatic Arrow2 memory win over Pandas is the string column. A Pandas object-dtype string column is a NumPy array of pointers to Python str objects; each str carries a ~50-byte PyObject header, an internal length, and a UTF-8 byte buffer. A Polars/Arrow2 string column is a single contiguous UTF-8 bytes buffer plus a small i32 offsets buffer. Walk through the difference on a realistic 100M-row column of short strings.

• Column. 100 million rows of event_type strings ("click", "purchase", "view", average ~7 chars each).
• Pandas layout. 100M × 8 bytes pointer + 100M × ~57 bytes PyObject header + ~700 MB of actual UTF-8 = ~7.2 GB.
• Polars/Arrow layout. ~700 MB UTF-8 buffer + 100M × 4 bytes offsets + ~12.5 MB validity bitmap = ~1.1 GB.

Question. Measure the memory footprint of the same 100M-row string column in Pandas vs Polars.

Input.

Component	Pandas	Polars/Arrow2
Values	Python str objects	UTF-8 bytes buffer
Per-row overhead	~57 B PyObject + 8 B pointer	4 B offset + 1 bit validity
Total memory (100M rows, ~7 chars)	~7.2 GB	~1.1 GB
Cache lines per scan	many (pointer chase)	few (linear)
SIMD-friendly	no	yes

Code.

import polars as pl
import pandas as pd
import pyarrow as pa
import psutil, os, gc, random

def rss_mb() -> float:
    return psutil.Process(os.getpid()).memory_info().rss / (1024 * 1024)

random.seed(0)
vocab = ["click", "purchase", "view", "add_to_cart", "signup"]
rows  = 100_000_000

# Pandas object-dtype baseline
gc.collect(); base = rss_mb()
pandas_col = pd.Series(random.choices(vocab, k=rows), dtype="object")
print(f"Pandas object-dtype:  {rss_mb() - base:>8.1f} MB after allocation")

del pandas_col; gc.collect()

# Polars string column (Arrow2 under the hood)
base = rss_mb()
polars_col = pl.Series(random.choices(vocab, k=rows))
print(f"Polars/Arrow string:  {rss_mb() - base:>8.1f} MB after allocation")

# Polars with dictionary/categorical encoding for the low-cardinality case
base = rss_mb()
polars_cat = pl.Series(random.choices(vocab, k=rows), dtype=pl.Categorical)
print(f"Polars categorical:   {rss_mb() - base:>8.1f} MB after allocation")

# Expected order-of-magnitude output:
# Pandas object-dtype:      7156.3 MB after allocation
# Polars/Arrow string:      1092.1 MB after allocation
# Polars categorical:        102.4 MB after allocation

Step-by-step explanation.

1. The Pandas object dtype stores each string as a full Python object. Even after CPython's small-string interning for the tiny vocabulary here, the 100M pointers still cost 800 MB, and the per-object headers add another 5+ GB. Total: ~7 GB.
2. The Polars string column stores the entire vocabulary once (interned into the values buffer) plus 100M i32 offsets into that buffer. The offsets buffer is 400 MB; the values buffer is ~700 MB. Total: ~1.1 GB, a ~7x reduction.
3. The Polars categorical (dictionary-encoded) column notices the low cardinality (5 distinct values) and stores each row as an 8-bit index into a dictionary of the 5 strings. Total: 100 MB, a ~72x reduction over Pandas.
4. Beyond raw memory, the Arrow layout is cache-friendly. A scan over the string column reads contiguous bytes; a Pandas scan chases pointers to scattered Python objects, blowing every L1/L2 cache line. This alone accounts for a large fraction of Polars' string-op speed advantage.
5. SIMD-wise, Arrow's contiguous UTF-8 buffer means operations like str.contains("click") can use SIMD instructions (SSE / AVX / NEON) that scan 16-32 bytes per cycle. Pandas' object-array cannot — the pointer chase makes SIMD impossible.

Output.

Layout	Memory (100M × ~7-char strings)	Scan speed
Pandas object dtype	~7,200 MB	slow (pointer chase)
Polars/Arrow string	~1,100 MB	fast (linear scan)
Polars categorical	~100 MB	fastest (dict index)

Rule of thumb. For any string column with fewer than ~1000 distinct values, use pl.Categorical — the memory reduction is 50x – 100x and every equality check becomes an integer comparison. For high-cardinality strings, plain Polars strings are already 5x – 10x smaller than Pandas objects. Never carry Pandas object dtype through a hot loop.

Worked example — chunk-based execution and the rechunk operation

Detailed explanation. A Polars DataFrame is not a single Arrow buffer per column — it's a Vec<Chunk> per column, where each chunk is a contiguous slice. This allows scans, concats, and parallel operations to work on independent chunks without copying. But some operations (like sorting or complex expressions) benefit from a single contiguous chunk, and Polars offers .rechunk() to collapse a multi-chunk column into one. Understanding when to rechunk — and when not to — is a senior-level detail.

• When chunks help. After a concat of many DataFrames, each source contributes its own chunks; downstream filters and aggregates run per chunk in parallel with no copy.
• When chunks hurt. Sorting is inherently global; the sort implementation must first rechunk (implicitly) to have a single contiguous buffer. Forcing rechunk explicitly upstream can avoid repeated rechunks.
• When to be explicit. After a heavy concat, if the next steps are sort-heavy, df.rechunk() once amortises the cost; if the next steps are scan-heavy, leave the chunks alone.

Question. Show the chunk count before and after common operations and demonstrate an explicit rechunk.

Input.

Operation	Effect on chunk count
pl.scan_parquet(...).collect()	one chunk per row group
pl.concat([df1, df2, df3])	chunk count = sum
.filter(...)	preserves chunks (may leave some empty)
.sort(...)	implicit rechunk to 1
.rechunk()	explicit rechunk to 1

Code.

import polars as pl

# 1. Concat 100 small DataFrames — the result has 100 chunks per column
parts = [pl.DataFrame({"x": [i, i + 1, i + 2]}) for i in range(100)]
df_multi = pl.concat(parts)

print("After concat:")
print("  n_chunks(x):", df_multi["x"].n_chunks())   # → 100

# 2. Filter preserves chunks (some may become empty)
df_filt = df_multi.filter(pl.col("x") > 50)
print("After filter:")
print("  n_chunks(x):", df_filt["x"].n_chunks())    # → 100 (some empty)
print("  height:    ", df_filt.height)

# 3. Rechunk explicitly for the next sort-heavy stage
df_rechunked = df_filt.rechunk()
print("After rechunk:")
print("  n_chunks(x):", df_rechunked["x"].n_chunks()) # → 1

# When to NOT rechunk — after a concat that feeds a parallel scan
import time

df_multi = pl.concat([
    pl.read_parquet(f"part_{i}.parquet")
    for i in range(64)
])

# 64 chunks means 64 units of parallelism for the next scan-heavy op
t0 = time.perf_counter()
result_no_rechunk = df_multi.filter(pl.col("region") == "EMEA").group_by("product").agg(pl.col("units").sum())
print(f"No rechunk: {time.perf_counter() - t0:.2f}s")

t0 = time.perf_counter()
df_re = df_multi.rechunk()   # collapses to 1 chunk
result_rechunk = df_re.filter(pl.col("region") == "EMEA").group_by("product").agg(pl.col("units").sum())
print(f"With rechunk: {time.perf_counter() - t0:.2f}s")

# On a 16-core box, expect:
# No rechunk:    1.8s   (parallelism preserved)
# With rechunk:  3.6s   (one huge copy + less parallelism)

Step-by-step explanation.

1. pl.concat(parts) preserves each source DataFrame's chunks by default (there's a rechunk=True kwarg to opt in). This is cheap — no data copying, just a Vec<Chunk> extended. After concatenating 100 3-row DataFrames, each column has 100 chunks.
2. .filter(pl.col("x") > 50) preserves chunk boundaries — each chunk is filtered independently (parallelisable) and some may end up empty. The chunk count stays at 100; the row count drops.
3. .rechunk() walks all chunks and copies their bytes into a single contiguous chunk per column. This is expensive (O(N) copies) but sometimes worth it before a sort-heavy operation that will rechunk implicitly anyway.
4. Counter-example: if the next operation is a scan-heavy parallel filter+groupby, rechunking first destroys parallelism (one chunk = one core for the scan). Leaving the 64 chunks intact gives 64 units of scan parallelism and finishes twice as fast.
5. The rule: rechunk only before sort-heavy or expression-heavy operations that would rechunk implicitly. Never rechunk before a scan or a group-by that benefits from per-chunk parallelism. When in doubt, .explain() and time both variants.

Output.

Situation	Rechunk?	Reason
After 100-way concat, next is a sort	yes	sort would rechunk implicitly
After 64-file scan, next is filter + groupby	no	preserve chunk parallelism
Right before a heavy expression (e.g. expr.entropy())	yes	some expressions want contiguous
At the end of a pipeline for downstream consumers	usually yes	callers expect single-chunk cols

Rule of thumb. Chunks are Polars' parallelism unit. Preserve them before scans, filters, and simple aggregations; collapse them (via .rechunk()) before sorts and complex expressions. Explicit .rechunk() is a performance tool, not a hygiene requirement.

Senior interview question on Arrow2 + interop

A senior interviewer might ask: "You have a Polars pipeline that produces a 5 GB result DataFrame, and a downstream service needs to run ad-hoc SQL on it (some analysts) and also feed it into a scikit-learn model (some data scientists). Walk me through the Arrow-based interop that lets you serve both without duplicating memory, the buffer-lifecycle pitfalls, and how you'd verify zero-copy is actually happening."

Solution Using to_arrow + DuckDB view + PyArrow → Pandas ExtensionArray bridge

import polars as pl
import pyarrow as pa
import duckdb

# 1. The Polars pipeline (compute the 5GB result once)
result = (
    pl.scan_parquet("events.parquet")
      .filter(pl.col("region") == "EMEA")
      .group_by(["user_id", "day"])
      .agg(
          events=pl.len(),
          revenue=pl.col("revenue_cents").sum(),
      )
      .collect(streaming=True)
)
print(f"Result: {result.estimated_size('mb'):.0f} MB, {result.height:,} rows")

# 2. Hand the SAME buffers to PyArrow — no copy
tbl = result.to_arrow()

# 3. Register with DuckDB — analysts run SQL against `tbl`
duckdb.register("emea_summary", tbl)

analyst_query = duckdb.query("""
    SELECT day,
           SUM(events)  AS total_events,
           SUM(revenue) AS total_revenue
    FROM   emea_summary
    GROUP  BY day
    ORDER  BY day
""").arrow()

# 4. Hand the SAME buffers to Pandas as ExtensionArrays — no copy for compatible dtypes
import pandas as pd

pdf_for_sklearn = tbl.to_pandas(types_mapper=pd.ArrowDtype)
# → all columns are pd.ArrowDtype (Pandas 2.x); underlying buffers still shared with Arrow

# 5. Verify zero-copy by comparing buffer addresses
polars_id_buf = result["user_id"].to_arrow().buffers()[1]
arrow_id_buf  = tbl.column("user_id").chunk(0).buffers()[1]
print("Polars ↔ PyArrow zero-copy:", polars_id_buf.address == arrow_id_buf.address)
# → True

# 6. Buffer lifecycle — ensure the source Polars df outlives all consumers
# The Arrow buffers are refcounted; when the Polars df is dropped, PyArrow / DuckDB
# still holds refs and the memory sticks around. This is safe but easy to forget.
import weakref
print("PyArrow buffer refcount (approx):", pa.total_allocated_bytes())

Step-by-step trace.

Step	Consumer	Memory behaviour
result from streaming collect	Polars owns 5 GB buffers	one copy
result.to_arrow()	PyArrow Table refs same buffers	zero-copy
duckdb.register("emea_summary", tbl)	DuckDB reads via Arrow C interface	zero-copy
DuckDB analyst query	new Arrow Table for aggregated result	small allocation
tbl.to_pandas(types_mapper=pd.ArrowDtype)	Pandas ExtensionArrays over same buffers	zero-copy
scikit-learn df.values	forces materialisation to NumPy	copy at boundary

After deployment, one 5 GB Polars result serves three consumers (SQL analysts, DataFusion pipeline, scikit-learn training) from a single memory footprint. The scikit-learn path forces one final NumPy copy when .values is called; every other consumer reads the same bytes.

Output:

Consumer	Wire format	Copy?
Polars → PyArrow	Arrow C data interface	zero-copy
PyArrow → DuckDB	Arrow C data interface	zero-copy
DuckDB SQL result → PyArrow	new Arrow Table	small copy
PyArrow → Pandas ExtensionArray	Arrow-backed Pandas	zero-copy
Pandas → NumPy (.values for sklearn)	NumPy dense array	one copy

Why this works — concept by concept:

• to_arrow() as the interop pivot — .to_arrow() hands PyArrow a view over the exact Polars/Arrow2 buffers. Every downstream consumer that speaks Arrow (DuckDB, DataFusion, R arrow, Go arrow) gets zero-copy access from this single pivot point.
• DuckDB register + Arrow C interface — DuckDB's Arrow ingestion is native; duckdb.register binds a PyArrow Table as a virtual table without any per-row conversion. SQL runs directly on the shared buffers.
• PyArrow → Pandas ExtensionArray — Pandas 2.x supports pd.ArrowDtype which wraps Arrow buffers in a NumPy-compatible ExtensionArray. Downstream Pandas code sees a normal DataFrame; the buffers stay shared until someone forces .values.
• Buffer lifecycle — Arrow buffers are refcounted. Dropping the original Polars DataFrame is safe as long as PyArrow / DuckDB / Pandas still hold refs; the memory persists until every consumer releases it. The only pitfall is forgetting this and expecting del df to free memory.
• Cost — one 5 GB allocation total across five consumers (vs 25 GB if each did its own copy). Zero-copy interop is only possible because Arrow is a shared columnar contract, not a serialisation format. This is why "Arrow is the lingua franca of columnar data" is not marketing — it's the byte-level ABI that makes multi-engine architectures affordable.

Joins
Topic — joins
Join problems for columnar-execution drills

Practice →

Design Topic — design Design problems on columnar interop systems

Practice →

---

4. Native parallelism + streaming engine

Rayon work-stealing + morsel-driven streaming means one laptop punches above its weight — 100 GB on a 16 GB box, all cores, no GIL

The mental model in one line: polars parallelism uses Rust's Rayon work-stealing thread pool to schedule every parallelisable operator (scan, filter, group_by, join, sort) across every physical core simultaneously — no GIL to contend for because Polars' compute is Rust-native — and the streaming engine (.collect(streaming=True)) processes data as ~10 MB morsels through the physical plan, spilling to disk when the working set exceeds RAM, which turns "aggregate 100 GB of Parquet on a 16 GB laptop" from a fantasy into a Tuesday-morning task. Every senior data engineer who runs Polars in production knows the two flags (.collect(streaming=True) for out-of-core, pl.thread_pool_size() for parallelism budgeting) and reaches for them without hesitation.

The Rayon work-stealing model — why it beats Pandas' single-core execution.

• Rayon in one sentence. A Rust library implementing a work-stealing thread pool where each worker thread has its own local task queue; when a worker's queue is empty it steals from a random other worker. This is the same design as Java's ForkJoinPool and TBB.
• Why work-stealing wins. Load imbalance is automatic — a slow chunk on one worker doesn't starve the pipeline because idle workers steal fast chunks from busy ones. Compare to a naive "chunk N to worker N" partitioning that stalls at the slowest chunk.
• No GIL — the Rust win. Every parallel operator runs in Rust threads, not Python threads. The Python GIL only exists in the Python interpreter; once you're inside pl.LazyFrame.collect(), control has crossed into Rust and the GIL is released. All 16 cores run at 100% simultaneously.
• What's parallel. Scans (per row group in Parquet, per chunk in memory), filters (per chunk), group_by (hash-partition across cores), sorts (parallel sort with a final merge), joins (parallel hash-partitioning of both sides), most row-wise expressions.
• What's serial. Some sinks (writing a single Parquet file involves serial commit), some window functions with global ordering, Python UDFs (the GIL is re-acquired for each call).

The morsel-driven streaming engine — the out-of-core superpower.

• What a morsel is. A batch of rows (default ~10 MB, tunable) that flows through the physical plan pipeline. Each operator in the plan (scan → filter → group_by → sink) processes morsels; the working set is bounded by the sum of in-flight morsels rather than the total data size.
• .collect(streaming=True). Switches the physical plan from the in-memory engine (all rows materialised before the next stage) to the streaming engine (morsels flow through). Not all operators support streaming yet (as of Polars 1.x, ~85% coverage); check explain(streaming=True) to see which parts run streaming vs which fall back to in-memory.
• Spill to disk. When a streaming operator (e.g. group_by on a wide key space) exceeds its memory budget, it spills partial hash tables to disk. On disk-heavy queries this is slower than in-memory, but "slow" beats "OOM crash".
• sink_parquet / sink_csv / sink_ipc. True streaming write — the input is scanned as morsels, transformed, and written directly without ever materialising the full result. This is how you process 500 GB of Parquet on a modest server.

Thread-pool sizing and CPU affinity.

• pl.thread_pool_size(). Reports the current Rayon pool size (default = number of physical cores). Setting POLARS_MAX_THREADS=N env var caps it — useful in shared-tenancy environments.
• CPU affinity. By default Polars uses every core. In containerised deploys (Kubernetes with CPU limits), set POLARS_MAX_THREADS to match your CPU request so Polars doesn't over-schedule.
• NUMA awareness. Polars is not explicitly NUMA-aware; on 2-socket servers with large NUMA-remote memory penalties, pinning the process to one socket can beat the default.

Polars-GPU — the 2024 NVIDIA collaboration.

• What it is. An opt-in GPU backend that runs the Polars physical plan on NVIDIA GPUs via the cuDF library (RAPIDS). Announced October 2024; production-ready for a growing subset of operators in 2026.
• When it wins. Multi-billion-row aggregations, joins on GPU-friendly key sizes, workloads where memory transfer cost is amortised over enough compute.
• When it loses. Small data (transfer cost dominates), string-heavy workloads (GPU string ops are less mature), any query with unsupported operators (falls back to CPU with a transfer round-trip).
• How to enable. pl.Config.set_engine("gpu") or .collect(engine="gpu"). Requires polars[gpu] install and a compatible NVIDIA driver.

Common interview probes on parallelism + streaming.

• "How does Polars use all cores?" — required answer: Rayon work-stealing thread pool over Rust threads (no GIL).
• "When would you use streaming=True?" — required answer: dataset larger than RAM, or when you want bounded memory.
• "What's a morsel?" — required answer: a ~10 MB batch that flows through the physical plan.
• "How do you cap Polars' CPU usage?" — required answer: POLARS_MAX_THREADS env var.
• "Polars-GPU tradeoffs?" — senior signal: transfer cost vs compute gain; not universally faster.

Worked example — parallel scan + group_by on a wide Parquet

Detailed explanation. The canonical Polars-parallelism demo: read a 20 GB Parquet with 500M rows, filter to ~10% of rows, group by a 100-cardinality key, and aggregate. On a 16-core laptop, expect end-to-end wall-clock in single-digit seconds. Walk through the parallelism story stage by stage.

• Scan parallelism. Parquet row groups are the natural chunk — one row group per Rayon task, all cores scan simultaneously.
• Filter parallelism. Each row group's filter runs on its own core; results accumulate per-core.
• Group-by parallelism. Hash-partition the keys across cores; each core aggregates its partition independently; final concat is trivial.
• Sort parallelism. Small result (100 groups × ~10 aggregates) — sort is negligible.

Question. Run the pipeline, measure per-stage time, and prove all 16 cores were utilised.

Input.

Component	Value
File	events.parquet (20 GB, 500M rows, 40 cols)
Filter	10% selective
Group key	event_type (~100 distinct values)
Aggregate	count, unique users, revenue sum
Machine	16 physical cores, 64 GB RAM

Code.

import polars as pl
import time
import os

# Confirm thread-pool size
print(f"Rayon threads: {pl.thread_pool_size()}")   # → 16

# The pipeline — every stage parallelisable
lf = (
    pl.scan_parquet("events.parquet")
      .filter(pl.col("region") == "EMEA")
      .group_by("event_type")
      .agg(
          n_events=pl.len(),
          n_users=pl.col("user_id").n_unique(),
          revenue=pl.col("revenue_cents").sum(),
      )
      .sort("revenue", descending=True)
)

# Warm the filesystem cache with a tiny scan first (avoids I/O dominating the first run)
_ = pl.scan_parquet("events.parquet").head(1).collect()

# Time the full pipeline
t0 = time.perf_counter()
df = lf.collect()
elapsed = time.perf_counter() - t0

print(f"Wall-clock:  {elapsed:.2f} s")
print(f"Rows out:    {df.height:,}")
print(f"Peak memory (approx): {df.estimated_size('mb'):.0f} MB result")

# Prove all cores were utilised — monitor while the above runs
# In a second terminal:
top -pid $(pgrep -f 'python.*run.py') -stats cpu

# Expect: ~1500-1600% CPU (16 cores at ~100%)

# Compare to a single-thread run for the parallelism ratio
os.environ["POLARS_MAX_THREADS"] = "1"
# Restart the interpreter for the env var to take effect

t0 = time.perf_counter()
df_1t = lf.collect()
elapsed_1t = time.perf_counter() - t0
print(f"1 thread:  {elapsed_1t:.2f} s")
print(f"16 threads: {elapsed:.2f} s")
print(f"Speed-up:   {elapsed_1t / elapsed:.1f}x")
# → 1 thread:   112.4 s
# → 16 threads:   9.8 s
# → Speed-up:    11.5x   (typical: 10x–14x on 16 physical cores; not linear due to I/O + sync)

Step-by-step explanation.

1. pl.thread_pool_size() confirms Polars sees all 16 cores. On a container with a CPU limit, this reports the effective limit — always check before benchmarking. POLARS_MAX_THREADS overrides the default.
2. The scan_parquet node dispatches one Rayon task per row group. A 20 GB Parquet typically has 100-200 row groups; each task decompresses + decodes one row group on its own core. All 16 cores are saturated during the I/O + decode phase.
3. The filter runs inside each scan task (predicate pushdown into the scan) — no separate parallel step, just an inline check on each decoded row. Selective filters mean ~10% of rows survive.
4. The group_by hash-partitions the surviving rows across cores by hash of event_type. Each core builds its own hash table over its partition; the final union is cheap because there are only ~100 distinct keys.
5. The wall-clock speed-up (~11.5x on 16 cores) is a bit below the theoretical 16x because I/O bandwidth becomes the bottleneck at high parallelism, and there's a small sync cost for hash-partitioning. Getting > 10x parallel speed-up over single-thread is the Polars norm on scan-heavy workloads.

Output.

Stage	Parallelism	Effect
Parquet scan	one task per row group	16x cores utilised
Filter (pushed into scan)	inline per row	~10% row survival
group_by hash-partition	one partition per core	16x hash agg
Sort (100 rows result)	trivial	not a bottleneck
Total speed-up over 1 thread	~11.5x	I/O + sync limits linear scaling

Rule of thumb. For any scan-heavy Polars workload, expect 10x – 14x parallel speed-up on 16 cores relative to single-thread. If you see less than 8x, you probably have a Python UDF (single-core), a wildcard blocking projection pushdown, or a mis-sized POLARS_MAX_THREADS. top / htop while the job runs is the quickest diagnostic.

Worked example — streaming collect on a 100 GB Parquet with a 16 GB laptop

Detailed explanation. The single most impressive Polars trick is aggregating a 100 GB Parquet on a laptop with 16 GB of RAM. The streaming engine processes data as ~10 MB morsels through the physical plan; the working set never exceeds a few hundred MB even though the source is 100 GB. Walk through the pipeline and prove memory stays flat.

• Source. huge.parquet — 100 GB, 5B rows, 20 columns.
• Target. aggregated result: ~1000 rows.
• Machine. 16 GB laptop, 8 physical cores, SSD.
• Trick. .collect(streaming=True).

Question. Aggregate the 100 GB source into a small result on a memory-constrained laptop and prove the working set stays bounded.

Input.

Component	Value
Source file	100 GB, 5B rows, 20 cols
Filter	date >= 2026-01-01 (~20% of rows)
Group	by category (~500 distinct)
Aggregate	count, sum, avg
Available RAM	16 GB total, ~12 GB usable

Code.

import polars as pl
import psutil, os, time

def rss_gb() -> float:
    return psutil.Process(os.getpid()).memory_info().rss / (1024 ** 3)

# The pipeline — same shape as in-memory, but with streaming=True
lf = (
    pl.scan_parquet("huge.parquet")
      .filter(pl.col("date") >= pl.date(2026, 1, 1))
      .group_by("category")
      .agg(
          n=pl.len(),
          total_revenue=pl.col("revenue_cents").sum(),
          avg_revenue=pl.col("revenue_cents").mean(),
      )
      .sort("total_revenue", descending=True)
)

# Verify the plan is streaming-friendly
print(lf.explain(streaming=True))
# Look for STREAMING at the top; if you see PARTITIONED_SINK or NON-STREAMING,
# one of the operators falls back to in-memory

# Peak memory tracker running in a background thread
import threading
peak_mem = [0.0]
stop_flag = [False]

def track_memory():
    while not stop_flag[0]:
        peak_mem[0] = max(peak_mem[0], rss_gb())
        time.sleep(0.1)

t = threading.Thread(target=track_memory, daemon=True)
t.start()

# Execute with streaming
t0 = time.perf_counter()
df = lf.collect(streaming=True)
elapsed = time.perf_counter() - t0

stop_flag[0] = True
t.join()

print(f"Wall-clock:      {elapsed:.1f} s")
print(f"Result rows:     {df.height}")
print(f"Peak RSS memory: {peak_mem[0]:.1f} GB")
# → Wall-clock:      ~380 s
# → Result rows:     500
# → Peak RSS memory: ~2.4 GB   (never touches the 12 GB ceiling)

# The sink_parquet variant — never materialises the result in Python
(
    pl.scan_parquet("huge.parquet")
      .filter(pl.col("date") >= pl.date(2026, 1, 1))
      .select("category", "revenue_cents", "date")
      .sink_parquet("filtered.parquet")   # streaming write; bounded memory
)
# → Wall-clock: ~410 s
# → Peak RSS: ~1.8 GB
# → Output file: ~18 GB (post-filter)

Step-by-step explanation.

1. .explain(streaming=True) reports whether each operator can run in the streaming engine. For this pipeline (scan + filter + group_by + sort), all operators are streaming-capable in Polars 1.x, so the whole plan runs streaming.
2. The scan reads Parquet row groups one at a time and emits morsels of ~10 MB each. Each morsel flows through the filter (drops non-matching rows), into the group_by (updates per-partition hash tables), and eventually into the sort at the end. The working set is dominated by the group_by hash tables (~500 categories × per-core partitions).
3. Peak memory stays under 3 GB throughout the ~6 minutes of execution. The 100 GB source never touches RAM in aggregate — only ~10 MB is in-flight at any moment per morsel, times the number of Rayon workers.
4. The sort at the end operates on the aggregated result (~500 rows) — trivial, sub-millisecond. If the sort had to operate on 5B rows, it would spill to disk (order-of-magnitude slower but still bounded memory).
5. sink_parquet is even better: it streams both input and output. Useful for "filter this 100 GB file down to a 20 GB file" workloads where you don't need the result in Python — the Parquet output is written directly to disk without materialising in RAM.

Output.

Metric	Value
Source size	100 GB
Peak RSS during collect	~2.4 GB
Wall-clock (aggregate)	~6 min 20 s
Wall-clock (sink_parquet)	~6 min 50 s
Result size	~30 KB (500 rows)
OOM risk	zero (streaming bounded)

Rule of thumb. For any input larger than half your available RAM, use .collect(streaming=True) (or sink_parquet / sink_csv for streaming output). Verify with .explain(streaming=True) that no operator falls back to in-memory. Streaming is slightly slower per-unit than in-memory (5-15% overhead) but the OOM protection is worth it.

Worked example — Polars-GPU on a multi-billion-row aggregation

Detailed explanation. The Polars-GPU backend (opt-in via .collect(engine="gpu")) runs the physical plan on an NVIDIA GPU via cuDF. It's a decisive win for GPU-friendly workloads at multi-billion-row scale on one node with a beefy GPU, and a decisive loss for tiny workloads or unsupported operators (transfer cost dominates). Walk through the tradeoff.

• Workload. 3B rows, group by (300 keys), sum + mean of a numeric column, no strings.
• GPU. NVIDIA A100 80 GB.
• CPU baseline. 32 physical cores, 512 GB RAM.
• Expectation. GPU beats CPU by 3x – 5x on this shape; loses on string-heavy or small-data workloads.

Question. Compare CPU-parallel vs GPU-parallel execution on the same query and identify when each wins.

Input.

Component	CPU (Rayon)	GPU (cuDF)
Compute unit	32 cores	~7000 CUDA cores
Memory	512 GB DDR5	80 GB HBM
Transfer cost	zero (already in RAM)	~15 GB from RAM to GPU
Best for	strings, unsupported ops	numeric aggregations, joins
Fallback	(n/a)	falls back to CPU on unsupported

Code.

import polars as pl
import time

lf = (
    pl.scan_parquet("big_numeric.parquet")   # 3B rows, 5 cols, all numeric
      .group_by("bucket_id")
      .agg(
          n=pl.len(),
          sum_val=pl.col("value").sum(),
          avg_val=pl.col("value").mean(),
          max_val=pl.col("value").max(),
      )
)

# CPU baseline (Rayon parallel)
t0 = time.perf_counter()
df_cpu = lf.collect(engine="cpu")
print(f"CPU (32 cores):  {time.perf_counter() - t0:.1f} s")

# GPU (requires polars[gpu] + NVIDIA driver + compatible GPU)
t0 = time.perf_counter()
df_gpu = lf.collect(engine="gpu")
print(f"GPU (A100):      {time.perf_counter() - t0:.1f} s")

# Expected order-of-magnitude output on this workload:
# CPU (32 cores):  22.4 s
# GPU (A100):       5.1 s
# Speed-up:          4.4x

# Counter-example — string-heavy workload where GPU loses
lf_strings = (
    pl.scan_parquet("logs.parquet")   # 500M rows, mostly string columns
      .filter(pl.col("message").str.contains("ERROR"))
      .group_by("service_name")
      .agg(pl.len())
)

t0 = time.perf_counter()
df_cpu2 = lf_strings.collect(engine="cpu")
print(f"CPU:  {time.perf_counter() - t0:.1f} s")

t0 = time.perf_counter()
df_gpu2 = lf_strings.collect(engine="gpu")   # falls back to CPU for string ops
print(f"GPU: {time.perf_counter() - t0:.1f} s (with CPU fallback)")

# Expected:
# CPU:  18.7 s
# GPU:  24.2 s   (worse because of transfer cost + CPU fallback)

Step-by-step explanation.

1. lf.collect(engine="gpu") requests GPU execution. Polars checks whether every operator in the physical plan has a GPU implementation; if any doesn't, that stage falls back to CPU with a transfer round-trip (RAM → GPU → RAM). The 2026 coverage is roughly: numeric aggregations yes, joins on primitive keys yes, string operations mostly no, complex expressions varies.
2. On the numeric aggregation, the A100's ~7000 CUDA cores demolish the CPU's 32 cores for the hash-partition and reduce phases. The transfer cost (~15 GB from RAM to GPU) is amortised over 3B rows of compute — negligible.
3. On the string-heavy workload, the string operations must fall back to CPU (GPU string coverage is limited), forcing a GPU → RAM transfer, CPU execution, then possibly RAM → GPU again if the next stage is GPU-capable. Net: slower than pure CPU.
4. The Polars-GPU decision matrix: numeric aggregations at ≥ 100M rows on one node with a decent GPU → GPU wins by 3x – 10x. String-heavy or small (< 100M) → stick with CPU. Mixed workloads → benchmark both explicitly per query.
5. The operational cost: polars[gpu] requires an NVIDIA GPU with recent drivers and consumes GPU memory that other processes might want. In shared-tenancy notebooks this is a config nightmare; in dedicated batch jobs it's fine.

Output.

Workload	CPU (32 cores)	GPU (A100)	Winner
3B rows, numeric group-by	22.4 s	5.1 s	GPU 4.4x
500M rows, string filter + group	18.7 s	24.2 s	CPU
10M rows, any op	~1 s	~2 s (transfer)	CPU
Complex Python UDF chain	45 s	45 s (falls back)	tie

Rule of thumb. Benchmark GPU explicitly per workload — do not assume Polars-GPU is always faster. For numeric aggregations on ≥ 100M rows on a machine with an NVIDIA GPU, opt in via engine="gpu". For string-heavy, small-data, or Python-UDF-heavy workloads, stay on CPU. Check the operator coverage in the Polars docs before committing.

Senior interview question on parallelism + streaming

A senior interviewer might ask: "You have a 250 GB daily Parquet ingest, a 32 GB server, and a nightly ETL that filters, joins with a 10 GB reference table, aggregates by hour, and writes a partitioned Parquet output. Walk me through the Polars design that keeps memory bounded, exploits every CPU core, and stages the streaming pipeline correctly — including how you'd handle the case where the streaming engine falls back to in-memory for the join."

Solution Using scan + streaming filter + broadcast join + sink_parquet

import polars as pl

# 1. Reference table — 10 GB fits in memory, materialise once
ref = (
    pl.scan_parquet("reference.parquet")
      .select("product_id", "product_name", "category", "tax_rate")
      .collect()                                # eager; ~10 GB in RAM
)

# 2. Streaming pipeline — main ingest never fully materialises
main_lf = (
    pl.scan_parquet("ingest_2026-07-21/*.parquet")     # 250 GB across many files
      .filter(pl.col("event_ts") >= pl.datetime(2026, 7, 21))
      .filter(pl.col("event_ts") <  pl.datetime(2026, 7, 22))
      .select(
          "event_ts", "product_id", "user_id",
          "units", "revenue_cents",
      )
)

# 3. Broadcast-join the reference table
#    (ref is small; join runs streaming with broadcast side)
joined = (
    main_lf
      .join(ref.lazy(), on="product_id", how="left")
      .with_columns(
          hour=pl.col("event_ts").dt.truncate("1h"),
          revenue_with_tax=pl.col("revenue_cents") * (1 + pl.col("tax_rate")),
      )
)

# 4. Aggregate by hour + category — streaming group_by
agg = (
    joined
      .group_by(["hour", "category"])
      .agg(
          n_events=pl.len(),
          n_users=pl.col("user_id").n_unique(),
          units=pl.col("units").sum(),
          revenue=pl.col("revenue_with_tax").sum(),
      )
)

# 5. Verify streaming coverage
print(agg.explain(streaming=True))
# Look for STREAMING on scan, filter, group_by; join may fall back if too large

# 6. Sink to partitioned Parquet — streaming write
agg.sink_parquet(
    "hourly_agg_2026-07-21.parquet",
    row_group_size=100_000,
    compression="zstd",
)

# Monitoring wrapper — bound memory + timeout
import psutil, os, threading, time

MAX_RSS_GB = 24.0   # leave 8 GB headroom on a 32 GB box

def watchdog(pid):
    p = psutil.Process(pid)
    while p.is_running():
        rss_gb = p.memory_info().rss / (1024 ** 3)
        if rss_gb > MAX_RSS_GB:
            print(f"ALERT: RSS {rss_gb:.1f} GB exceeds {MAX_RSS_GB} GB budget")
            os._exit(2)
        time.sleep(2)

threading.Thread(target=watchdog, args=(os.getpid(),), daemon=True).start()

Step-by-step trace.

Layer	Component	Streaming?	Notes
Reference load	pl.scan_parquet(ref).collect()	eager	10 GB fits; materialise once
Main scan	scan_parquet(ingest/*)	streaming	250 GB never in RAM
Filter (date)	pushed into scan	streaming	row-group skip
Projection	5 of 30 columns	streaming	6x less I/O
Join	left join on product_id	streaming (broadcast)	ref is broadcast to every worker
Group by hour+category	hash-partition	streaming	bounded hash tables
Sink	sink_parquet	streaming write	0 result-set materialisation
Watchdog	RSS check every 2s	(external)	kills process at 24 GB

After the run, the 250 GB daily ingest lands as a partitioned Parquet output in ~45 minutes on the 32 GB server, with peak RSS staying under 20 GB throughout. The reference table's 10 GB is a fixed baseline; the streaming pipeline adds ~8 GB of morsels + hash tables on top; the watchdog never fires.

Output:

Metric	Value
Source	250 GB across many Parquet files
Reference table	10 GB in RAM (eager)
Peak RSS	~19 GB (under 24 GB budget)
Wall-clock	~45 min
Output	~4 GB partitioned Parquet
CPU utilisation	~2900% (32 cores * ~90%)
OOM incidents	zero

Why this works — concept by concept:

• scan_parquet + streaming — the 250 GB never fully materialises. pl.scan_parquet("ingest/*") returns one LazyFrame over all files; .collect(streaming=True) (or sink_parquet) processes them as morsels through the physical plan.
• Broadcast join — because the reference table (10 GB) is much smaller than the main stream (250 GB), Polars can broadcast it to every worker's local memory rather than hash-partitioning both sides. This makes the join streaming-safe.
• Predicate + projection pushdown — date filter and 5-of-30 column projection push into every Parquet scan; only ~50 GB of the 250 GB actually gets decoded.
• Streaming group_by with bounded hash tables — hash-partitioning across 32 cores means each core's hash table is 1/32 of the total; with ~1000 distinct (hour, category) keys, each table is trivially small. If it grew unbounded, spill to disk would kick in.
• sink_parquet — the final aggregation never materialises in Python. Rows flow from group_by → Parquet writer directly. This is the difference between a nightly ETL that runs in 45 minutes and one that OOMs at hour 3. O(streaming morsel size) memory for O(N) throughput — the ideal ratio.

Joins
Topic — joins
Join problems for parallel-hash-join drills

Practice →

Design Topic — design Design problems on streaming ETL pipelines

Practice →

---

5. Polars vs Pandas / DuckDB / Spark + interview signals

polars vs pandas is the wrong framing — the right question is "which tool for which data size and interface style"

The mental model in one line: Polars, Pandas, DuckDB, and Spark are not competitors so much as four points on a two-axis grid (data size × interface style), and the senior data-engineering answer is always "pick per workload, not per personal preference" — Pandas for tiny data + rich ecosystem interop, Polars for 100 MB – 100 GB tabular pipelines on one machine, DuckDB for SQL-first interactive analytics, Spark for > 1 TB distributed processing — and the interview signal is whether you can defend that boundary with concrete numbers rather than tribal loyalty. Every senior data engineer has migrated a codebase across at least two of these tools; understanding why each migration happened is the interview competency being probed.

The four-tool comparison matrix — the whiteboard artifact.

• Pandas. Eager, single-core, NumPy + Python objects, no optimizer. Sweet spot: 0 – 1 GB tabular data in a Python program with heavy ecosystem interop (statsmodels, scikit-learn, matplotlib, Jupyter). Loses at speed and scale; wins on ecosystem and inertia.
• Polars. Lazy option, all-core, Arrow2, real optimizer, streaming. Sweet spot: 100 MB – 100 GB tabular data on one machine, program-embedded, throughput-sensitive. Loses to Spark at cluster scale; loses to DuckDB for pure SQL interactive workloads.
• DuckDB. SQL-first analytical engine, columnar, vectorised, single-machine, embedded. Sweet spot: 1 GB – 500 GB tabular data queried via SQL, interactive analytics, "attach to any Parquet folder and query." Loses to Polars for program-embedded pipelines; loses to Spark at cluster scale.
• Spark. Distributed lazy DataFrame + SQL engine on JVM (or via PySpark/Photon). Sweet spot: > 1 TB tabular data on a cluster, complex ETL with cluster autoscaling and fault tolerance, Delta Lake / Iceberg first-class. Loses to all three at single-machine scale (JVM overhead, cluster startup cost).

The "which tool" senior answer template.

• Interviewer. "We have a 20 GB Parquet daily ingest that Pandas takes 45 minutes to process — should we move to Spark?"
• Weak answer. "Sure, Spark scales horizontally." (Recommends Spark for a single-machine workload.)
• Senior answer. "20 GB on one machine is Polars territory — Spark would be over-provisioning. Migrate to pl.scan_parquet + LazyFrame + .collect(streaming=True); expect 30-second wall-clock and bounded memory. Only move to Spark if the daily ingest grows past ~1 TB or the pipeline needs to run across a cluster for other reasons (fault tolerance, multi-tenant scheduling)."

The Modin / Pandas 2.x / PyArrow-backed Pandas alternative.

• Modin. A drop-in Pandas API backed by Ray or Dask. Softens the migration cost; wins on "I have 10,000 lines of Pandas code and can't rewrite them all." Loses to Polars on peak throughput (Modin can't apply an optimizer to Pandas code).
• Pandas 2.x with PyArrow backend. pd.read_parquet("...", dtype_backend="pyarrow") gives Pandas Arrow-backed columns. Fixes the string-column memory disaster; doesn't fix the eager execution or single-core problems. A stopgap, not a solution.
• cuDF. GPU DataFrame from RAPIDS. Wins on very large numeric workloads on a beefy GPU; niche outside NVIDIA-heavy environments.

Interview signals across the four tools.

• Do you name the size boundaries (100 MB / 100 GB / 1 TB) as decision inputs, not vague adjectives? — required.
• Do you distinguish "program-embedded" (Polars/Pandas) from "attach a SQL client" (DuckDB) as interface-style axes? — senior signal.
• Do you say "Spark for > 1 TB or cluster requirements, not for speed on one machine"? — required.
• Do you name Polars for the 100 MB – 100 GB sweet spot rather than defaulting to Pandas out of habit? — required in 2026.
• Do you note the migration cost between tools (Pandas → Polars is fast; Pandas → Spark is slow; Polars ↔ DuckDB is trivial via Arrow)? — senior signal.

Worked example — the 20 GB migration decision

Detailed explanation. The most common Polars-adoption interview scenario: an existing Pandas pipeline processing a medium-size daily ingest, and the interviewer asks "what would you migrate it to". Walk through the analysis and defend the choice with concrete numbers.

• Current state. 20 GB daily Parquet, Pandas pipeline, 45 min wall-clock, occasional OOM on the 64 GB laptop it runs on.
• Candidates. Polars (single-machine, faster), DuckDB (SQL-first, embedded), Spark (distributed).
• Decision. Polars — the data is not distributed-scale; the pipeline is program-embedded (not SQL); the throughput win is 30-100x; the migration cost is low.

Question. Present the decision analysis a senior architect would give in a design review.

Input.

Axis	Value	Implication
Data size	20 GB	Fits on single-machine tools
Ingest cadence	Daily	Not real-time; batch OK
Interface	Python program (Airflow task)	Program-embedded, not SQL
Team	3 data engineers	Small; can adopt new tool
Downstream	Snowflake dashboards	Any tool can write Parquet
Constraint	Budget prohibits Spark cluster	Single-machine preferred

Code.

# The migration diff — Pandas to Polars in ~15 lines
# BEFORE (Pandas)
import pandas as pd

def daily_pipeline(path: str) -> pd.DataFrame:
    df = pd.read_parquet(path)                     # 20 GB into RAM
    df = df[df["status"] == "confirmed"]
    df["revenue"] = df["units"] * df["price_cents"]
    result = (
        df.groupby(["region", "product_id"], as_index=False)
          .agg(units=("units", "sum"),
               revenue=("revenue", "sum"))
    )
    return result.sort_values("revenue", ascending=False)


# AFTER (Polars)
import polars as pl

def daily_pipeline(path: str) -> pl.DataFrame:
    return (
        pl.scan_parquet(path)
          .filter(pl.col("status") == "confirmed")
          .with_columns(revenue=pl.col("units") * pl.col("price_cents"))
          .group_by(["region", "product_id"])
          .agg(
              units=pl.col("units").sum(),
              revenue=pl.col("revenue").sum(),
          )
          .sort("revenue", descending=True)
          .collect(streaming=True)
    )

# The decision table — what convinces the review board
DECISION_TABLE = """
Option           Wall-clock   Peak RAM   Migration cost   Ongoing cost
---------------  -----------  ---------  ---------------  ------------
Pandas (today)   45 min       ~55 GB     0                laptop OOMs
Polars           ~30 s        ~4 GB      ~1 sprint        laptop
DuckDB           ~40 s        ~6 GB      ~1 sprint        laptop
Spark local      ~5 min       ~12 GB     ~2 sprints       laptop (weird)
Spark cluster    ~2 min       n/a        ~4 sprints       cluster $$
"""
print(DECISION_TABLE)

Step-by-step explanation.

1. Data size (20 GB) is well below the Spark break-even point. Distributed processing overhead (task serialisation, network shuffle, cluster management) dominates the compute for anything under ~1 TB on one machine.
2. Interface style is program-embedded (Airflow task, not a SQL client). DuckDB would work but wraps the entire pipeline in SQL; the team's existing Python skills favour Polars.
3. Migration cost: the Pandas → Polars diff is ~15 lines for this pipeline — a single afternoon of work per pipeline. Pandas → Spark would require rewriting to PySpark DataFrame semantics (subtly different from Pandas) and provisioning a cluster.
4. Ongoing cost: Polars runs on the existing laptop. Spark cluster would require provisioning EMR / Databricks / self-hosted YARN. The Polars TCO is roughly $0 above what's already spent.
5. Result: Polars wins on every axis except "we already have a Spark cluster and would prefer to consolidate." In the absence of that constraint, Polars is the clear recommendation. The senior signal is drawing this table before committing to a choice.

Output.

Option	Wall-clock	Peak RAM	Migration	Verdict
Pandas (today)	45 min	55 GB	0	OOMs; unacceptable
Polars	30 s	4 GB	1 sprint	recommended
DuckDB	40 s	6 GB	1 sprint	valid alternative
Spark local	5 min	12 GB	2 sprints	JVM overhead not worth it
Spark cluster	2 min	(cluster)	4 sprints	over-provisioned for 20 GB

Rule of thumb. For any single-machine tabular workload in the 100 MB – 100 GB range, Polars is the default choice. Deviate only when a hard constraint pushes you elsewhere (existing SQL client → DuckDB; > 1 TB or cluster requirements → Spark; < 10 MB with heavy scikit-learn interop → Pandas). Never migrate to Spark for speed on a single machine.

Worked example — Polars ↔ DuckDB workflow patterns

Detailed explanation. The most useful "two tools" pattern in 2026 is Polars + DuckDB — Polars for the programmatic pipeline (transforms, feature engineering, model I/O) and DuckDB for the ad-hoc SQL that analysts want to run against the intermediate results. Because both speak Arrow natively, the handoff is zero-copy. Walk through the pattern.

• Direction 1. Polars transforms → DuckDB SQL for analyst exploration.
• Direction 2. DuckDB SQL for complex analytical joins → Polars for downstream ML feature engineering.
• Handoff. Always via df.to_arrow() / duckdb.arrow(...) — never via Pandas.

Question. Show the two-tool pattern where Polars does the ETL, DuckDB does the analyst-facing SQL, and both share buffers.

Input.

Step	Tool	Reason
Ingest 50 GB Parquet	Polars	streaming + parallel
Filter + aggregate	Polars	lazy + optimizer
Ad-hoc SQL on result	DuckDB	analyst-friendly
Feature engineering	Polars	expression API
Model training input	Pandas (bridge)	scikit-learn expects DataFrame

Code.

import polars as pl
import duckdb
import pandas as pd

# 1. Polars — the heavy ETL
transactions = (
    pl.scan_parquet("transactions_*.parquet")            # 50 GB
      .filter(pl.col("status") == "settled")
      .group_by(["user_id", "day"])
      .agg(
          n_txns=pl.len(),
          total=pl.col("amount_cents").sum(),
          categories=pl.col("category").unique(),
      )
      .collect(streaming=True)                            # ~2M rows result
)
print(f"Aggregated: {transactions.height:,} rows, "
      f"{transactions.estimated_size('mb'):.0f} MB")

# 2. Hand to DuckDB for analyst-facing SQL (zero-copy via Arrow)
duckdb.register("user_daily", transactions.to_arrow())

# 3. Analyst runs any SQL they want
top_spenders = duckdb.query("""
    WITH ranked AS (
        SELECT user_id,
               SUM(total)                  AS lifetime_spend,
               COUNT(DISTINCT day)         AS active_days,
               AVG(n_txns)                 AS avg_txns_per_day
        FROM   user_daily
        GROUP  BY user_id
    )
    SELECT *,
           ROW_NUMBER() OVER (ORDER BY lifetime_spend DESC) AS rank
    FROM   ranked
    QUALIFY rank <= 1000
""").arrow()   # returns pyarrow.Table

# 4. Back to Polars for feature engineering
features = (
    pl.from_arrow(top_spenders)
      .with_columns(
          spend_per_day=pl.col("lifetime_spend") / pl.col("active_days"),
          is_high_freq=pl.col("avg_txns_per_day") > 3.0,
      )
)

# 5. Bridge to Pandas for scikit-learn
X = features.select(["spend_per_day", "avg_txns_per_day", "active_days"]).to_pandas()
y = features["is_high_freq"].to_pandas()

# ... sklearn model training ...

Step-by-step explanation.

1. The 50 GB ingest and heavy aggregation run in Polars — the lazy engine + streaming collect handle the size, and the parallel group_by uses every core. Result: ~2M rows in Polars memory.
2. duckdb.register("user_daily", transactions.to_arrow()) binds the result as a virtual table DuckDB can query. Zero copy — DuckDB reads the exact same Arrow buffers.
3. The analyst SQL query uses DuckDB features that are more ergonomic in SQL than in Polars expressions — window functions, QUALIFY clause, complex CTEs. Both are technically expressible in Polars, but the SQL reads more naturally for the analyst persona.
4. .arrow() returns the DuckDB result as another PyArrow Table. pl.from_arrow(...) brings it back into Polars for the next stage of processing (feature engineering with the expression API).
5. The final bridge to Pandas via .to_pandas() is the sole copy in the whole pipeline (scikit-learn expects NumPy-backed DataFrames). Every other handoff between Polars and DuckDB is zero-copy — the 50 GB source becomes small enough to fit in the sklearn training set without any wasteful materialisation in between.

Output.

Step	Rows	Memory	Copy?
Polars aggregate	2M	~200 MB	(source materialisation)
Polars → DuckDB via Arrow	2M	~200 MB shared	zero-copy
DuckDB SQL (top 1000)	1K	~100 KB	small new alloc
DuckDB → Polars via Arrow	1K	~100 KB shared	zero-copy
Polars → Pandas (sklearn)	1K	~100 KB	one copy (to NumPy)

Rule of thumb. For any two-persona pipeline (data engineers writing programs + analysts writing SQL), pair Polars with DuckDB via Arrow. Never bridge them through Pandas — that forces a serialisation and defeats the zero-copy interop. The Polars ↔ DuckDB round-trip is the 2026 default for hybrid workflows.

Worked example — when Spark is still the right answer

Detailed explanation. The one scenario where Polars loses decisively to Spark is when the working set genuinely exceeds one machine — > 1 TB tabular data, cluster-scale parallelism, fault tolerance across machine failures. Senior candidates who recommend Spark reflexively for "big data" without checking the actual size fail the interview; those who correctly identify when Spark wins pass. Walk through the boundary.

• When Spark wins. > 1 TB regularly, cluster requirement (fault tolerance, multi-tenant scheduling), Delta Lake / Iceberg first-class needs, existing Spark ecosystem investment.
• When Spark loses. Under 1 TB on one machine (JVM overhead, task serialisation cost, cluster startup latency), interactive analytics (Spark SQL is slower than DuckDB), Python-first teams (PySpark is a poor Python experience).

Question. Identify a workload that genuinely needs Spark and explain why Polars would not work.

Input.

Component	Value	Verdict
Data size	8 TB daily, growing 20%/year	> 1 TB → cluster
Fault-tolerance requirement	job must survive machine failure	Spark natively
Delta Lake tables	table format for the org	Spark first-class
Skill set	existing Spark team	migration cost matters
Latency target	overnight batch (8h SLA)	Spark tolerable

Code.

# The Spark version — genuinely needed at 8 TB scale
from pyspark.sql import SparkSession
from pyspark.sql import functions as F

spark = (
    SparkSession.builder
      .appName("daily_agg")
      .config("spark.sql.adaptive.enabled", "true")
      .config("spark.sql.adaptive.coalescePartitions.enabled", "true")
      .getOrCreate()
)

df = (
    spark.read.format("delta")
        .load("s3://acme-lake/transactions/")
        .filter(F.col("event_date") == "2026-07-21")
        .groupBy("region", "product_id")
        .agg(
            F.count("*").alias("n"),
            F.sum("units").alias("units"),
            F.sum("revenue_cents").alias("revenue"),
        )
)

df.write.format("delta").mode("overwrite").save("s3://acme-lake/hourly_agg/")


# The Polars version — this would OOM on any single machine
# (even a beefy 512 GB server can't hold 8 TB post-decompression)
# The Polars streaming engine could technically process it in ~14 hours
# on one machine, but a 40-node Spark cluster does it in 25 minutes.

# The decision framework — when Polars is the right answer at TB scale
# ONLY when:
#   1. The daily ingest can be pre-partitioned into single-day slices <= 100 GB, AND
#   2. Each slice can be processed independently, AND
#   3. There is no cross-partition state (joins across the full history), AND
#   4. Fault-tolerance requirements can be met by "retry the whole slice on failure"
#
# If ANY of these fail, Spark or another cluster engine is the correct answer.

Step-by-step explanation.

1. 8 TB daily is beyond the reasonable Polars-on-one-machine threshold. Even with the streaming engine, processing 8 TB on one machine would take ~10-15 hours; a 40-node Spark cluster does it in ~25 minutes with fault tolerance.
2. Delta Lake as the table format is a Spark strength — the transaction log, optimistic concurrency, time-travel queries are all native to Spark. Polars has read-only Delta support in 2026; write support is still catching up.
3. The existing Spark team is a real constraint. Migration to Polars would require re-training and rewriting years of PySpark code. Even if Polars won on speed for the current 8 TB workload, the migration cost would dwarf the savings.
4. Latency target (8h SLA) is generous — either tool can meet it. Speed alone doesn't drive the decision; operational fit does.
5. The senior-answer template: "For > 1 TB regularly, Spark. Under 1 TB, always check the single-machine story first — Polars usually wins unless there's a specific cluster requirement (fault tolerance, Delta writes, existing team investment). Don't recommend Spark reflexively for 'big data' — always check the actual number."

Output.

Workload	Recommend	Why
8 TB daily, Delta Lake, Spark team	Spark	> 1 TB + ecosystem fit
500 GB weekly, one machine, Python team	Polars	Single-machine + Python-first
5 TB daily, could pre-partition to 100 GB slices	Polars or Spark	Depends on fault-tolerance need
100 GB daily, analyst SQL primary	DuckDB	SQL-first interactive
2 GB daily, scikit-learn ML	Polars or Pandas	Ecosystem fit

Rule of thumb. Reserve Spark for genuine > 1 TB workloads or hard cluster requirements (fault tolerance, multi-tenant scheduling, Delta Lake writes). Never recommend Spark reflexively for "big data" without checking the actual size — for anything under 1 TB on one machine, Polars is almost always the better answer. This distinction is the single strongest senior signal in the 2026 tabular-tool interview.

Senior interview question on tool selection

A senior interviewer might ask: "Your team has three tabular workloads: (1) a 300 MB nightly Pandas pipeline that's slow, (2) a 15 GB Snowflake unload that analysts want to SQL-query directly, and (3) a 3 TB daily Delta Lake ingest currently on Spark. Walk me through the tool decision for each, the migration paths, and which order you'd tackle them in."

Solution Using Polars for (1), DuckDB for (2), keep Spark for (3) — with concrete migration steps

# Workload 1 (300 MB nightly Pandas — MIGRATE TO POLARS)
# Migration effort: ~1 day
# Expected win: 45 min → 5 s wall-clock, 8 GB → 500 MB peak RAM

# BEFORE
import pandas as pd
df = pd.read_parquet("nightly.parquet")
result = df[df["status"] == "ok"].groupby("region")["revenue"].sum()

# AFTER
import polars as pl
result = (
    pl.scan_parquet("nightly.parquet")
      .filter(pl.col("status") == "ok")
      .group_by("region")
      .agg(pl.col("revenue").sum())
      .collect()
)

# Workload 2 (15 GB Snowflake unload for analyst SQL — USE DUCKDB)
# Migration effort: ~half a day (analysts already know SQL)
# Expected win: attach directly to the S3 parquet folder; no data movement

import duckdb

conn = duckdb.connect()
conn.execute("INSTALL httpfs; LOAD httpfs;")
conn.execute("""
    CREATE VIEW nightly_unload AS
    SELECT * FROM 's3://acme-analytics/nightly/*.parquet'
""")

# Analysts query it directly — no ETL step
result = conn.execute("""
    SELECT day, region, SUM(revenue) AS total
    FROM   nightly_unload
    WHERE  day >= '2026-07-01'
    GROUP  BY day, region
    ORDER  BY day, total DESC
""").arrow()

# Workload 3 (3 TB daily Delta Lake on Spark — KEEP SPARK)
# Do NOT migrate. Reasons:
#   - 3 TB is above the single-machine break-even
#   - Delta Lake is Spark-native
#   - Existing team investment
#   - Fault tolerance requirements

# Instead, TUNE the existing Spark job for adaptive query execution + Photon
from pyspark.sql import SparkSession

spark = (
    SparkSession.builder
      .config("spark.sql.adaptive.enabled", "true")
      .config("spark.sql.adaptive.coalescePartitions.enabled", "true")
      .config("spark.sql.adaptive.skewJoin.enabled", "true")
      .getOrCreate()
)

Step-by-step trace.

Workload	Size	Current	Recommended	Effort	Wall-clock win
Nightly pipeline	300 MB	Pandas 45 min	Polars 5 s	1 day	500x
Snowflake unload SQL	15 GB	analyst pain	DuckDB attach	0.5 day	analyst UX
Delta Lake daily	3 TB	Spark 90 min	Spark tuned 60 min	2 days	modest
Migration order	—	—	2 → 1 → 3	—	quickest wins first

After the three-track plan, workload 2 ships in half a day and immediately unblocks the analysts; workload 1 ships in a day and eliminates the OOM incidents on the nightly job; workload 3 stays on Spark but gets ~30% faster with AQE tuning. Total effort: 3.5 days; total impact: one persona unblocked plus two pipelines faster.

Output:

Metric	Before	After
Nightly pipeline wall-clock	45 min	5 s
Analyst SQL onboarding	days (via Snowflake tickets)	minutes (DuckDB attach)
Daily Spark job	90 min	60 min
Migration total effort	—	3.5 days
OOM incidents	~2 / week	0

Why this works — concept by concept:

• Right tool per size boundary — 300 MB → Polars, 15 GB SQL-first → DuckDB, 3 TB Delta → Spark. Each pick respects the size × interface style axes; no reflexive recommendation.
• Migration order — analyst unblock first (smallest effort, biggest UX win), then Pandas → Polars (medium effort, massive perf win), then Spark tuning (biggest effort, modest perf win). Sequencing matters as much as choice.
• Zero-copy interop everywhere — Polars ↔ DuckDB via Arrow; both write Parquet that Spark can consume. No serialisation hell.
• Preserve existing investment — Spark stays where it wins. Migrating the 3 TB workload for the sake of consolidation would cost weeks and yield modest gains; keeping it and tuning is the sane choice.
• Cost — 3.5 engineer-days total; recoup by day 5 through avoided OOM incidents alone. The composite architecture (Polars + DuckDB + Spark) is more powerful than any single tool would be; the interview signal is recommending the composite, not defending a favourite. O(days) engineering cost, O(years) engineering payoff.

Window functions
Topic — window-functions
Window-function problems across Polars / DuckDB / Spark

Practice →

Design
Topic — design
Design problems on multi-tool tabular architectures

Practice →

---

Cheat sheet — Polars internals recipes

• Which internals when. LazyFrame + .collect() for every non-trivial pipeline (that's where 90% of the wins live). Eager pl.DataFrame only for tiny ad-hoc exploration in a Jupyter cell. Streaming (.collect(streaming=True) or sink_parquet) whenever the input exceeds half your available RAM. Polars-GPU only for > 100M-row numeric aggregations on a machine with an NVIDIA GPU — benchmark explicitly before adopting.
• LazyFrame construction template. pl.scan_parquet(path) (or scan_csv, scan_ndjson, scan_delta) → .filter(pl.col("x") == v) (predicate-pushdown-friendly filters) → .select("a", "b", "c") (projection pushdown) → .group_by(keys).agg([pl.col("y").sum(), ...]) → .sort(col, descending=True) → .head(N) → .collect(streaming=True). Every hot pipeline follows this shape; the optimizer rewards it.
• Reading explain(optimized=True) output. PROJECT k/n COLUMNS on the scan means projection pushdown fired (k of n cols read). SELECTION: [ ... ] on the scan means predicate pushdown fired. SLICE offset: 0 len: N above a sort means slice pushdown fired for head(N). If any of these are missing on a query you expected to be fast, you have a UDF, a wildcard, or an unsupported operator blocking the optimizer.
• UDF and wildcard traps. .map_elements(python_fn) blocks predicate pushdown (rewrite to native expressions). .with_columns(pl.all().cast(...)) before a .select(...) blocks projection pushdown (narrow first, then cast). .apply and struct.apply are also UDF traps. When a UDF is truly unavoidable, place it after every native filter and aggregation so it operates on the smallest possible row set.
• Arrow zero-copy interop template. Polars → PyArrow via df.to_arrow() (zero-copy on flat cols). PyArrow → DuckDB via duckdb.register("name", tbl) then duckdb.query("...") (zero-copy view). PyArrow → Pandas via tbl.to_pandas(types_mapper=pd.ArrowDtype) (zero-copy ExtensionArray). Never route through .to_pandas() in the middle of a Polars + DuckDB flow; that forces a NumPy-plus-Python-objects materialisation and defeats every zero-copy win.
• String column optimisation. For low-cardinality strings (< 1000 distinct values), cast to pl.Categorical — 50x – 100x memory reduction and integer-index equality checks. For high-cardinality strings, plain Polars strings are already 5x – 10x smaller than Pandas object dtype. Never carry Pandas object strings through a hot loop; convert to Polars via pl.from_pandas(df, schema_overrides={"col": pl.Categorical}).
• Rechunk decision. Rechunk (.rechunk()) before sort-heavy or complex-expression operations (they would rechunk implicitly). Do not rechunk before parallel scans, filters, or group-bys — chunks are the parallelism unit. df["x"].n_chunks() reports the current count. After pl.concat(...) of many DataFrames, chunk count = sum; before a downstream sort, .rechunk() once.
• Streaming coverage check. Before relying on .collect(streaming=True), run .explain(streaming=True) and confirm no operator falls back to in-memory. As of Polars 1.x, streaming coverage is ~85% (scan, filter, projection, group_by, most joins). Non-streaming operators (some window functions, complex sorts) trigger a full materialisation of that stage.
• Thread-pool and CPU sizing. pl.thread_pool_size() reports the current Rayon pool. Set POLARS_MAX_THREADS=N env var to cap it (essential in containers with CPU limits — otherwise Polars over-schedules). On NUMA hosts (2-socket servers), pinning to one socket sometimes beats the default.
• Polars ↔ Pandas ↔ DuckDB matrix. Polars → Pandas: df.to_pandas(use_pyarrow_extension_array=True) (zero-copy where compatible). Pandas → Polars: pl.from_pandas(df) (fast; drops the Pandas index by default). Polars → DuckDB: duckdb.register("name", df.to_arrow()) (zero-copy). DuckDB → Polars: pl.from_arrow(rel.arrow()) (zero-copy). Pandas ↔ DuckDB should also go through Arrow; never through DataFrame conversion.
• Failure semantics reminder. LazyFrame .collect() OOMs → switch to .collect(streaming=True). Streaming OOMs → check for a non-streaming operator via .explain(streaming=True) and rewrite or split. UDF slowness → rewrite as native expression or move UDF post-aggregation. Slow-scan surprise → verify PROJECT k/n in explain output; add explicit .select() if wildcards crept in. Every failure has a diagnostic; explain-and-verify is the workflow.
• Decision matrix vs Pandas / DuckDB / Spark. Data size: Pandas < 1 GB, Polars 100 MB – 100 GB, DuckDB 1 GB – 500 GB, Spark > 1 TB. Interface: Polars/Pandas program-embedded, DuckDB SQL-first attach, Spark distributed cluster. Speed: Polars > DuckDB > Spark-local >> Pandas on single-machine tabular workloads. Distributed: only Spark. Ecosystem: Pandas > everyone (scikit-learn, statsmodels). Pick per workload; defend with concrete numbers, not preferences.
• Migration cost between tools. Pandas → Polars: ~1 day per pipeline (syntax is close; add .lazy() / .collect()). Pandas → DuckDB: ~half day (rewrite as SQL). Pandas → Spark: ~1 week per pipeline (rewrite as PySpark; provision cluster). Polars ↔ DuckDB: ~0 days (Arrow bridge). Polars → Spark: ~1-2 weeks (rewrite as PySpark). Never migrate to Spark for speed on one machine; only migrate for genuine cluster requirements.

Frequently asked questions

What are Polars internals in one sentence?

Polars internals are the four composed engineering decisions — lazy execution via LazyFrame, a real query optimizer with predicate / projection / slice pushdown, Arrow2 columnar memory layout with validity bitmaps and zero-copy interop, and Rayon-based work-stealing parallelism across every physical core plus a morsel-driven streaming engine — that together give the Rust-native DataFrame library a 10x – 100x wall-clock speed-up over Pandas on 100 MB – 100 GB tabular workloads on a single machine, without needing a distributed cluster the way Spark does. Every senior data-engineering interview in 2026 probes polars internals because they are the load-bearing reason the tool has displaced Pandas as the default for medium-size analytics, and the four decisions compose — none of them in isolation delivers the full win.

LazyFrame vs eager DataFrame — when do I pick each?

Default to LazyFrame for every non-trivial pipeline. pl.scan_parquet(path) returns a polars lazyframe with no I/O; every subsequent .filter(), .select(), .group_by(), .agg() appends to a logical plan; only .collect() fires the optimizer and executes. This is where the pushdowns, CSE, and join reordering live — using pl.DataFrame (eager) throws all of that away and turns Polars into "slightly faster Pandas" instead of the "10x – 100x speed-up" tool. Reach for eager pl.DataFrame only for tiny ad-hoc exploration in a Jupyter cell where the data is already small and you want immediate feedback per line. In production code, every Polars pipeline should start with scan_* and end with .collect() (or sink_* for streaming write). The migration from eager to lazy is usually a single .lazy() call chained after pl.DataFrame(...); adopting the pattern is one of the highest-leverage code reviews you can do on an existing Polars codebase.

What is the Polars query optimizer and what passes does it run?

The polars query optimizer is the rewriter that runs when you call .collect() on a LazyFrame — it walks the logical plan and applies a series of well-defined passes before physical planning. The load-bearing passes are: predicate pushdown (move .filter() clauses down into the scan so Parquet row-group min/max statistics can skip whole groups of rows without decoding them), projection pushdown (only read the columns the query actually references — often 3 of 200 for typical analytics queries), slice pushdown (push head(N) and limit(N) down through sorts and aggregations so a top-K becomes a bounded-heap operation instead of a full sort), common-subexpression elimination (compute repeated expressions once), and join reordering (build the hash table on the smaller side; reorder multi-way joins to minimise intermediate cardinality). You inspect the results with lf.explain(optimized=True) — look for PROJECT k/n COLUMNS and SELECTION: [ ... ] on the scan node to confirm the pushdowns fired; if they didn't, hunt for UDFs, wildcards, or unsupported expressions blocking the optimizer.

How does Arrow2 differ from PyArrow and why does it matter?

Arrow2 is a Rust reimplementation of the Apache Arrow columnar format that Polars uses as its in-memory representation; PyArrow is the Python binding to the C++ Arrow implementation used by DuckDB, Pandas 2.x, and many other tools. From a user's perspective the two are interchangeable — df.to_arrow() returns a pyarrow.Table that shares buffers with the Polars DataFrame, thanks to the Arrow C data interface (a small C ABI defined by the Arrow project so any Arrow implementation can hand data to any other without a copy). What matters for polars performance is not which crate is used but the layout itself: every column is a contiguous typed buffer plus a separate validity (null) bitmap, with variable-width types (strings, lists) getting an additional offsets buffer. This layout is SIMD-friendly (16 – 64 byte contiguous scans), cache-friendly (linear reads, no pointer chasing), and cross-engine friendly (zero-copy handoff to DuckDB, DataFusion, R arrow, Go arrow). The specific win over Pandas object-dtype string columns is a 5x – 10x memory reduction, plus categorical / dictionary encoding for low-cardinality strings brings another 10x — for a 100M-row string column with 5 distinct values, Pandas uses ~7 GB and Polars categorical uses ~100 MB.

Can Polars replace Spark?

Not for genuine > 1 TB distributed workloads — Spark still wins for cluster-scale processing, fault tolerance across machine failures, native Delta Lake / Iceberg writes, and multi-tenant scheduling. But Polars replaces Spark decisively for single-machine tabular workloads in the 100 MB – 100 GB range where teams historically over-provisioned to Spark out of habit. A 20 GB daily Parquet aggregation that Pandas takes 45 minutes on and Spark-local takes 5 minutes on runs in ~30 seconds on Polars with .collect(streaming=True) — the JVM overhead, task serialisation, and cluster startup latency of Spark are pure waste at that size. The senior-interview signal is drawing the boundary at ~1 TB and defending it with concrete numbers rather than reflexively recommending Spark for "big data". A common architecture in 2026: Spark for the multi-terabyte lake-scale ingest, Polars for downstream single-machine feature engineering and ETL, DuckDB for analyst-facing SQL over intermediate results — three tools, each in their sweet spot, all sharing data zero-copy via Arrow. Never migrate to Spark for speed on one machine; only migrate for genuine cluster requirements.

Is Polars-GPU production ready in 2026?

Yes, for a specific and growing subset of workloads. Polars-GPU (the NVIDIA collaboration announced October 2024) runs the physical plan on GPU via the cuDF library from RAPIDS, opt-in via .collect(engine="gpu") or pl.Config.set_engine("gpu"). As of 2026 it wins by 3x – 10x on multi-billion-row numeric aggregations, joins on primitive keys, and vectorised expressions on a machine with an NVIDIA GPU (A100, H100, and consumer RTX 40-series). It loses to CPU on: (a) small workloads where the RAM → GPU transfer cost dominates (typically below ~100M rows), (b) string-heavy operations where GPU coverage is still limited and the CPU fallback with transfer round-trip is worse than pure CPU, (c) any query with unsupported operators (fallback penalty). The operational cost is real — polars[gpu] requires an NVIDIA GPU with recent drivers and consumes GPU memory that other processes might want, so in shared-tenancy notebooks it's a config nightmare and in dedicated batch jobs it's fine. Senior interview signal: never claim "GPU is always faster" — benchmark explicitly per workload and know the string-heavy failure mode. The 2026 production pattern is opt-in per pipeline, not blanket adoption.

Practice on PipeCode

• Drill the SQL practice library → for the query-plan, optimizer, and columnar-execution problems that senior Polars interviewers translate into DataFrame API drills.
• Rehearse aggregation patterns on the aggregation practice library → to build the muscle memory for group_by + agg chains that map directly to Polars LazyFrame code.
• Sharpen the join axis with the joins practice library → for parallel-hash-join, broadcast-join, and streaming-join scenarios the Polars physical planner runs.
• Level up the analytical patterns with the window-functions practice library → for the rolling / lag / rank workloads where Polars, DuckDB, and Spark each have their own optimizer strengths.
• Stack the prerequisites against PipeCode's broader 450+ data-engineering catalogue to anchor the four-internals decision matrix against real graded inputs and design problems.

Lock in polars internals muscle memory

Docs explain LazyFrames. PipeCode drills explain the decision — when predicate pushdown fires and when a Python UDF silently blocks it, when Arrow2 zero-copy interop saves 25 GB of duplicate memory, when the streaming engine turns a 100 GB Parquet on a 16 GB laptop from OOM into a Tuesday-morning task, and when Spark is genuinely still the right answer. Pipecode.ai is Leetcode for Data Engineering — pattern-first practice tuned for the production trade-offs senior data engineers actually face.

Practice SQL problems →
Practice aggregation problems →