Apache DataFusion for Data Engineers: Rust-Native Query Engine Under Ballista, InfluxDB 3, Comet

apache datafusion is the pick-one query-engine substrate that quietly reshaped the modern data stack — the Rust-native columnar engine that InfluxDB 3.0, Comet, Ballista, Sail, GreptimeDB, ROAPI, Cube.dev, and dozens of other vendors ship inside their products rather than build again from scratch. Every operational analytics query your team runs against Parquet on object storage, every time-series roll-up, every embedded query engine inside a Rust microservice, and every accelerated Spark stage now has a plausible path that ends with a DataFusion plan tree, a physical operator graph, and an Arrow2 RecordBatch stream — because writing a correct SQL parser plus a rule-based optimizer plus a volcano executor plus a Parquet reader plus a Tokio-integrated async I/O layer once, in a memory-safe language, is a multi-year investment nobody wants to repeat.

This guide is the senior-DE walkthrough you wished existed the first time an interviewer asked "what is Apache DataFusion and why do you keep seeing it under other people's products," or "walk me through the DataFusion plan lifecycle from SQL to execution stream," or "when would you pick DataFusion over DuckDB or Polars." It walks through the Rust-native columnar engine that grew out of the Ballista sub-project, the four-stage plan lifecycle (SQL / DataFrame API → logical plan → optimizer → physical plan → execution stream) over Arrow2 RecordBatches, the rule-based optimizer passes (filter pushdown, projection pushdown, join reorder, CSE, constant fold) and how they compare to Spark's Catalyst and Flink/Trino's Calcite, the downstream projects that adopted it as their shared runtime, and the interview signals that separate architects who have embedded DataFusion from candidates who have only read the docs. Each section pairs a teaching block with a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the SQL practice library →, rehearse on the ETL practice library →, and sharpen the streaming axis with the streaming practice library →.

---

On this page

• Why DataFusion matters in 2026
• DataFusion architecture — plan lifecycle over Arrow2
• DataFusion optimizer and expression rewriter
• Downstream projects and vendor adoption
• Using DataFusion directly and interview signals
• Cheat sheet — DataFusion recipes
• Frequently asked questions
• Practice on PipeCode

---

1. Why DataFusion matters in 2026

The Rust-native columnar substrate quietly reshaping the modern data stack

The one-sentence invariant: Apache DataFusion is a Rust-native, Arrow2-based, columnar query engine that exposes both SQL and DataFrame APIs, ships as an embeddable library rather than a server, and has become the shared substrate under InfluxDB 3.0 (IOx), Comet (Spark's Rust-native execution accelerator), Ballista (distributed DataFusion on Kubernetes), Sail (distributed Rust engine), GreptimeDB, ROAPI, Cube.dev, ParadeDB, and Coralogix — because writing a correct SQL parser plus a rule-based optimizer plus a volcano-style executor plus a Parquet reader plus a Tokio-integrated async I/O layer once, in a memory-safe language, is a five-year investment nobody wants to repeat. The pattern that mattered five years ago was "vendor writes their own engine"; the pattern that matters now is "vendor picks a shared substrate and contributes back," and DataFusion is the substrate.

The four axes senior interviewers actually probe.

• Language substrate. DataFusion is Rust, not Java or C++. That single choice cascades: zero JVM garbage collection pauses, memory-safety without a garbage collector, cargo-based dependency management, and a runtime footprint measured in single-digit megabytes rather than the JVM's hundreds. Interviewers who probe language choice are separating candidates who have deployed DataFusion inside a service from those who have only read the docs.
• Execution model. Volcano-style pull-based iteration over Arrow2 RecordBatches, with async I/O via Tokio. Each physical operator implements the same ExecutionPlan trait that returns a SendableRecordBatchStream. This is qualitatively different from Spark's whole-stage code-generation-per-stage; it trades peak throughput for embed-ability and startup latency.
• Extensibility. UDF, UDAF, UDWF, custom TableProvider, custom OptimizerRule, custom physical operator. Every extension point that Spark exposes via Catalyst is exposed via DataFusion's LogicalPlan and ExecutionPlan traits. The extensibility surface is why downstream projects (InfluxDB 3, Comet, GreptimeDB) can plug their own storage layer and their own optimizer passes without forking the engine.
• Deployment shape. Embedded library first, distributed second. cargo add datafusion gives you a query engine inside your Rust binary. Ballista wraps it in a Kubernetes-native distributed runner. Sail wraps it in a different distributed runner. The engine itself is the same; the deployment shape is the choice.

The shared-substrate trend — the sentence senior interviewers listen for.

• What it means. Instead of every vendor building their own query engine from scratch, the industry converged on a small set of shared substrates: Rust-native → DataFusion; C++-native → Velox (Meta) and Photon (Databricks); JVM-native → Catalyst (Spark) and Calcite (Flink / Trino). One substrate, many downstreams.
• Why it happened. Building a correct query engine takes years and yields a commodity output. Vendors want to compete on storage layer, on distribution model, on connectors, on UX — not on whether their WHERE clause pushdown works. Sharing the substrate lets each vendor allocate engineering budget to their differentiator.
• The DataFusion payoff. A bug found in DataFusion by InfluxDB gets fixed once and adopted by Comet, GreptimeDB, ROAPI, and every other downstream. A new optimizer rule contributed by Coralogix flows to the whole ecosystem. This is the "bugs fixed once, adopted by all" property that senior architects cite when defending a shared-substrate decision.
• The competitor landscape. Velox is C++-first (Meta, Presto, Spark accelerator). Photon is proprietary (Databricks). Catalyst / Calcite are JVM-first. DataFusion is the answer when your product ships as a static Rust binary and cannot afford a JVM.

The 2026 reality — DataFusion is the default for Rust analytics.

• Greenfield Rust analytics. If you're writing a query engine, a time-series database, an analytics gateway, or a Parquet-native tool in Rust and you have a scoped-enough workload that "just call DataFusion" answers the question, you should call DataFusion. Building your own logical plan tree in 2026 is a career-limiting move.
• Embedded engines. Inside application binaries where a JVM is off the table (edge, WASM, mobile server, single-binary CLI), DataFusion is the answer for anything more complex than SQLite. datafusion-cli is the tool developers reach for when they want DuckDB-shaped ergonomics but with Rust-native embedding.
• Distributed shape. Ballista and Sail wrap DataFusion for distributed execution. Neither has yet reached Spark's ecosystem maturity, but both are production-viable for scoped workloads; Comet is the pragmatic bridge that runs DataFusion inside a Spark stage, giving Spark users the accelerator without a full platform migration.
• The competitor list. DuckDB (single-node analytics champion; C++), Polars (DataFrame-first; Rust), Trino (distributed federated; JVM), Spark (distributed batch + streaming; JVM), Clickhouse (columnar OLAP; C++). DataFusion competes in a specific slice: Rust-native, embeddable, both SQL and DataFrame, and increasingly distributed.

What interviewers listen for.

• Do you name Apache DataFusion as "the Rust-native columnar query engine substrate" in sentence one? — required answer.
• Do you name at least three downstream projects (InfluxDB 3, Comet, Ballista, GreptimeDB) unprompted? — senior signal.
• Do you describe DataFusion as "an embeddable library, not a server" and contrast with Spark? — senior signal.
• Do you name Arrow2 as the in-memory format and Tokio as the async runtime? — required answer.
• Do you place DataFusion on the shared-substrate axis (Rust ↔ Velox ↔ Catalyst) rather than as "another Rust query tool"? — senior signal.

Worked example — the four-axis DataFusion positioning card

Detailed explanation. The most useful artifact for a DataFusion interview is a memorised positioning card that answers "where does DataFusion sit versus DuckDB, Polars, Spark, and Trino." Walk through building the card with the four axes above (language, execution model, extensibility, deployment shape). Interviewers ask this question at some point in every senior data-platform conversation; having the card in your head is what separates a fluent answer from a stumbling one.

• Language axis. Rust for DataFusion, Polars. C++ for DuckDB, Velox, ClickHouse. JVM for Spark, Trino, Flink.
• Execution model axis. Volcano pull-based for DataFusion, DuckDB, Trino. Whole-stage codegen for Spark. Vectorised interpreted for Polars, ClickHouse.
• Extensibility axis. DataFusion exposes LogicalPlan, OptimizerRule, ExecutionPlan, TableProvider, UDF/UDAF/UDWF. DuckDB has extensions but less pluggable. Polars is DataFrame-first, less SQL-plan-tree oriented.
• Deployment shape axis. Embedded for DataFusion, DuckDB, Polars. Distributed for Spark, Trino, Ballista (DF-wrapped), Sail.

Question. Build the four-axis positioning card for DataFusion versus DuckDB, Polars, Spark, and Trino, and pick the axis on which DataFusion is uniquely positioned.

Input.

Engine	Language	Execution	Deployment	Extensibility surface
DataFusion	Rust	volcano pull	embedded + distributed (Ballista)	LogicalPlan / OptimizerRule / ExecutionPlan / TableProvider
DuckDB	C++	vectorised	embedded	extensions (C++ only)
Polars	Rust	vectorised	embedded	limited plan-tree API
Spark	JVM	whole-stage codegen	distributed	Catalyst API
Trino	JVM	volcano pull	distributed	connector SPI

Code.

// The DataFusion "hello world" that reveals its positioning
// Cargo.toml:  datafusion = "43"
use datafusion::prelude::*;

#[tokio::main]
async fn main() -> datafusion::error::Result<()> {
    // 1. A single SessionContext holds catalog + config + optimizer
    let ctx = SessionContext::new();

    // 2. Register a Parquet file as a table — this is the TableProvider layer
    ctx.register_parquet("orders", "s3://bucket/orders/*.parquet",
                         ParquetReadOptions::default()).await?;

    // 3. Run SQL — this exercises the whole plan lifecycle
    let df = ctx.sql("
        SELECT customer_id, COUNT(*) AS n, SUM(total_cents) AS revenue
        FROM   orders
        WHERE  status = 'shipped'
        GROUP  BY customer_id
        ORDER  BY revenue DESC
        LIMIT  10
    ").await?;

    // 4. Collect the RecordBatch stream — volcano pull to completion
    df.show().await?;
    Ok(())
}

Step-by-step explanation.

1. The SessionContext is the DataFusion equivalent of a Spark SparkSession — it holds the catalog (which tables exist), the optimizer configuration, the physical planner, and the execution runtime. In an embedded deployment it's a single Rust struct; there's no separate driver process.
2. register_parquet wires a Parquet file (or a glob of files, or an S3 prefix) into the catalog via the built-in ListingTable provider. This is the TableProvider extension point at work — you could just as easily register a custom source (a REST endpoint, a Kafka topic, a Redis hash) by implementing TableProvider.
3. ctx.sql(...) runs the query through the four-stage plan lifecycle: sqlparser-rs parses the string into a SQL AST; the plan builder converts the AST into a logical plan; the optimizer runs the rule-based passes; the physical planner converts the optimized logical plan into an ExecutionPlan. All of this happens inside the same Rust process.
4. df.show() triggers execute() on the physical plan, which returns a SendableRecordBatchStream. The volcano pull-based executor consumes the stream to completion, printing each batch as it arrives. No JVM, no driver process, no cluster — a single Rust binary is the whole runtime.
5. The uniquely-positioned axis for DataFusion is the combination: Rust-native + embedded-first + full SQL surface + full plan-tree extensibility + optional distributed via Ballista. No other engine hits all five simultaneously. DuckDB is embedded but C++. Polars is Rust but DataFrame-first. Spark is distributed but JVM. This intersection is why downstream projects converge here.

Output.

Axis intersection	Engine that fits
Rust-native + embedded + full SQL + plan extensibility	DataFusion (unique)
C++ + embedded + full SQL	DuckDB
Rust + embedded + DataFrame-first	Polars
JVM + distributed + full SQL + plan extensibility	Spark, Trino
Rust + distributed (embedded engine wrapped)	Ballista, Sail (both use DataFusion)

Rule of thumb. DataFusion wins whenever the requirement contains the words "Rust-native," "embeddable in our binary," "extensible plan tree," and "SQL surface required." Drop any one of those constraints and another engine becomes viable; keep all four and DataFusion is essentially the only answer.

Worked example — the shared-substrate argument in one whiteboard

Detailed explanation. The single most-cited senior architecture argument for DataFusion is "shared substrate." Interviewers listen for whether you can articulate what a shared substrate is, why it emerged in 2022-2024, and what the payoff is for a vendor choosing to embed DataFusion instead of writing their own engine. Walk through the argument with the InfluxDB 3.0 case study — Paul Dix's team ripped out InfluxDB 2's Go-based engine and rebuilt IOx on Rust + Arrow + DataFusion + Parquet — because rewriting a query engine every three years is not a viable business model.

• The pre-substrate world. Every vendor writes their own query engine. Presto, Spark, Trino, Impala, Drill, ClickHouse, DuckDB, TimescaleDB — each with its own optimizer, its own operator library, its own SQL parser. Duplicated engineering everywhere.
• The substrate insight. SQL parsing, logical planning, optimizer rules, physical operators, and Arrow-based execution are commodities. What's not a commodity is the storage layer, the distribution model, the connectors, and the UX. Share the commodities; compete on the differentiators.
• The three substrates. Rust → DataFusion. C++ → Velox. JVM → Catalyst / Calcite. Each substrate wins in its language ecosystem.
• The payoff for downstream vendors. Multi-year engineering budget freed up. Optimizer bugs fixed by other vendors flow in automatically. New optimizer rules contributed back to the shared repo. The shared runtime becomes stronger every year.

Question. Walk through the shared-substrate argument for an interviewer who asks "why did InfluxDB 3.0 pick DataFusion instead of building their own engine?"

Input.

Vendor decision	Substrate choice	Rationale
InfluxDB 3.0 IOx	DataFusion	Rust-native; time-series storage differentiator; free query engine
Comet	DataFusion	Rust-native accelerator for Spark stages; JVM ⇄ native handoff
Meta (Presto, Spark)	Velox	C++ ecosystem; vectorised execution focus
Databricks Photon	in-house C++	Proprietary; couples to Databricks runtime
Trino	Calcite (partially)	JVM ecosystem; connector-first

Code.

The shared-substrate argument (5-minute whiteboard)
====================================================

1. THE COMMODITY
   SQL parser, logical planner, optimizer rules, physical operators, Arrow
   execution — all commodities in 2026. Every vendor writing their own is
   duplicated effort with no competitive payoff.

2. THE THREE SUBSTRATES
   Rust  → Apache DataFusion  (Arrow2, sqlparser-rs, Tokio async)
   C++   → Velox              (Meta, ripped through Presto + Spark)
   JVM   → Catalyst / Calcite (Spark's Catalyst, Trino/Flink's Calcite)

3. THE VENDOR PLAYBOOK
   • Pick the substrate that matches your product's language.
   • Plug in your storage layer via TableProvider (or equivalent).
   • Plug in your custom OptimizerRule for pushdown into your storage.
   • Ship your product with the substrate as a dependency.
   • Contribute bug fixes and new rules back upstream.

4. THE PAYOFF
   • Multi-year engineering budget freed for differentiation.
   • Optimizer improvements from other vendors flow in for free.
   • Compatibility with the wider Arrow / Parquet / Iceberg ecosystem
     comes as a byproduct.

5. THE INFLUXDB 3.0 CASE
   • InfluxDB 2 used a bespoke Go-based engine.
   • InfluxDB 3.0 (IOx) rebuilt on Rust + Arrow + DataFusion + Parquet.
   • The team focuses on time-series storage; DataFusion handles queries.
   • Result: sub-second queries, 100× ingest, Parquet-native storage,
     no bespoke query engine to maintain.

Step-by-step explanation.

1. The commodity insight is the load-bearing sentence. Interviewers want to hear you explain why the shared-substrate pattern emerged, not just that it exists. The commoditisation of SQL parsing and rule-based optimization is what made the pattern viable.
2. Naming the three substrates by language grouping is a fluency signal. Weak candidates name "DataFusion" without positioning it; senior candidates name "DataFusion, Velox, Catalyst — one per language ecosystem" and explain why each won its slot.
3. The vendor playbook is the how. Every downstream project follows roughly the same recipe: pick substrate, plug in storage via TableProvider, plug in custom optimizer rules, contribute back. Being able to walk this recipe is what separates a candidate who has embedded DataFusion from one who has only read the readme.
4. The payoff is the why. Freed engineering budget is the load-bearing benefit; the compounding improvements from other vendors are the secondary benefit; the ecosystem compatibility is the tertiary benefit. Interviewers grade the ordering.
5. The InfluxDB 3.0 case study is the concrete example. Paul Dix's team publicly documented the rewrite (KubeCon 2022, InfluxDays 2023); citing it by name is a strong senior signal.

Output.

Sentence in your answer	Grading signal
"DataFusion is the Rust substrate — Velox is C++, Catalyst is JVM"	senior
"Vendors converged on shared substrates because query engines commoditised"	senior
"InfluxDB 3.0 rebuilt IOx on Rust + Arrow + DataFusion + Parquet"	senior
"The vendor plugs in storage via TableProvider"	required
"Bug fixes and rules flow back to the shared repo"	senior

Rule of thumb. Never talk about DataFusion in isolation in a senior interview. Always position it on the shared-substrate axis: Rust ↔ Velox ↔ Catalyst, then name at least three downstream projects that adopted it. The positioning is the answer; the naming is the fluency proof.

Worked example — the "when NOT to use DataFusion" pushback

Detailed explanation. Every senior interview probes whether you can defend the negative case — when not to pick your favourite tool. DataFusion is not the answer to every query problem; there are three explicit non-answers a senior architect names unprompted. Walk through each, then name the alternative that wins in that slot.

• Non-answer 1. Single-node analytics against local Parquet with no need for embed-ability → DuckDB. DuckDB is faster in benchmarks, has a larger community, and ships a more mature CLI/Python UX. If you don't need Rust-native embedding, DuckDB is the pragmatic pick.
• Non-answer 2. DataFrame-first ergonomics for interactive Python analysis → Polars. Polars is Rust under the hood too, but its API is DataFrame-first (pandas-shaped); DataFusion is SQL-first with a DataFrame afterthought.
• Non-answer 3. Distributed workloads at Spark ecosystem scale → Spark. Ballista and Sail are viable but haven't reached Spark's connector ecosystem, streaming maturity, or cluster-management story. If the requirement is "1000-node cluster with 15 years of ecosystem," pick Spark.

Question. Draft the "when NOT to pick DataFusion" pushback answer and quantify the switching cost against each alternative.

Input.

Alternative	When it wins	Switching cost from DataFusion
DuckDB	Single-node analytics, no embed-ability required	Small — SQL surface mostly compatible
Polars	DataFrame-first Python interactive	Medium — API paradigm shift
Spark	Distributed at ecosystem scale	Large — full platform
ClickHouse	Real-time OLAP with sub-second aggregations at scale	Large — different storage model
Trino	Federated queries across many source systems	Medium — different deployment shape

Code.

The "when NOT to use DataFusion" pushback
==========================================

Q: "Why not DuckDB?"
A: "For a pure single-node analytics workload with no Rust-embedding
    requirement, DuckDB is faster in most benchmarks and has a more
    mature Python/CLI UX. Pick DuckDB when you don't need to embed
    the engine inside a Rust binary."

Q: "Why not Polars?"
A: "Polars is Rust too, but its API is DataFrame-first (pandas-shaped).
    DataFusion is SQL-first with a DataFrame afterthought. Pick Polars
    when your team lives in DataFrame chains; pick DataFusion when your
    team writes SQL and needs the plan tree extensibility."

Q: "Why not Spark?"
A: "For distributed workloads with a mature ecosystem — Delta Lake,
    Structured Streaming, MLlib, GraphX, 15 years of connectors —
    Spark is still the answer. Ballista is production-viable but does
    not have Spark's ecosystem depth in 2026."

Q: "Why not ClickHouse?"
A: "ClickHouse wins on real-time OLAP with billions of rows and sub-
    second aggregations. It's not embeddable, but for a hosted OLAP
    service ClickHouse's MergeTree storage beats DataFusion + Parquet
    for that specific workload."

Q: "Why not Trino?"
A: "Trino wins on federated queries — 30 different source systems joined
    in one SQL query. DataFusion has good connector coverage but Trino's
    connector SPI is more mature and more actively curated."

Step-by-step explanation.

1. Naming the negative case up front is a senior signal. Interviewers explicitly probe "when would you not pick DataFusion?" because it separates candidates who genuinely understand the trade-offs from candidates who just parrot the docs.
2. DuckDB is the closest competitor for single-node analytics. Every senior answer names DuckDB as the pragmatic non-answer when Rust-native embedding is not a requirement. Failing to name DuckDB here is a red flag.
3. Polars is the closest competitor within Rust itself. Naming it as "DataFrame-first" rather than "SQL-first" is the correct positioning; conflating DataFusion and Polars is a fluency failure.
4. Spark remains the answer for distributed at scale. Being honest about Ballista's ecosystem-maturity gap is a senior signal; overselling Ballista is a red flag.
5. ClickHouse and Trino are the "narrow-slice wins" — ClickHouse for real-time OLAP, Trino for federated queries. Naming both shows breadth without needing to claim DataFusion is universally superior.

Output.

Scenario	Pick	Reason
Rust embedded engine, Parquet queries, SQL	DataFusion	uniquely positioned
Single-node analytics, no embed	DuckDB	faster, more mature Python UX
Rust DataFrame ergonomics	Polars	pandas-shaped API
Distributed at Spark ecosystem scale	Spark	connectors, streaming, MLlib
Real-time OLAP at billions of rows	ClickHouse	MergeTree beats Parquet for this
Federated queries across many systems	Trino	connector SPI depth

Rule of thumb. Never sell DataFusion as universally superior. Position it precisely: Rust-native + embedded + SQL + plan-tree extensibility. Name the alternatives that win in adjacent slots. The precision is the credibility.

Senior interview question on DataFusion positioning

A senior interviewer often opens with: "We're building a time-series database in Rust. Walk me through why you'd pick DataFusion as the query engine, what the alternatives are, and what you'd have to build yourself on top."

Solution Using DataFusion as the substrate with custom TableProvider, custom optimizer rule, and vendored async runtime

// Cargo.toml
// [dependencies]
// datafusion = "43"
// arrow      = "53"
// tokio      = { version = "1", features = ["full"] }
// object_store = "0.11"

use std::sync::Arc;
use datafusion::prelude::*;
use datafusion::datasource::TableProvider;
use datafusion::execution::context::SessionContext;
use datafusion::optimizer::OptimizerRule;

// 1. A time-series storage layer wired as a TableProvider
//    (skeleton — details of the real storage layer elided)
struct TimeSeriesTable {
    schema: arrow::datatypes::SchemaRef,
    // ... your storage layer handles here
}

impl TableProvider for TimeSeriesTable {
    // Implement schema(), scan(), supports_filter_pushdown(), statistics()
    // to expose your time-series storage to the DataFusion planner.
    // Full impl elided; see datafusion::datasource::TableProvider trait.
}

// 2. A custom optimizer rule that recognises time-series patterns
//    (e.g. downsample() UDAF over a tumbling window)
struct TimeSeriesDownsampleRule;

impl OptimizerRule for TimeSeriesDownsampleRule {
    // Rewrite matching subtrees to push downsample() into storage
}

// 3. Wire everything into a SessionContext at startup
#[tokio::main]
async fn main() -> datafusion::error::Result<()> {
    let ctx = SessionContext::new();

    // Register the custom TableProvider
    let ts_table = Arc::new(TimeSeriesTable { schema: Arc::new(/* ... */) });
    ctx.register_table("metrics", ts_table)?;

    // Add the custom optimizer rule to the session config
    let state = ctx.state();
    let mut rules = state.optimizer().rules.clone();
    rules.push(Arc::new(TimeSeriesDownsampleRule));
    // (in real code: rebuild the SessionContext with the new rule set)

    // Now users can write SQL that hits your time-series storage
    let df = ctx.sql("
        SELECT time_bucket('1 minute', ts) AS minute,
               metric,
               avg(value) AS avg_value
        FROM   metrics
        WHERE  ts >= now() - INTERVAL '1 hour'
          AND  metric IN ('cpu.user', 'cpu.system')
        GROUP  BY 1, 2
        ORDER  BY 1
    ").await?;

    df.show().await?;
    Ok(())
}

Step-by-step trace.

Layer	Component	Purpose
Substrate	datafusion = "43"	full SQL / DataFrame / optimizer / executor
Storage adapter	TimeSeriesTable: TableProvider	plug time-series files into the DataFusion catalog
Pushdown	supports_filter_pushdown	let time-range filters skip file scans
Rewrite	TimeSeriesDownsampleRule	push downsample() into storage instead of scanning raw
Runtime	Tokio	async I/O to object storage or the local FS
API	SessionContext + SQL	end-users write standard SQL

After deployment, users can query your time-series database with standard SQL; the DataFusion planner + optimizer + executor handles the plan lifecycle; your TimeSeriesTable provider handles storage; your custom OptimizerRule handles the domain-specific pushdown. You wrote roughly 3,000 lines of storage code and 500 lines of glue; you did not write a SQL parser, an optimizer, a physical operator library, or an executor.

Output:

Component	Build yourself	Get from DataFusion
SQL parser	no	yes (sqlparser-rs)
Logical planner	no	yes
Rule-based optimizer	no	yes (extend with your rules)
Physical planner	no	yes
Volcano executor	no	yes
Arrow2 in-memory format	no	yes
Tokio async I/O	no	yes
Parquet reader	no	yes
Your storage layer	yes	plug in via TableProvider
Your domain optimizer rules	yes	plug in via OptimizerRule

Why this works — concept by concept:

• Shared-substrate architecture — DataFusion is the commodity plumbing (parser, planner, optimizer, executor, Arrow, Parquet); your team contributes only the storage layer and domain-specific optimizer rules. You inherit years of engine-quality work for free.
• TableProvider extension point — the load-bearing plug for storage. Implementing schema(), scan(), and supports_filter_pushdown() is enough to expose any source to the planner. Filter pushdown lets the planner skip whole files when the WHERE clause is selective.
• Custom OptimizerRule — the load-bearing plug for domain-specific rewrites. Your TimeSeriesDownsampleRule recognises the "downsample over window" pattern and pushes it into storage where the raw data can be aggregated on the way out, rather than shipping raw rows into the executor.
• Tokio async runtime — DataFusion is Tokio-native; every I/O in your TableProvider can be async. Object-store reads, network fetches, remote index lookups all fit naturally into the async model without blocking executor threads.
• Cost — one Cargo dependency (datafusion = "43"), plus the storage layer you were going to write anyway. The freed engineering budget (parser + planner + optimizer + executor = 5+ engineer-years) goes to your differentiator. Net saving is measured in engineer-decades over the product's lifetime.

SQL
Topic — sql
SQL query engine and columnar analytics problems

Practice →

Design Topic — design Design problems on embedded query engines

Practice →

---

2. DataFusion architecture — plan lifecycle over Arrow2

The four-stage lifecycle from SQL / DataFrame API to volcano-style execution stream

The one-sentence invariant: DataFusion's architecture is a four-stage plan lifecycle — a SQL string or DataFrame call becomes a logical plan tree via the plan builder, the logical plan is rewritten by rule-based optimizer passes, the optimized logical plan is lowered to a physical plan of ExecutionPlan operators, and the physical plan is executed as a volcano-style pull-based stream of Arrow2 RecordBatch objects flowing through Tokio async I/O — every extension point (TableProvider, UDF, UDAF, UDWF, OptimizerRule, custom operator) hooks into a specific stage of this lifecycle, and knowing which stage each hook lives at is the difference between someone who has embedded DataFusion and someone who has only read the docs.

The four stages senior interviewers name in order.

• Stage 1 — plan builder. A SQL string is parsed by sqlparser-rs into a SQL AST, then converted into a LogicalPlan tree. A DataFrame call (df.filter(...).aggregate(...)) builds the same LogicalPlan tree directly, skipping the parser. Both paths converge on the same intermediate representation; every downstream stage operates on that IR.
• Stage 2 — logical plan + optimizer. The LogicalPlan is a tree of typed operators (TableScan, Filter, Projection, Aggregate, Join, Sort, Limit). The optimizer runs a fixed-point loop of rule-based passes (filter pushdown, projection pushdown, join reorder, CSE, constant folding); each pass takes a LogicalPlan and returns a rewritten LogicalPlan.
• Stage 3 — physical plan. The physical planner lowers the optimized LogicalPlan into a tree of ExecutionPlan implementations. Where the logical plan says "Join," the physical plan picks a specific algorithm — hash join, sort-merge join, nested-loop join — based on statistics and hints. Where the logical plan says "TableScan," the physical plan resolves it to a specific TableProvider::scan() call.
• Stage 4 — execution stream. Each ExecutionPlan implements execute() which returns a SendableRecordBatchStream (a Tokio-friendly async stream of Arrow2 RecordBatches). The volcano executor pulls batches from the root of the stream tree; each operator pulls from its children on demand. Backpressure is natural; async I/O is native.

The Arrow2 RecordBatch — the unit of exchange between operators.

• What it is. A RecordBatch is a columnar batch of rows: one Schema plus an Array per column. All arrays are the same length (~1024-8192 rows per batch is the common range). This is Apache Arrow's canonical in-memory format, used by DataFusion, DuckDB (partially), Polars, and every Arrow-native tool.
• Why columnar. Vectorised operations on columnar data are 10-100× faster than row-by-row iteration because the CPU can SIMD-vectorise the tight loops. Every DataFusion physical operator is written to consume and produce RecordBatches; the vectorisation is built in.
• Why batched. Batches amortise per-batch overhead (task scheduling, backpressure signalling) over 1024+ rows. Too small and overhead dominates; too large and cache misses and latency dominate. DataFusion defaults to 8192.
• Zero-copy. RecordBatches can be shared between operators by Arc<RecordBatch> cloning — no memory copy, just a reference bump. This is the Rust ownership model paying dividends in a query engine.

Volcano pull-based execution — how batches flow.

• The model. Each operator is a state machine that responds to poll_next() calls. The executor polls the root; the root polls its child; the child polls its child; eventually a leaf TableScan reads a batch from storage and hands it up. Backpressure propagates naturally — the root doesn't ask for more batches than it can process.
• Why volcano. The alternative is whole-stage code generation (Spark's WholeStageCodegen, Photon), which compiles adjacent operators into a single tight loop for peak throughput. Volcano is slower per row but faster to start (no codegen delay), simpler to reason about, and better suited to embedded / short-lived queries.
• Tokio integration. SendableRecordBatchStream is a Stream in the futures crate sense. .next().await yields the next batch. This lets DataFusion sit naturally inside any Tokio-based async server without blocking a worker thread on I/O.
• The alternative. For workloads where startup latency doesn't matter and per-row throughput does, Spark's whole-stage codegen wins. For embedded, low-latency, short-lived queries, volcano wins. DataFusion picks volcano because its target deployments are the latter.

Extension points — where you plug in.

• TableProvider. Custom storage source. Implement schema(), scan(), supports_filter_pushdown(), statistics(); register with ctx.register_table("name", Arc::new(YourProvider)).
• ScalarUDF. Custom row-level function. Implement the function body over ColumnarValue, register with ctx.register_udf(...).
• AggregateUDF. Custom aggregate (UDAF). Implement state(), update(), merge(), evaluate().
• WindowUDF. Custom window function (UDWF). Same shape as UDAF plus a frame contract.
• OptimizerRule. Custom logical-plan rewrite rule. Called during the optimizer's fixed-point loop.
• ExecutionPlan. Custom physical operator. Implement execute() returning a stream; register with a matching physical-plan rewrite rule.

Common interview probes on architecture.

• "Walk me through the four stages of the DataFusion plan lifecycle." — required answer.
• "What is a RecordBatch and why is it columnar?" — required answer.
• "Why does DataFusion pick volcano over whole-stage codegen?" — senior signal.
• "How does async I/O fit into the executor?" — Tokio + SendableRecordBatchStream.
• "What's the difference between LogicalPlan and ExecutionPlan?" — logical is IR; execution is concrete.

Worked example — hello-world SQL over Parquet in Rust

Detailed explanation. The canonical DataFusion program: register a Parquet file as a table, run SQL against it, print the result. Every senior DE should be able to write this from memory. It exercises every stage of the plan lifecycle — parser, plan builder, optimizer, physical planner, executor — in fewer than 20 lines of Rust.

• Cargo dependency. datafusion = "43" and tokio = { version = "1", features = ["full"] }.
• API entry point. SessionContext::new().
• Table registration. ctx.register_parquet("tbl", "path.parquet", ...).
• Query. ctx.sql("SELECT ...").await?.show().await?.

Question. Write a complete DataFusion program that reads a Parquet file of orders, filters by status, groups by customer, and prints the top 10 customers by revenue.

Input.

Component	Value
Source	orders.parquet (local file)
Schema	id, customer_id, total_cents, status
Filter	status = 'shipped'
Aggregation	count + sum(total_cents) grouped by customer_id
Output	top 10 by revenue

Code.

// Cargo.toml
// [package]
// name = "datafusion-hello"
// version = "0.1.0"
// edition = "2021"
//
// [dependencies]
// datafusion = "43"
// tokio      = { version = "1", features = ["full"] }

use datafusion::prelude::*;

#[tokio::main]
async fn main() -> datafusion::error::Result<()> {
    // 1. Create the session context (catalog + config + runtime)
    let ctx = SessionContext::new();

    // 2. Register the Parquet file as a table
    ctx.register_parquet(
        "orders",
        "data/orders.parquet",
        ParquetReadOptions::default(),
    ).await?;

    // 3. Run a SQL query
    let df = ctx.sql("
        SELECT customer_id,
               COUNT(*)               AS order_count,
               SUM(total_cents)       AS revenue_cents
        FROM   orders
        WHERE  status = 'shipped'
        GROUP  BY customer_id
        ORDER  BY revenue_cents DESC
        LIMIT  10
    ").await?;

    // 4. Execute and print
    df.show().await?;
    Ok(())
}

Step-by-step explanation.

1. SessionContext::new() allocates a fresh session with the default catalog, default optimizer rules, default physical planner, and a Tokio runtime handle. All subsequent DataFusion API calls thread through this context.
2. register_parquet(...) uses the built-in ListingTable (a TableProvider implementation) to wrap the Parquet file. ParquetReadOptions::default() accepts the default file format inference; you can override to specify partition columns, custom schema, etc.
3. ctx.sql(...) walks the whole lifecycle. sqlparser-rs parses the SQL string into a SQL AST; the plan builder converts it to a LogicalPlan; the optimizer runs filter-pushdown (pushes status = 'shipped' into the Parquet scan), projection-pushdown (only reads customer_id, total_cents, status columns from Parquet), and aggregation-pushdown where applicable.
4. The df.show() call kicks off execution. The physical planner picks a hash aggregate for the GROUP BY, a sort-limit for the ORDER BY + LIMIT (since 10 is small, sort-limit is cheaper than full sort). The executor pulls Arrow2 RecordBatches from the leaf ParquetExec through each operator up to the root.
5. show() collects the batches, formats them as a table, and prints. In a real deployment you'd use df.collect().await? to get Vec<RecordBatch> for further processing, or df.execute_stream().await? to stream batches directly.

Output.

+-------------+-------------+---------------+
| customer_id | order_count | revenue_cents |
+-------------+-------------+---------------+
| 42          | 87          | 1275000       |
| 78          | 65          |  944500       |
| 11          | 54          |  812000       |
| 305         | 49          |  735400       |
| 907         | 41          |  621200       |
| ...         | ...         | ...           |
+-------------+-------------+---------------+

Rule of thumb. For any Rust program that needs SQL over Parquet, this 20-line skeleton is the starting point. It exercises the full plan lifecycle, ships as a single binary, and needs no JVM, no cluster, no driver process. Adding a second table, a JOIN, or a UDF is additive on top.

Worked example — custom TableProvider skeleton

Detailed explanation. The TableProvider trait is the load-bearing extension point for storage. Implementing it lets you plug any data source — a REST endpoint, a Redis hash, a proprietary columnar format, an in-memory dictionary — into the DataFusion catalog as if it were a normal table. The skeleton is short; the four load-bearing methods are schema(), scan(), supports_filter_pushdown(), and statistics(). Walk through the skeleton for an in-memory HashMap-backed table.

• The four methods. schema() returns the Arrow schema; scan() returns an ExecutionPlan that produces RecordBatches; supports_filter_pushdown() tells the optimizer which filters the provider can handle itself; statistics() gives the optimizer row-count / column-value distribution hints.
• Registration. ctx.register_table("name", Arc::new(YourProvider)).
• Filter pushdown. Returning TableProviderFilterPushDown::Exact for a filter tells the planner the provider will apply it (so the planner drops the redundant Filter operator).

Question. Implement a TableProvider skeleton for an in-memory table backed by a fixed Vec<RecordBatch>.

Input.

Method	Purpose
schema()	Arrow schema of the table
scan()	return an ExecutionPlan producing RecordBatches
supports_filter_pushdown()	tell the planner which filters the provider handles
statistics()	give row-count / min/max hints
table_type()	Base / View / Temporary

Code.

use std::any::Any;
use std::sync::Arc;
use async_trait::async_trait;
use arrow::datatypes::SchemaRef;
use arrow::record_batch::RecordBatch;
use datafusion::datasource::{TableProvider, TableType};
use datafusion::execution::context::SessionState;
use datafusion::physical_plan::memory::MemoryExec;
use datafusion::physical_plan::ExecutionPlan;
use datafusion::logical_expr::{Expr, TableProviderFilterPushDown};
use datafusion::error::Result;

/// A trivial in-memory TableProvider: wraps a Vec<RecordBatch>.
pub struct MemoryTable {
    schema:  SchemaRef,
    batches: Vec<RecordBatch>,
}

impl MemoryTable {
    pub fn new(schema: SchemaRef, batches: Vec<RecordBatch>) -> Self {
        Self { schema, batches }
    }
}

#[async_trait]
impl TableProvider for MemoryTable {
    fn as_any(&self) -> &dyn Any { self }

    fn schema(&self) -> SchemaRef { self.schema.clone() }

    fn table_type(&self) -> TableType { TableType::Base }

    async fn scan(
        &self,
        _state:      &SessionState,
        projection:  Option<&Vec<usize>>,
        _filters:    &[Expr],
        _limit:      Option<usize>,
    ) -> Result<Arc<dyn ExecutionPlan>> {
        // Return a MemoryExec that emits our pre-loaded batches.
        // projection tells us which columns the planner wants back.
        Ok(Arc::new(MemoryExec::try_new(
            &[self.batches.clone()],
            self.schema.clone(),
            projection.cloned(),
        )?))
    }

    fn supports_filters_pushdown(
        &self,
        filters: &[&Expr],
    ) -> Result<Vec<TableProviderFilterPushDown>> {
        // We can't handle any filter ourselves; let the planner do them.
        Ok(vec![TableProviderFilterPushDown::Unsupported; filters.len()])
    }
}

// Registration
// let table = Arc::new(MemoryTable::new(schema, batches));
// ctx.register_table("mem", table)?;

Step-by-step explanation.

1. schema() returns the Arrow schema, which the planner uses for type-checking and column resolution. This is the load-bearing "what does this table look like" method; every other extension point depends on it.
2. scan() is the load-bearing "how do we read from you" method. It receives a projection (which columns the planner wants), a slice of filter expressions (which the planner may want the provider to apply), and an optional limit. It returns an ExecutionPlan — for an in-memory table, MemoryExec::try_new(...) is the built-in helper.
3. supports_filters_pushdown() is the load-bearing "which filters can you handle" method. Return Exact for a filter your provider will apply itself (and the planner drops it); return Inexact if the provider applies a superset (planner keeps the filter for correctness); return Unsupported to leave the filter entirely to the planner. Getting this right unlocks whole-file skipping for columnar formats.
4. table_type() returns Base for a normal table, View for a view, Temporary for a session-lifetime table. Almost always Base for a real storage adapter.
5. Registration with ctx.register_table("name", Arc::new(YourProvider)) makes the table visible to ctx.sql("SELECT ... FROM name"). From that point the provider is indistinguishable from a built-in Parquet or CSV table from the SQL surface.

Output.

Registered table	Behind the scenes	User query
mem	MemoryTable provider wrapping Vec	SELECT * FROM mem
orders	ListingTable provider wrapping Parquet files	SELECT * FROM orders
kafka_events	Custom Kafka-backed provider	SELECT * FROM kafka_events
redis_cache	Custom Redis-backed provider	SELECT * FROM redis_cache

Rule of thumb. For any custom storage source, implement schema(), scan(), and supports_filters_pushdown() first; add statistics() later once you know the workload. The four-method skeleton is enough to expose any source to the full DataFusion SQL surface.

Senior interview question on DataFusion architecture

A senior interviewer might ask: "Walk me through the DataFusion plan lifecycle for the query SELECT customer_id, SUM(total_cents) FROM orders WHERE status = 'shipped' GROUP BY customer_id. Name each stage, what the input and output are at each stage, and where the Arrow2 RecordBatch enters the picture."

Solution Using the four-stage lifecycle trace with concrete IR shapes at each boundary

Stage 1 — SQL → SQL AST (sqlparser-rs)
======================================
Input:  "SELECT customer_id, SUM(total_cents) FROM orders
         WHERE status = 'shipped' GROUP BY customer_id"

Output: SQL AST:
  Select {
    projection: [customer_id, SUM(total_cents)],
    from:       [orders],
    selection:  Some(status = 'shipped'),
    group_by:   [customer_id],
  }

Stage 2 — SQL AST → LogicalPlan (plan builder)
==============================================
Output:
  Sort/Limit? — none
  └─ Aggregate: group_by=[customer_id], aggr=[SUM(total_cents)]
     └─ Filter: status = 'shipped'
        └─ TableScan: orders, projection=[customer_id, total_cents, status]

Stage 3 — LogicalPlan → Optimized LogicalPlan (optimizer rules)
================================================================
After PushDownFilter, PushDownProjection, TypeCoercion, SimplifyExpressions:

  Aggregate: group_by=[customer_id], aggr=[SUM(total_cents)]
    └─ TableScan: orders,
                   projection=[customer_id, total_cents],
                   filters=[status = 'shipped']    ← filter pushed
                                                     projection pruned

(status was dropped from projection because the filter pushed into
 the scan; the executor never materialises the status column.)

// Stage 4 — Optimized LogicalPlan → ExecutionPlan (physical planner)
// Rust representation (illustrative)
//
// AggregateExec: mode=Final, group_by=[customer_id], aggr=[SUM(total_cents)]
//   RepartitionExec: partitioning=Hash([customer_id], 8)
//     AggregateExec: mode=Partial, group_by=[customer_id], aggr=[SUM(total_cents)]
//       ParquetExec:
//         file_groups: [orders/part-*.parquet],
//         projection:  [customer_id, total_cents],
//         predicate:   status = 'shipped',
//         pruning:     enabled (uses Parquet page indexes)

// Stage 5 — ExecutionPlan → RecordBatch stream (volcano executor)
//
// let plan: Arc<dyn ExecutionPlan> = /* from above */;
// let stream: SendableRecordBatchStream = plan.execute(0, ctx.task_ctx())?;
//
// while let Some(batch) = stream.next().await {
//     let rb: RecordBatch = batch?;
//     // rb contains up to 8192 rows in columnar layout:
//     //   customer_id column: Arrow2 Int64Array
//     //   sum(total_cents):   Arrow2 Int64Array
// }

Step-by-step trace.

Stage	Input	Output	Where extensibility hooks in
1. Parser	SQL string	SQL AST	(rarely extended)
2. Plan builder	SQL AST	LogicalPlan	UDF resolution during binding
3. Optimizer	LogicalPlan	Optimized LogicalPlan	custom OptimizerRule inserted here
4. Physical planner	Optimized LogicalPlan	Arc<dyn ExecutionPlan>	custom ExecutionPlan + rewrite rule
5. Executor	ExecutionPlan	SendableRecordBatchStream of Arrow2	TableProvider::scan() returns leaf

After the trace, each stage's IR is a specific Rust type; each boundary is a specific method call on the DataFusion API. Extension points are anchored to specific stages: TableProvider at stage 5 (the scan leaf), OptimizerRule at stage 3, custom ExecutionPlan at stage 4. The Arrow2 RecordBatch enters at stage 5 (leaf) and flows upward through operator pipes until the executor consumes the root.

Output:

Metric	Value
Stages in lifecycle	4 planning + 1 execution
IR types	SQL AST, LogicalPlan, ExecutionPlan, RecordBatch
Optimizer passes (default)	~20 (filter/proj/join/CSE/const-fold/...)
Physical operator library	~30 (Hash/Sort/Filter/Project/Join/Agg/...)
Async model	Tokio Stream, backpressure natural
In-memory format	Arrow2 columnar (Int64Array, StringArray, ...)

Why this works — concept by concept:

• Four-stage plan lifecycle — separates concerns cleanly. Parsing is one component; planning is another; optimization is a third; execution is a fourth. Each has a well-typed input and output, which makes each replaceable and testable in isolation.
• LogicalPlan vs ExecutionPlan — the load-bearing distinction. LogicalPlan is IR: what to compute. ExecutionPlan is code: how to compute it. Optimizer rules operate on the "what"; physical planner picks the "how." Blurring them was the mistake older engines made.
• Arrow2 RecordBatch as the unit of exchange — columnar, batched, zero-copy shareable via Arc. Every operator produces and consumes the same type; the executor's job is trivial pipe-glue between typed streams.
• Volcano pull-based execution — natural backpressure, natural async fit, no codegen delay. The trade-off is per-row overhead versus whole-stage-codegen engines like Spark, but for embedded / short-lived queries that trade-off is the right one.
• Cost — O(operators) time for planning, O(rows / batch_size) volcano polls for execution, ~zero allocation churn thanks to Arrow's buffer sharing. Compared to a naive row-at-a-time interpreter, this is 10-100× faster; compared to whole-stage codegen it's typically 1.5-3× slower but embeds in an order of magnitude smaller footprint. O(1) startup latency versus O(seconds) for JVM alternatives.

SQL
Topic — sql
SQL plan-tree and query-execution problems

Practice →

ETL Topic — etl ETL problems on columnar Parquet pipelines

Practice →

---

3. DataFusion optimizer and expression rewriter

Rule-based passes that rewrite the logical plan tree — filter pushdown, projection pushdown, join reorder, CSE, constant folding

The one-sentence invariant: DataFusion's optimizer is a rule-based rewrite engine that runs a fixed-point loop of OptimizerRule passes over the LogicalPlan tree — canonical rules include filter pushdown (push WHERE predicates as close to the source as possible), projection pushdown (only read the columns downstream operators actually need), join reorder (put the smallest side on the build), common-subexpression elimination (compute x + y once, not four times), and constant folding (evaluate 1 + 2 at plan time not row time) — each rule is a fn optimize(plan: LogicalPlan) -> LogicalPlan and custom rules plug in by implementing OptimizerRule, which is why downstream projects like InfluxDB 3.0 can add time-series-specific rewrites without forking the engine.

The canonical rule set — the passes senior interviewers name.

• PushDownFilter. Pushes Filter operators as close to the TableScan as possible. Filter(status='shipped') → Join → TableScan becomes Join → Filter(status='shipped') → TableScan; when the TableProvider reports filter pushdown support, the filter is pushed into the scan and disappears from the plan entirely.
• PushDownProjection. Prunes columns unused by downstream operators. If the SQL is SELECT customer_id FROM orders, projection pushdown ensures the scan reads only the customer_id column from Parquet — a 10× speedup on wide tables where most columns are unused.
• JoinReorder / EliminateCrossJoin. Reorders join trees so the smaller side is the build side; eliminates cross joins where a join predicate exists elsewhere. DataFusion's join reordering is heuristic-based (rule-driven) rather than cost-based like Spark's CBO, but for typical queries it produces the right shape.
• CSE (CommonSubexpressionElimination). Recognises when the same subexpression appears multiple times (e.g. CASE WHEN x > 0 THEN y/x ELSE 0 in both SELECT and WHERE) and computes it once, aliasing the result. Cheap wins on complex expressions.
• ConstantFolding / SimplifyExpressions. Evaluates constant subexpressions at plan time. WHERE 1 = 1 becomes WHERE TRUE becomes nothing. col + 0 becomes col. NOT NOT x becomes x. Small optimizations that compound.
• TypeCoercion. Inserts implicit casts where the SQL surface allows loose types. WHERE integer_col = '42' becomes WHERE integer_col = 42 with a cast on the literal.
• EliminateLimit / PushDownLimit. Pushes LIMIT down through operators when safe. LIMIT 10 above a Sort combined with PushDownLimit can turn a full sort into a bounded top-K.

The fixed-point loop — how the rules interact.

• The mechanism. The optimizer runs each rule in order, then repeats until no rule produces a change (fixed point). This lets rules compose: ConstantFolding produces WHERE TRUE, which EliminateFilter then removes.
• The rule order. Configured in SessionState::optimizer_rules(). Custom rules can be inserted at specific positions in the pipeline. Position matters — a custom rule that produces a Filter should run before PushDownFilter.
• The termination guarantee. Rules must monotonically reduce plan complexity (or leave it unchanged) to guarantee termination. Cyclic rewrites (rule A produces plan X → rule B produces plan Y → rule A produces plan X) would hang the optimizer; the rule authoring guide bans them.
• The debugging tool. EXPLAIN VERBOSE prints the plan after each rule. This is your friend when a custom rule mysteriously doesn't fire or a stock rule produces an unexpected shape.

The Catalyst / Calcite / DataFusion comparison — the one senior interviewers probe.

• Catalyst (Spark). Written in Scala. Deeply integrated with Spark's whole-stage codegen. Cost-based optimizer for join reordering (CBO). Extensively battle-tested at TB / PB scale. The reference implementation of a rule-based optimizer in industry.
• Calcite (Trino, Flink). Written in Java. A framework, not an engine — Trino, Flink, Beam all use Calcite as their planner. Supports both rule-based and cost-based optimization; the cost model plugs into the framework.
• DataFusion. Written in Rust. Rule-based (no CBO yet in mainline 2026, though experimental cost-based rules exist in datafusion-optimizer). Deeply integrated with Arrow2 execution. Simpler than Catalyst but growing quickly; contributed to by the whole shared-substrate ecosystem.

The expression rewriter — a sub-system worth naming.

• What it is. A separate rewrite layer for Expr trees inside operators (as opposed to LogicalPlan trees). Simplifies x + 0 → x, NOT (a AND b) → NOT a OR NOT b (De Morgan), evaluates constant subexpressions, etc.
• Where it lives. Called from the SimplifyExpressions optimizer rule and from various physical planning steps.
• Why it matters. Expression rewriting is where CPU is saved per row — turning col + 0 - 0 into col at plan time skips two additions per row at execution time.

Common interview probes on the optimizer.

• "Name the top five DataFusion optimizer rules." — required answer.
• "Walk through filter pushdown for SELECT * FROM orders WHERE status = 'shipped'." — required answer.
• "How does DataFusion's optimizer compare to Catalyst?" — senior signal (rule-based vs rule+CBO).
• "How would you add a custom optimizer rule?" — implement OptimizerRule, insert into session state.
• "What is the fixed-point loop and why does it matter?" — termination + composition.

Worked example — tracing a filter through the optimizer

Detailed explanation. The clearest way to internalise DataFusion's optimizer is to trace one query end-to-end using EXPLAIN VERBOSE. Take a query with an obvious pushdown opportunity, print the plan before and after each rule, and observe what each rule does. Walk through the trace for SELECT id FROM orders WHERE status = 'shipped' AND total_cents > 1000.

• Initial plan. Projection(id) → Filter(status='shipped' AND total_cents > 1000) → TableScan(orders, *).
• After PushDownFilter. Filter pushed into TableScan (assuming Parquet supports filter pushdown).
• After PushDownProjection. TableScan projection pruned to [id, status, total_cents] (only these are needed).
• After further pruning. Because the filter is inside the scan, status and total_cents are pruned too — final projection is [id].

Question. Run EXPLAIN VERBOSE on the sample query and record the plan after each optimizer pass.

Input.

| SQL | SELECT id FROM orders WHERE status = 'shipped' AND total_cents > 1000 |
| Source | Parquet-backed orders(id, customer_id, status, total_cents, created_at) |
| Optimizer rules of interest | PushDownFilter, PushDownProjection, SimplifyExpressions |

Code.

use datafusion::prelude::*;

#[tokio::main]
async fn main() -> datafusion::error::Result<()> {
    let ctx = SessionContext::new();
    ctx.register_parquet("orders", "data/orders.parquet",
                         ParquetReadOptions::default()).await?;

    // EXPLAIN VERBOSE prints the plan after every optimizer pass
    let df = ctx.sql("
        EXPLAIN VERBOSE
        SELECT id
        FROM   orders
        WHERE  status = 'shipped'
          AND  total_cents > 1000
    ").await?;

    df.show().await?;
    Ok(())
}

# Approximate EXPLAIN VERBOSE output (abridged)

initial_logical_plan
  Projection: orders.id
    Filter: orders.status = Utf8("shipped") AND orders.total_cents > Int64(1000)
      TableScan: orders

after apply TypeCoercion
  Projection: orders.id
    Filter: orders.status = Utf8("shipped") AND orders.total_cents > Int64(1000)
      TableScan: orders

after apply SimplifyExpressions
  Projection: orders.id
    Filter: orders.status = Utf8("shipped") AND orders.total_cents > Int64(1000)
      TableScan: orders

after apply PushDownFilter
  Projection: orders.id
    TableScan: orders,
               full_filters=[orders.status = Utf8("shipped"),
                             orders.total_cents > Int64(1000)]

after apply PushDownProjection
  Projection: orders.id
    TableScan: orders,
               projection=[id, status, total_cents],
               full_filters=[orders.status = Utf8("shipped"),
                             orders.total_cents > Int64(1000)]

final_logical_plan
  Projection: orders.id
    TableScan: orders,
               projection=[id],                     ← further pruned
               full_filters=[orders.status = Utf8("shipped"),
                             orders.total_cents > Int64(1000)]

physical_plan
  ProjectionExec: expr=[id@0 as id]
    ParquetExec:
      file_groups: [orders/part-*.parquet],
      projection:  [id],                            ← Parquet only reads `id` column
      predicate:   orders.status = 'shipped' AND
                   orders.total_cents > 1000,
      pruning:     enabled (uses Parquet min/max page indexes)

Step-by-step explanation.

1. The initial plan mirrors the SQL AST — a Projection wrapping a Filter wrapping a TableScan. No optimizations have run yet; this is the "before" picture.
2. TypeCoercion and SimplifyExpressions are no-ops here because there's no type mismatch and no simplifiable expression. EXPLAIN VERBOSE still prints the plan after each pass, so you can confirm that the pass did nothing meaningful.
3. PushDownFilter is the load-bearing pass. It moves the Filter operator's predicate into the TableScan's full_filters list. This is a hint to the physical planner that the scan should apply the filter itself. For Parquet, this enables row-group pruning via min/max page indexes.
4. PushDownProjection first prunes to the columns the filter needs (id, status, total_cents); then further passes recognise that status and total_cents are consumed inside the scan and not needed above, so the projection prunes to just id. The Parquet reader only decodes the id column.
5. The physical plan shows a ParquetExec with projection: [id] and predicate: status = 'shipped' AND total_cents > 1000. The Parquet reader uses the predicate to skip row groups whose min/max ranges don't overlap; when a group is read, only the id column pages are decoded. This is the pattern that turns a "read the whole table" plan into a "read the tiny fraction of the table you actually need" plan.

Output.

Optimizer pass	Plan shape after	Effect
initial	Projection → Filter → TableScan	starting shape
TypeCoercion	(unchanged)	no coercions needed
SimplifyExpressions	(unchanged)	no simplifications available
PushDownFilter	Projection → TableScan(filters=...)	filter pushed into scan
PushDownProjection	Projection → TableScan(projection=[id], filters=...)	projection pruned
Physical planning	ProjectionExec → ParquetExec(projection=[id], predicate=...)	Parquet reads only id, prunes row groups

Rule of thumb. For any DataFusion query you want to understand or debug, run EXPLAIN VERBOSE. Track which optimizer passes are firing, which are noops, and what the final physical plan looks like. Nine out of ten "why is my query slow" investigations are answered by "the filter didn't push down because the TableProvider returned Unsupported."

Worked example — writing a custom OptimizerRule

Detailed explanation. The load-bearing extension point for domain-specific rewrites is OptimizerRule. Implement the trait, insert the rule into the session state, and it runs on every subsequent query. Walk through a rule that rewrites SELECT COUNT(*) FROM tbl WHERE ... into a metadata-only read for tables whose TableProvider reports exact row counts (a common trick for Iceberg / Delta tables where the manifest has row counts).

• The trait. fn try_optimize(&self, plan: &LogicalPlan, config: &dyn OptimizerConfig) -> Result<Option<LogicalPlan>>. Return Some(new_plan) if the rule fired; None if it didn't apply.
• The insertion. Modify SessionState's optimizer rules vector; add your rule after the built-in rules or at a specific position.
• The debug story. EXPLAIN VERBOSE shows plans after every pass; your custom rule shows up by name.

Question. Sketch a CountStarMetadataOnly optimizer rule that rewrites SELECT COUNT(*) FROM tbl (with no WHERE clause) to read the row count from statistics rather than scanning.

Input.

Component	Purpose
OptimizerRule trait	plug into the fixed-point loop
try_optimize method	return Some(new plan) if the rule matched
Match target	Aggregate(count(*)) directly on TableScan (no Filter)
Rewrite target	a synthesized single-row LogicalPlan with the stat value

Code.

use std::sync::Arc;
use datafusion::logical_expr::{
    Aggregate, LogicalPlan, TableScan,
    expr::{AggregateFunction},
};
use datafusion::optimizer::{OptimizerRule, OptimizerConfig};
use datafusion::common::Result;

/// Rewrite `SELECT COUNT(*) FROM tbl` to a metadata read
/// when the TableProvider reports exact row count and no filter is present.
pub struct CountStarMetadataOnly;

impl OptimizerRule for CountStarMetadataOnly {
    fn name(&self) -> &str { "count_star_metadata_only" }

    fn try_optimize(
        &self,
        plan:    &LogicalPlan,
        _config: &dyn OptimizerConfig,
    ) -> Result<Option<LogicalPlan>> {

        // Match: Aggregate(count(*)) directly on TableScan with no filters
        let LogicalPlan::Aggregate(Aggregate { input, aggr_expr, group_expr, .. }) = plan
            else { return Ok(None); };
        if !group_expr.is_empty() { return Ok(None); }
        if aggr_expr.len() != 1   { return Ok(None); }

        // Confirm it's count(*)
        // (details of matching the Expr elided — inspect for AggregateFunction::Count with args=[Literal(1)])

        let LogicalPlan::TableScan(TableScan { source, filters, .. }) = input.as_ref()
            else { return Ok(None); };
        if !filters.is_empty() { return Ok(None); }

        // Ask the TableProvider for exact row count via statistics()
        let stats = source.statistics().ok_or_else(|| /* err */ todo!())?;
        let Some(exact) = stats.num_rows.get_value() else {
            return Ok(None);   // not exact; let normal execution handle it
        };

        // Synthesize a one-row plan with the count constant
        // (details elided — build a LogicalPlan::Values with (exact,))
        //   LogicalPlan::Values(Values { schema, values: vec![vec![lit(exact)]] })
        todo!()
    }
}

// Register with the session
// let mut rules = ctx.state().optimizer().rules.clone();
// rules.push(Arc::new(CountStarMetadataOnly));
// (rebuild the SessionContext with the extended rule set)

Step-by-step explanation.

1. The rule matches only the narrow pattern of Aggregate(count(*)) directly on a TableScan with no filters and no group-by columns. Returning None for any non-match lets the optimizer proceed to the next rule; only exact matches trigger a rewrite.
2. The TableProvider::statistics() method is the load-bearing dependency. If the provider reports exact row counts (Iceberg manifest, Delta transaction log, sometimes Parquet if page indexes cover the whole file), we can synthesize the answer without executing. If the count is inexact, we bail — the normal execution path will produce the correct answer.
3. The rewrite target is a LogicalPlan::Values with a single row containing the count. Downstream planning turns this into a ValuesExec that emits one RecordBatch with one row — no I/O, no scan, no aggregation.
4. Registration means cloning the session's optimizer-rule vector, appending your rule, and rebuilding the session with the new rule set. In production code you'd factor this into a helper that also handles rule-order concerns (should your rule run before or after PushDownFilter?).
5. This rule is worth ~1000× on tables with cheap metadata: for a 100-million-row Iceberg table, SELECT COUNT(*) goes from "read every row group" (minutes) to "look up the manifest sum" (milliseconds). Custom optimizer rules are how downstream vendors (like InfluxDB 3.0 with its downsampling rewrite) buy back that kind of speedup without forking the engine.

Output.

Query	Without rule	With rule
SELECT COUNT(*) FROM iceberg_100m_rows	scans every row group	reads manifest count
SELECT COUNT(*) FROM iceberg_100m_rows WHERE ...	(rule doesn't apply — filter present)	(rule doesn't apply)
SELECT customer_id, COUNT(*) FROM ... GROUP BY 1	(rule doesn't apply — group-by present)	(rule doesn't apply)
Execution time delta	60s → 5ms	~12,000× speedup

Rule of thumb. Custom OptimizerRule implementations are how domain-specific engines (time-series, geospatial, ML feature stores) buy back optimizations without forking DataFusion. The pattern is always: match a narrow subtree shape, ask the storage layer if it can answer more efficiently, rewrite if yes, None if no. Never rewrite optimistically; you'll shoot yourself in correctness.

Senior interview question on the DataFusion optimizer

A senior interviewer might ask: "You're building a time-series query engine on DataFusion. Users write SELECT time_bucket('1m', ts), avg(value) FROM metrics WHERE ts >= now() - INTERVAL '1 hour' GROUP BY 1. Design a custom optimizer rule that pushes the time_bucket aggregation into the storage layer where pre-computed 1-minute rollups already exist. Show the rule, the plan before and after, and the interaction with the built-in rules."

Solution Using a custom RollupPushDown rule inserted after PushDownFilter with a downgrade-safe fallback

use std::sync::Arc;
use datafusion::logical_expr::{Aggregate, LogicalPlan, Expr};
use datafusion::optimizer::{OptimizerRule, OptimizerConfig};
use datafusion::common::Result;

/// Custom rule that recognises time_bucket(interval, ts) as the group
/// key on a metrics TableScan, and pushes the group-by + aggregate
/// into the TableProvider's scan() call.  Falls back gracefully when
/// the provider doesn't know how to compute a rollup for the given
/// interval (returns None; the built-in agg path handles it).
pub struct RollupPushDown;

impl OptimizerRule for RollupPushDown {
    fn name(&self) -> &str { "rollup_push_down" }

    fn try_optimize(
        &self,
        plan:    &LogicalPlan,
        _config: &dyn OptimizerConfig,
    ) -> Result<Option<LogicalPlan>> {

        // Match:  Aggregate(group_by=[time_bucket(interval, ts)],
        //                   aggr=[avg(value)])
        //   over  TableScan(metrics, filters=[ts >= <lower>])
        //   ...
        // Rewrite to a synthetic scan with (interval, aggr_specs) as scan hints.
        //
        // Full match elided; the pattern is:
        //   1. plan is Aggregate; group_expr matches ScalarUDF::time_bucket
        //   2. child is TableScan with a `filters` slot indicating time range
        //   3. TableScan's source (MetricsTable) offers offer_rollup(interval)
        //   4. If rollup available: build a new TableScan that reads pre-rolled
        //      RecordBatches directly, wrapped in a passthrough Aggregate for
        //      any final combines.
        //   5. If no rollup available: return Ok(None); fall through to stock
        //      PushDownFilter + built-in Aggregate.

        // (Skeleton — real implementation is ~200 lines but the *shape* is above)
        Ok(None)   // placeholder for skeleton
    }
}

// Session wiring — insert AFTER PushDownFilter so the ts >= <lower>
// filter is already in the scan before our rule inspects it
use datafusion::execution::context::{SessionContext, SessionConfig};
use datafusion::optimizer::optimizer::Optimizer;

fn build_ctx() -> SessionContext {
    let config = SessionConfig::new();
    let mut ctx = SessionContext::new_with_config(config);

    // (details of extending optimizer rules elided; the canonical pattern
    //  is to build a Vec<Arc<dyn OptimizerRule>> that starts with the
    //  defaults, inserts our rule after PushDownFilter, and rebuilds the
    //  SessionState.  See datafusion::optimizer::optimizer::Optimizer::with_rules.)

    ctx
}

# Plan traces before and after

BEFORE (with only built-in rules)
=================================
Aggregate: group_by=[time_bucket('1 minute', metrics.ts)],
           aggr=[avg(metrics.value)]
  TableScan: metrics,
             projection=[ts, value],
             filters=[metrics.ts >= now() - INTERVAL '1 hour']

Execution:
  1. ParquetExec reads every row of every file where ts >= <lower>
  2. AggregateExec computes the 1-minute buckets

AFTER (with RollupPushDown inserted after PushDownFilter)
=========================================================
Aggregate: group_by=[bucket],            ← "combine" step for edge alignment
           aggr=[avg_combine(avg_sum, avg_count)]
  TableScan: metrics_rollup_1m,          ← rewritten to the pre-aggregated table
             projection=[bucket, avg_sum, avg_count],
             filters=[bucket >= now() - INTERVAL '1 hour']

Execution:
  1. ParquetExec reads the pre-aggregated 1-minute rollup table
     (much smaller — ~60× fewer rows for a raw-vs-1m ratio)
  2. AggregateExec re-combines partial aggregates for correctness

Step-by-step trace.

Layer	Component	Behaviour
Rule registration	optimizer.rules.insert(after PushDownFilter, RollupPushDown)	position matters — filter must be in the scan first
Match	Aggregate(time_bucket(...)) on TableScan(metrics)	narrow pattern; skip anything else
Provider call	MetricsTable::offer_rollup('1 minute')	provider decides yes/no
Rewrite (yes)	swap TableScan source → metrics_rollup_1m	pre-aggregated data
Fallback (no)	return Ok(None)	stock path handles it
Correctness	combine step preserves boundary correctness	avg_sum + avg_count → avg

After the rule fires, queries against 1-minute rollups skip the raw-row scan entirely, reading the pre-aggregated table via a distinct TableProvider. When no rollup is available (e.g. user requests time_bucket('7 seconds', ts)), the rule returns None and DataFusion's stock path scans the raw table — the safety net that keeps correctness untouched.

Output:

Query	Without rule	With rule
time_bucket('1 minute', ts) over 1 hour	scan 60M raw rows	scan 60 rollup rows
time_bucket('5 minutes', ts) over 1 hour	scan 60M raw rows	scan 12 rollup rows
time_bucket('7 seconds', ts) over 1 hour	scan 60M raw rows	rule bails; scan 60M raw rows
Speedup	baseline	~1000× for supported intervals

Why this works — concept by concept:

• Custom OptimizerRule as the plug — the trait exposes exactly one method, try_optimize(plan) -> Result<Option<LogicalPlan>>. Return Some(rewritten) when you match; return None when you don't. The fixed-point loop retries until nobody rewrites; termination is guaranteed as long as each rewrite reduces plan complexity or leaves it unchanged.
• Rule position after PushDownFilter — critical. The filter has to already be in the TableScan before your rule inspects the scan, otherwise you can't tell what time range the query wants. Rule ordering is an intentional design choice, not an accident.
• Provider decides via offer_rollup — the custom TableProvider (your MetricsTable) exposes a domain-specific method that answers "do I have a pre-computed rollup for this interval?" If yes, the rule rewrites; if no, the rule bails. This keeps the optimizer rule tiny and pushes the domain logic into the provider.
• Correctness combine step — you can't just replace an avg(raw_value) with a scan of avg_sum/avg_count unless you also handle the "user's query window doesn't align perfectly with rollup boundaries" edge case. The synthetic avg_combine aggregate takes partial (sum, count) states and re-combines them for correctness. Getting this wrong ships silent incorrectness.
• Cost — for supported intervals, ~1000× speedup by scanning pre-aggregated tables. For unsupported intervals, Ok(None) means zero cost (rule adds a few microseconds of match-attempt overhead per query). The eliminated cost is the raw-row scan; the retained cost is a small combine step for boundary alignment. Big-O reduction from O(raw_rows) to O(rolled_rows) — the raw:rolled ratio is your speedup ceiling.

SQL
Topic — sql
SQL optimizer and query-rewrite problems

Practice →

Design Topic — design Design problems on rule-based optimizers

Practice →

---

4. Downstream projects and vendor adoption

One Rust engine, many downstreams — InfluxDB 3.0, Comet, Ballista, Sail, GreptimeDB, ROAPI, Cube.dev

The one-sentence invariant: DataFusion's business impact comes not from the engine itself but from the ecosystem of downstream projects that embed it as a shared substrate — InfluxDB 3.0 (IOx) rebuilt its time-series query layer on DataFusion, Comet accelerates Spark stages by routing them through DataFusion at the JVM ⇄ native boundary, Ballista wraps DataFusion for distributed execution on Kubernetes, Sail offers a distinct distributed-Rust deployment shape, and GreptimeDB, ROAPI, Cube.dev, ParadeDB, and Coralogix all ship DataFusion inside their products — and the pattern is the "shared substrate" story: bugs fixed in DataFusion by any vendor are fixed for everyone, contribution flows in from the whole ecosystem, and no single vendor bears the cost of maintaining a query engine alone.

InfluxDB 3.0 (IOx) — the flagship rewrite.

• The story. Paul Dix's team rebuilt InfluxDB 3.0 from scratch on Rust + Arrow + DataFusion + Parquet, replacing the Go-based InfluxDB 2 engine. The result: sub-second queries at scale, 100× ingest improvement, Parquet-native storage, unlimited cardinality (fixing InfluxDB 2's cardinality-blowup problem).
• The architecture. Ingestion writes into an Arrow2-backed in-memory buffer; buffers flush to Parquet in object storage; queries route through DataFusion, which reads the buffer + the Parquet files, applies the optimizer's filter/projection pushdown, and returns Arrow2 RecordBatches to the InfluxQL / SQL / FlightSQL surface.
• Why DataFusion. Time-series query languages (InfluxQL, Flux, SQL) all boil down to filters, aggregations, and window functions — the exact operators DataFusion already implements. IOx contributes time-series-specific optimizer rules (like the RollupPushDown sketch in section 3) but doesn't rewrite the engine.
• The public documentation. Paul Dix keynote at KubeCon 2022; InfluxData tech-talks 2023-2024; the influxdb3 GitHub organisation; multiple published blog posts on "why we chose DataFusion." This case is the reference story for the shared-substrate argument.

Comet — the Spark Rust accelerator.

• The story. Comet is Apache DataFusion Comet, an accelerator for Apache Spark that swaps Spark's JVM physical operators for DataFusion's native Rust operators on a per-stage basis. Published in SIGMOD 2024 by the Apple team.
• The architecture. A Spark query plans normally through Catalyst. Before physical execution, Comet's plan-rewriting layer inspects each stage; if all operators are supported by DataFusion, Comet compiles the stage to a native execution graph, ships Arrow2 batches across the JVM ⇄ native boundary via JNI, and executes the stage in Rust. Non-supported stages fall back to Spark's JVM executor transparently.
• Why DataFusion. Rewriting Spark's operators in Rust would take years. Adopting DataFusion means Comet inherits a full physical operator library, an Arrow-native execution model, and Tokio async I/O — for free. Comet's engineering work is the plan-rewriting bridge, not the engine.
• The performance win. TPC-H benchmarks in the SIGMOD paper show 2-5× per-stage speedup for CPU-bound stages, with the win coming from Arrow's vectorised execution and Rust's lack of JVM GC pauses. The bridge overhead (JNI + Arrow batch marshalling) is amortised across the stage.

Ballista — the distributed DataFusion runner.

• The story. Ballista is the "Spark for DataFusion" — a Kubernetes-native distributed runner that partitions a DataFusion physical plan across worker pods. It grew out of the original DataFusion project (which was itself originally the Ballista sub-project of Arrow — the naming has confused everyone).
• The architecture. A scheduler pod plans queries the same way an embedded DataFusion does; the physical plan is partitioned into stages; each stage's tasks are shipped to executor pods; executors run DataFusion natively and shuffle intermediate results back through the scheduler.
• Why Ballista instead of Spark. For all-Rust shops that want a distributed engine without a JVM footprint. Ballista is production-viable in 2026 for scoped analytics workloads but does not have Spark's connector ecosystem, streaming maturity, or 15 years of tuning. The right question is "does my workload fit Ballista's scope?" rather than "is Ballista as good as Spark?"
• The distinguishing feature. Serialisation of plans via Substrait — an emerging cross-engine IR. Ballista serialises DataFusion physical plans as Substrait, which means Ballista could (in principle) execute plans produced by any Substrait-emitting frontend, not just DataFusion.

Sail — the alternative distributed shape.

• The story. Sail is a distinct distributed-Rust engine that also embeds DataFusion, focused on Spark-API compatibility. Where Ballista offers a Kubernetes-native shape, Sail offers a Spark-compatible shape (PySpark-callable, similar deployment story).
• Why it exists. Different vendors have different opinions on the right distributed shape. Sharing DataFusion as the substrate lets each of them ship without duplicating the engine work.
• The choice for your team. Neither Ballista nor Sail is a de-facto standard yet. If you need distributed Rust query execution, evaluate both against your workload; if you need PySpark API compatibility, Sail leans that way; if you need Kubernetes-native scheduling, Ballista leans that way.

GreptimeDB, ROAPI, Cube.dev, ParadeDB, Coralogix — the long tail.

• GreptimeDB. Rust-native, cloud-native, time-series + IoT + logs. Uses DataFusion for the query layer; adds its own storage engine (region-based, tiered).
• ROAPI. Turns any file (Parquet, CSV, JSON, ...) into a REST + GraphQL + FlightSQL API. Uses DataFusion for the query engine; the ROAPI value-add is the API layer over DataFusion's SessionContext.
• Cube.dev. Semantic layer for BI tools. Embeds DataFusion as one of several backend query engines; contributes to the shared repo.
• ParadeDB. Postgres-compatible search + analytics. Uses DataFusion for the analytics half; wraps it inside a Postgres extension.
• Coralogix. Log observability platform. Uses DataFusion for query execution over columnar log storage; contributes optimizer rules for log-specific patterns.
• The pattern. Every vendor gets a full query engine as a Cargo dependency; every vendor contributes back domain-specific extensions; the shared substrate compounds in quality every year.

Common interview probes on downstream adoption.

• "Name three downstream projects that embed DataFusion." — required answer.
• "Why did InfluxDB 3.0 pick DataFusion instead of building their own engine?" — senior signal (shared substrate).
• "What is Comet and how does it accelerate Spark?" — swaps operators, JVM ⇄ native via JNI.
• "What's the difference between Ballista and Sail?" — different distributed shapes.
• "When would you pick DataFusion + Ballista over Spark?" — Rust-native shop, scoped workload, no JVM tolerance.

Worked example — how Comet swaps Spark operators for DataFusion physical operators

Detailed explanation. The clearest picture of "shared substrate in practice" is Comet's stage-level substitution mechanism. Walk through how a Spark query plans normally, how Comet's rewrite layer decides to substitute native DataFusion operators, how batches cross the JVM ⇄ native boundary via JNI + Arrow, and how the fallback path handles operators DataFusion doesn't yet support.

• Step 1. Spark parses SQL and produces a Catalyst logical plan.
• Step 2. Spark's physical planner produces a Spark physical plan with SparkPlan operators.
• Step 3. Comet's CometExecRule inspects each SparkPlan; if every operator in the stage has a DataFusion equivalent, Comet wraps the stage in CometNativeExec.
• Step 4. At execution time, CometNativeExec marshals input Arrow batches across JNI, executes the stage in Rust via DataFusion's operators, marshals output batches back to the JVM.
• Step 5. Non-native stages fall back to Spark's regular execution transparently.

Question. Sketch the Comet substitution logic and describe what happens per stage when a query mixes supported and unsupported operators.

Input.

Stage	Operators	Supported by DataFusion?	Comet decision
1	Scan(Parquet) + Filter + Project	yes	wrap in CometNativeExec
2	HashAggregate	yes	wrap in CometNativeExec
3	Custom UDF (Scala)	no	leave as Spark stage
4	Sort + Limit	yes	wrap in CometNativeExec

Code.

// Comet's substitution rule (illustrative Scala sketch)
object CometExecRule extends Rule[SparkPlan] {

  override def apply(plan: SparkPlan): SparkPlan = {
    plan.transformUp {
      case p if isFullyNative(p) =>
        // Every operator in this subtree has a DataFusion equivalent;
        // wrap it as a native stage.
        CometNativeExec(convertToNativePlan(p))
      case p =>
        // Some operator (e.g. Scala UDF) is not supported natively.
        // Leave the subtree in Spark; the parent JVM executor runs it.
        p
    }
  }

  private def isFullyNative(plan: SparkPlan): Boolean =
    plan.forall {
      case _: FileSourceScanExec  => true
      case _: FilterExec          => true
      case _: ProjectExec         => true
      case _: HashAggregateExec   => true
      case _: SortExec            => true
      case _: LimitExec           => true
      case _                      => false   // conservative bail
    }
}

// Native side (illustrative): DataFusion physical plan execution
// invoked via JNI from CometNativeExec
use datafusion::physical_plan::ExecutionPlan;
use arrow::record_batch::RecordBatch;

#[no_mangle]
pub extern "C" fn comet_execute_stage(
    plan_bytes: *const u8,
    plan_len:   usize,
    input:      *const RecordBatchFFI,
) -> *mut RecordBatchFFI {
    // 1. Deserialise the Comet-shipped physical plan (Substrait-encoded)
    let plan: Arc<dyn ExecutionPlan> = deserialise_from_substrait(unsafe {
        std::slice::from_raw_parts(plan_bytes, plan_len)
    });

    // 2. Wrap the JNI-input Arrow batches as a MemoryExec
    let input_batches = unsafe { arrow_from_ffi(input) };
    let ctx = build_session_with_input(input_batches);

    // 3. Execute; volcano-pull to a Vec<RecordBatch>
    let batches = pollster::block_on(execute_all(plan.clone(), ctx));

    // 4. Marshal back across JNI
    arrow_to_ffi(batches)
}

Step-by-step explanation.

1. Spark's Catalyst produces a logical plan and a physical plan exactly as it always does. Comet does not touch the parsing, the logical plan, or the initial physical planning — it operates strictly on the produced Spark physical plan.
2. CometExecRule.apply(plan) walks the physical plan bottom-up (transformUp). At each node it checks isFullyNative(p): is every operator in this subtree supported by DataFusion? If yes, it wraps the subtree in CometNativeExec; if no, it leaves the subtree in Spark.
3. The isFullyNative check is conservative — an unknown operator causes the whole subtree to bail. This keeps the substitution safe: Comet never accidentally omits a Spark-specific operator (like a custom Scala UDF) that DataFusion couldn't handle.
4. At execution time, CometNativeExec receives Spark's Arrow input batches, ships them across JNI to Rust via Arrow's C Data Interface (zero-copy where possible), runs the native DataFusion physical plan to completion, and ships the output batches back.
5. The JVM ⇄ native boundary is the load-bearing operational risk. Bridge overhead (JNI + Arrow marshalling) is fixed per batch; if a stage produces very small batches (1000 rows each), the bridge overhead can eat the native execution win. Comet's tuning story is largely "make batches bigger before crossing the boundary."

Output.

Query mix	Comet substitution result	Speedup
100% supported operators	every stage native	2-5× (SIGMOD paper)
50% supported	50% stages native, 50% JVM	1.3-2× amortised
100% unsupported (custom UDFs everywhere)	no substitution; 1.0×	1.0× (Comet is transparent)
Wide table scans + filters + aggs	dominant stages native	3-4× typical

Rule of thumb. Comet is the case study for "shared substrate reduces engineering scope." Comet's team wrote the plan-rewriting bridge and the JNI marshalling; the physical operators, the optimizer, the Arrow execution — all came from DataFusion. That's the shared-substrate payoff at scale: a two-person team can ship a Spark accelerator because they don't have to rewrite Spark's operators.

Worked example — the vendor decision tree "build vs pick DataFusion"

Detailed explanation. Every senior architect facing "we need a query engine" faces the "build vs pick DataFusion" decision. Walk through the tree that senior architects use to make the call, with three canonical scenarios: a Rust time-series database, a Java-based data lake gateway, and a C++ real-time OLAP.

• Q1. Is your product's primary language Rust? → yes = strong candidate for DataFusion; no = go to Q2.
• Q2. Is the JVM acceptable? → yes = Catalyst / Calcite / Trino; no = go to Q3.
• Q3. Is C++ your primary language? → yes = Velox; no = you're in a niche language and probably need to build.
• Q4. Is your workload scoped enough that DataFusion's SQL / DataFrame surface covers 80%+ of what users need? → yes = pick DataFusion + custom TableProvider + custom rules; no = you may need to fork or build.
• Q5. Does your differentiation come from the query engine itself? → yes = build (rare); no = pick DataFusion (common).

Question. Walk the tree for the three scenarios and record the recommended substrate for each.

Input.

Scenario	Q1 (Rust?)	Q2 (JVM ok?)	Q3 (C++?)	Q4 (scope covered?)	Q5 (engine is differentiator?)
Rust time-series DB (InfluxDB 3-shaped)	yes	—	—	yes	no
Java data-lake gateway	no	yes	—	yes	no
C++ real-time OLAP (ClickHouse-shaped)	no	no	yes	no	yes

Code.

# Decision tree helper (illustrative)
def pick_substrate(is_rust: bool,
                    jvm_ok: bool,
                    is_cpp: bool,
                    scope_covered: bool,
                    engine_is_differentiator: bool) -> str:
    if engine_is_differentiator:
        return "build custom"

    if is_rust:
        return "DataFusion" if scope_covered else "DataFusion + significant custom rules"

    if jvm_ok:
        return "Catalyst (embed Spark), Calcite (framework), or Trino"

    if is_cpp:
        return "Velox"

    return "you are in a niche language; consider a service call to DataFusion / Trino"


# Walk the three scenarios
print(pick_substrate(True,  False, False, True,  False))
# → 'DataFusion'

print(pick_substrate(False, True,  False, True,  False))
# → 'Catalyst (embed Spark), Calcite (framework), or Trino'

print(pick_substrate(False, False, True,  False, True))
# → 'build custom'

Step-by-step explanation.

1. Scenario 1 — Rust time-series DB. Q1 = yes; Q4 = yes (time-series queries are filters + aggregations + windows, all covered); Q5 = no (differentiation is storage, not engine). The tree short-circuits at Q1 → DataFusion. InfluxDB 3.0 walked exactly this tree in 2020-2022 and arrived at the same answer.
2. Scenario 2 — Java data-lake gateway. Q1 = no; Q2 = yes; scope covered; engine not the differentiator. Trino is the pragmatic pick because it already has a mature connector ecosystem for data-lake sources. Catalyst / Calcite are the frameworks if you want to build a custom Java engine; Trino is the shipped answer if you want to just deploy something.
3. Scenario 3 — C++ real-time OLAP. Q1 = no; Q2 = no; Q3 = yes; but Q5 = yes (ClickHouse's differentiator is the columnar OLAP engine). This tree lands on "build custom" because the engine is the product. Velox would be the choice if the differentiator were storage instead of engine.
4. The load-bearing Q is Q5. If your engine is your differentiator, you build. If your engine is commodity plumbing that lets you focus on a different differentiator (storage, distribution, connectors, semantic layer), you pick a substrate. Most vendors are in the second category.
5. The "no engine is your differentiator" answer is the shared-substrate payoff. Vendors who realise their differentiator is elsewhere save 5+ engineer-years by embedding DataFusion (Rust), Velox (C++), or Catalyst/Calcite (JVM) instead of writing their own.

Output.

Scenario	Substrate	Engineering saved
Rust time-series DB	DataFusion	~5 engineer-years
Java data-lake gateway	Trino (Calcite-based)	~5 engineer-years
C++ real-time OLAP where engine IS the product	build custom	none (correct call)
Semantic layer with pluggable backends	DataFusion (one of several)	~3 engineer-years per backend

Rule of thumb. Answer Q5 honestly. If your product's differentiator is the query engine itself, build. If it's anything else (storage, connectors, distribution, semantic layer, UX), pick a substrate. Nine out of ten vendors are in the second category; being honest about that is what senior architects do.

Senior interview question on downstream adoption

A senior interviewer might ask: "We're building a Rust-based observability platform. We need SQL over columnar log storage, sub-second query latency, and pluggable custom aggregations for log-specific patterns (regex extract, top-K by tag). Walk me through why you'd embed DataFusion, what you'd contribute back, and how you'd position your product against Coralogix (who also uses DataFusion)."

Solution Using DataFusion as the substrate with custom TableProvider, custom UDFs, and product differentiation on storage + UX

// Cargo.toml
// [dependencies]
// datafusion = "43"
// arrow      = "53"
// tokio      = { version = "1", features = ["full"] }
// object_store = "0.11"

use std::sync::Arc;
use datafusion::prelude::*;
use datafusion::logical_expr::{create_udaf, Volatility};

// 1. Custom TableProvider for our columnar log storage
//    (skeleton — details of the real storage layer elided)
pub struct LogStorageTable {
    // schema: Arrow schema; partitions: tiered storage handles
}

// impl TableProvider for LogStorageTable { ... }

// 2. Custom UDAF — top_k_by_tag(tag_col, k)
fn register_top_k_by_tag(ctx: &SessionContext) -> datafusion::error::Result<()> {
    // Full UDAF impl elided; the shape is:
    //   state:    priority queue of (tag, count)
    //   update:   increment count for row's tag
    //   merge:    combine partial priority queues
    //   evaluate: return top-k tags by count
    //
    // let top_k = create_udaf(...);
    // ctx.register_udaf(top_k);
    Ok(())
}

// 3. Wire everything into a SessionContext
#[tokio::main]
async fn main() -> datafusion::error::Result<()> {
    let ctx = SessionContext::new();

    // Register the custom TableProvider
    let logs = Arc::new(LogStorageTable { /* ... */ });
    ctx.register_table("logs", logs)?;

    // Register the custom UDAF
    register_top_k_by_tag(&ctx)?;

    // Users write standard SQL against our storage + our aggregations
    let df = ctx.sql("
        SELECT service,
               regex_extract(message, '(\\w+-\\w+-\\w+)') AS trace_id,
               COUNT(*) AS n
        FROM   logs
        WHERE  ts >= now() - INTERVAL '15 minutes'
          AND  level = 'ERROR'
        GROUP  BY service, trace_id
        ORDER  BY n DESC
        LIMIT  50
    ").await?;

    df.show().await?;
    Ok(())
}

Step-by-step trace.

Component	Origin	Purpose
SQL / DataFrame surface	DataFusion (free)	user-facing query API
Logical plan + optimizer	DataFusion (free)	rewrite / pushdown
Physical operators (join, agg, sort, filter, project)	DataFusion (free)	volcano execution
Arrow2 in-memory format	DataFusion (free)	columnar batches
Custom LogStorageTable (TableProvider)	us	our differentiator: tiered columnar log storage
Custom top_k_by_tag UDAF	us	our differentiator: log-specific aggregation
Custom optimizer rule for time-range pushdown	us + contributed back	shared with Coralogix, GreptimeDB, etc.

After the deployment, our observability platform ships with a full SQL query surface, sub-second latencies (Rust-native, Tokio-async, zero-JVM), and a growing library of log-specific UDAFs. We contribute the time-range pushdown rule to DataFusion upstream, benefiting Coralogix and GreptimeDB in the same pass. Our product differentiation is the storage layer and the UX; we do not compete with Coralogix on query-engine quality — we both benefit from the same substrate.

Output:

Layer	Ours	DataFusion	Contributed back
SQL parser	no	yes	no
Logical plan builder	no	yes	no
Optimizer core	no	yes	no
Time-range pushdown rule	yes	(accepted upstream)	yes
Physical operators	no	yes	no
Arrow2 execution	no	yes	no
LogStorageTable (TableProvider)	yes	plug point only	no (product IP)
Log-specific UDAFs (top_k_by_tag, regex_extract, ...)	yes	plug point only	some contributed
Tiered storage layer	yes	out of scope	no (product IP)
UX + dashboards + alerts	yes	out of scope	no (product IP)

Why this works — concept by concept:

• Shared substrate reduces engineering scope — we ship a query engine as a Cargo dependency (datafusion = "43"). Our engineering budget goes to the storage layer, the log-specific aggregations, and the UX — all product differentiators. We do not compete with Coralogix on query engine quality; we both benefit from it.
• TableProvider is the storage plug — our LogStorageTable exposes tiered columnar log storage to DataFusion. Filter pushdown lets time-range and tag filters skip whole storage segments. Statistics let the optimizer make good decisions.
• Custom UDAFs are the aggregation plug — top_k_by_tag, regex_extract, and other log-specific functions live in our codebase but expose through DataFusion's normal SQL surface. Users write standard SQL; the UDAFs handle the log-shaped work.
• Contribution flows both ways — the time-range pushdown rule is generic enough to benefit any time-series or log workload. We contribute it upstream; Coralogix and GreptimeDB inherit the improvement; DataFusion becomes stronger; we get their fixes in return. This is the compounding benefit of a shared substrate.
• Cost — one Cargo dependency, plus the storage layer and UDAFs we were going to write anyway. Compared to writing a query engine from scratch, this is ~5 engineer-years of savings. Compared to shipping without a query engine, this is a product feature we would not otherwise have. Net: shared-substrate adoption is a two-orders-of-magnitude reduction in engine work; the engineering budget goes to differentiation. O(1) cost to inherit the whole engine; O(N) contribution cost for domain-specific extensions.

Streaming
Topic — streaming
Streaming and time-series query problems

Practice →

Design Topic — design Design problems on observability query engines

Practice →

---

5. Using DataFusion directly and interview signals

datafusion-python, embedded Rust engines, DataFusion vs DuckDB vs Polars, and the interview probes senior engineers land

The one-sentence invariant: DataFusion is usable directly in three shapes — as a Rust library embedded inside your binary (cargo add datafusion), as a Python library via datafusion-python for interactive SQL over Parquet, and as an embedded engine inside a larger application (analytics service, edge processor, WASM module) — and the interview probes that separate senior candidates from juniors are (1) can you place DataFusion accurately versus DuckDB and Polars on the four axes, (2) can you name the plan lifecycle stages, (3) can you name the extension points, and (4) can you articulate the shared-substrate trend rather than treating DataFusion as an isolated tool.

Using DataFusion from Python — datafusion-python.

• The library. pip install datafusion. Wraps the Rust engine via PyO3 bindings; the Python API mirrors the Rust API closely.
• What it's for. Interactive SQL over Parquet from Python. Notebook analytics without a JVM. Local development against S3-hosted Parquet before a Rust deployment. Sanity-checking a query before wiring it into a Rust binary.
• What it's not for. Production data-frame pipelines that need pandas-shaped ergonomics — Polars is a better pick there. Distributed workloads — go to Ballista/Sail or Spark.
• The three-line hello. from datafusion import SessionContext, ctx.register_parquet("t", "path"), ctx.sql("...").show(). Same lifecycle as the Rust API; same optimizer; same executor.

Using DataFusion as an embedded engine in Rust.

• The pattern. cargo add datafusion in your Rust binary. Instantiate SessionContext. Register your tables. Run SQL. Consume RecordBatches.
• The deployment shape. Single static binary; no cluster; no driver process; no JVM; ~5-10 MB of dependencies at compile time; ~5-50 MB of RAM at runtime for a simple session.
• The use cases. Analytics service that queries Parquet on S3. Edge processor that runs SQL over local sensor data. WASM module that runs SQL in the browser. CLI tool that runs SQL over CSV. Every case where "just spawn a query engine inside my binary" is the answer.
• The tuning knobs. SessionConfig::batch_size (default 8192); .with_target_partitions(N) for parallelism; custom optimizer rules via SessionState.

DataFusion vs DuckDB vs Polars — the three-way comparison senior interviewers probe.

• DataFusion. Rust-native, SQL-first (DataFrame second), embeddable + distributable (via Ballista/Sail), plan-tree extensible. Wins when the requirement includes "Rust binary" and "SQL surface" and "extensible plan tree."
• DuckDB. C++-native, SQL-first, embeddable single-node only, extension mechanism for C++. Wins when the requirement is "single-node analytics on Parquet, no Rust-embedding needed, best-in-class SQL performance." DuckDB benchmarks slightly better than DataFusion on most TPC-H queries; the trade-off is language and extensibility surface.
• Polars. Rust-native, DataFrame-first (SQL is secondary), embeddable single-node (distributed roadmap), less pluggable plan tree. Wins when the requirement is "pandas-shaped Python DataFrames, Rust performance under the hood, no SQL needed." Polars is faster than pandas by 10-100× and is the pragmatic pandas replacement.
• The picker. SQL + Rust embedding + extensibility → DataFusion. SQL + best single-node perf + C++ ok → DuckDB. DataFrame Python + Rust perf → Polars. Distributed + Rust-native → Ballista/Sail (DataFusion under the hood). Distributed + JVM ok → Spark.

The six interview probes that separate senior candidates.

• Probe 1 — Arrow2 columnar format. "Why is Arrow2 the right in-memory format?" Answer: columnar layout enables SIMD vectorisation, batching amortises per-batch overhead, zero-copy sharing via Arc<RecordBatch> avoids memory churn, interop with any Arrow-native tool comes for free.
• Probe 2 — plan lifecycle. "Walk me through the four stages: SQL AST → LogicalPlan → optimizer → ExecutionPlan → RecordBatch stream." Answer: parser (sqlparser-rs), plan builder, optimizer (rule-based fixed point), physical planner, executor (volcano pull with Tokio async).
• Probe 3 — optimizer rules. "Name the top five optimizer rules." Answer: PushDownFilter, PushDownProjection, SimplifyExpressions, EliminateFilter, JoinReorder. Bonus: describe how to write a custom OptimizerRule.
• Probe 4 — TableProvider. "How would you plug a new storage source into DataFusion?" Answer: implement TableProvider; the four load-bearing methods are schema(), scan(), supports_filters_pushdown(), statistics().
• Probe 5 — UDF/UDAF/UDWF. "How do you add a custom aggregation?" Answer: create_udaf(name, input_types, return_type, volatility, accumulator_factory, state_type); register with ctx.register_udaf(...).
• Probe 6 — shared-substrate trend. "Why did InfluxDB 3.0 pick DataFusion instead of building their own?" Answer: shared substrate reduces engineering scope; downstream vendors focus on storage and differentiation, not on rebuilding a query engine.

Common interview probes on direct usage.

• "Show me hello-world DataFusion in Python." — required (three lines).
• "When would you pick DataFusion over DuckDB?" — required (Rust embedding + extensibility).
• "When would you pick DataFusion over Polars?" — required (SQL surface + plan-tree extensibility).
• "How do you benchmark DataFusion against DuckDB on TPC-H?" — senior signal (name TPC-H Q1 or similar).
• "What's the memory footprint of an embedded DataFusion session?" — senior signal (single-digit MB).

Worked example — datafusion-python hello world

Detailed explanation. The clearest way to try DataFusion is via datafusion-python. Three lines get you a query engine over Parquet; a fourth prints the result. Walk through the hello world and describe when this is the right tool.

• Install. pip install datafusion.
• API. from datafusion import SessionContext.
• Query. ctx.register_parquet("t", "path") then ctx.sql("...").
• When to use. Interactive analytics on Parquet from Python; sanity-checking queries before Rust deployment; notebook analytics without spinning up Spark.

Question. Write a Python program that queries a Parquet file of orders, groups by customer, sums revenue, and prints the top 10.

Input.

Component	Value
Parquet source	orders.parquet
Filter	status = 'shipped'
Group by	customer_id
Aggregation	count + sum(total_cents)
Output	top 10 by revenue

Code.

# pip install datafusion pyarrow
from datafusion import SessionContext

ctx = SessionContext()

# 1. Register the Parquet file as a table
ctx.register_parquet("orders", "data/orders.parquet")

# 2. Run SQL — same lifecycle as the Rust API
df = ctx.sql("""
    SELECT customer_id,
           COUNT(*)         AS order_count,
           SUM(total_cents) AS revenue_cents
    FROM   orders
    WHERE  status = 'shipped'
    GROUP  BY customer_id
    ORDER  BY revenue_cents DESC
    LIMIT  10
""")

# 3. Collect and print
df.show()

# 4. (Optional) Convert to a pyarrow.Table or pandas.DataFrame
arrow_table = df.to_arrow_table()
# import pandas as pd; df_pd = arrow_table.to_pandas()

Step-by-step explanation.

1. SessionContext() is the Python equivalent of the Rust SessionContext::new(). Under the hood it holds a Rust session; PyO3 forwards every method call. The API surface is deliberately mirrored: what works in Rust works in Python with the same shape.
2. ctx.register_parquet(...) calls the same ListingTable provider as the Rust API. From this point on, orders is a queryable table with the schema inferred from the Parquet file's metadata.
3. ctx.sql(...) triggers the full plan lifecycle: parser, plan builder, optimizer, physical planner. df is a Python object holding the physical plan; nothing has executed yet.
4. df.show() executes the plan (volcano pull), collects the batches, formats as a table, and prints. For programmatic use, df.collect() returns a list of pyarrow RecordBatches; df.to_arrow_table() returns a single pyarrow Table; both interop with the wider Arrow / pandas / Polars ecosystem.
5. This is the right tool for interactive analytics in a notebook, for sanity-checking a query before wiring it into Rust, for local development against S3 Parquet, and for teaching DataFusion concepts to team members who don't yet know Rust. It is not the right tool for production DataFrame pipelines (use Polars) or for distributed workloads (go to Ballista/Sail/Spark).

Output.

+-------------+-------------+---------------+
| customer_id | order_count | revenue_cents |
+-------------+-------------+---------------+
| 42          | 87          | 1275000       |
| 78          | 65          |  944500       |
| 11          | 54          |  812000       |
| ...         | ...         | ...           |
+-------------+-------------+---------------+

Rule of thumb. For any Python analytics workload against Parquet or CSV where you want SQL rather than pandas syntax and you don't want to spin up a JVM, datafusion-python is the right first pick. If you want DataFrame-shaped ergonomics, reach for Polars instead. Both live comfortably in the same notebook.

Worked example — DataFusion vs DuckDB benchmark on TPC-H Q1

Detailed explanation. The canonical "how does DataFusion perform" answer is a TPC-H benchmark comparison against DuckDB. TPC-H Q1 is the pricing-summary report — a scan + filter + group-by + aggregation on lineitem. Both engines handle it well; the relative timings show where each engine sits. Walk through the setup and the interpretation.

• The query. TPC-H Q1: scan lineitem, filter by ship date, group by return-flag + line-status, aggregate.
• The scale. SF10 (~60M lineitem rows) or SF100 (~600M).
• The engines. DataFusion 43+, DuckDB 1.1+.
• The results (indicative, 2026). DuckDB slightly faster (~15-30%) on TPC-H Q1 at SF10; DataFusion within 20% at most scale factors; both dramatically faster than Spark local mode.

Question. Run TPC-H Q1 against SF10 lineitem parquet in both DataFusion and DuckDB and compare wall-clock timings.

Input.

Engine	Version	Setup
DataFusion	43.0.0	pip install datafusion
DuckDB	1.1.0	pip install duckdb
Data	TPC-H SF10 lineitem.parquet	~60M rows, ~8 GB
Machine	Modern laptop (M-series or Ryzen 7+)	single-node

Code.

# Benchmark harness — TPC-H Q1
import time
from datafusion import SessionContext
import duckdb

TPCH_Q1 = """
SELECT l_returnflag,
       l_linestatus,
       sum(l_quantity)                                       AS sum_qty,
       sum(l_extendedprice)                                  AS sum_base_price,
       sum(l_extendedprice * (1 - l_discount))               AS sum_disc_price,
       sum(l_extendedprice * (1 - l_discount) * (1 + l_tax)) AS sum_charge,
       avg(l_quantity)                                       AS avg_qty,
       avg(l_extendedprice)                                  AS avg_price,
       avg(l_discount)                                       AS avg_disc,
       count(*)                                              AS count_order
FROM   lineitem
WHERE  l_shipdate <= DATE '1998-09-02'
GROUP  BY l_returnflag, l_linestatus
ORDER  BY l_returnflag, l_linestatus
"""

# ------- DataFusion -------
ctx = SessionContext()
ctx.register_parquet("lineitem", "tpch/sf10/lineitem.parquet")

t0 = time.perf_counter()
df_out = ctx.sql(TPCH_Q1).collect()
df_ms = (time.perf_counter() - t0) * 1000
print(f"DataFusion: {df_ms:8.1f} ms")

# ------- DuckDB -------
con = duckdb.connect()
con.execute(
    "CREATE VIEW lineitem AS SELECT * FROM read_parquet('tpch/sf10/lineitem.parquet')"
)

t0 = time.perf_counter()
_ = con.execute(TPCH_Q1).fetchall()
duckdb_ms = (time.perf_counter() - t0) * 1000
print(f"DuckDB:     {duckdb_ms:8.1f} ms")

Step-by-step explanation.

1. TPC-H Q1 is the canonical starter benchmark — a scan + filter + group-by + aggregate. It exercises filter pushdown, projection pushdown, hash aggregation, and Arrow-native execution. Both engines have implemented all of these; the query is not adversarial for either.
2. The DataFusion side: SessionContext() → register_parquet() → ctx.sql().collect(). The .collect() executes the plan and returns a list of RecordBatches; the wall-clock time is essentially the query time.
3. The DuckDB side: connect() → CREATE VIEW ... FROM read_parquet(...) → execute().fetchall(). Same shape, different engine. DuckDB tends to be slightly faster on TPC-H because its vectorised execution has had more benchmark-driven tuning than DataFusion's volcano executor.
4. The observed timings (indicative, 2026 hardware, SF10): DataFusion ~800-1200 ms; DuckDB ~600-900 ms. The gap has narrowed as DataFusion has matured. For SF100 or larger, DuckDB's memory management sometimes wins by another factor; DataFusion's Tokio-async I/O sometimes wins for cloud-storage-backed reads.
5. Neither engine is universally faster. Pick DataFusion when you need Rust embedding or plan-tree extensibility; pick DuckDB when you need the best single-node numbers on Parquet and don't need extensibility. Both are 10-50× faster than Spark local mode for this class of workload.

Output.

DataFusion:   987.4 ms
DuckDB:       734.2 ms
Ratio (DF / DuckDB): 1.35×

Scale factor	DataFusion (ms)	DuckDB (ms)	Ratio
SF1 (6M rows)	~120	~80	1.5×
SF10 (60M rows)	~1000	~750	1.35×
SF100 (600M rows)	~10000	~8000	1.25×

Rule of thumb. For pure single-node Parquet benchmarks, DuckDB is slightly faster in 2026. The gap is narrow enough that the engine choice should be driven by (a) whether you need Rust-native embedding, (b) whether you need plan-tree extensibility, (c) whether you want to be part of the shared-substrate ecosystem — not by raw benchmark numbers. If any of (a), (b), or (c) matter, pick DataFusion.

Senior interview question on direct DataFusion usage

A senior interviewer might ask: "We're building a WASM analytics module that runs in the browser — SQL over Parquet fetched from HTTP. Walk me through why DataFusion is a candidate, the Rust-to-WASM compilation story, the memory constraints, and the alternatives you evaluated."

Solution Using DataFusion compiled to WASM with an HTTP-backed ObjectStore and a bounded batch-size configuration

// Cargo.toml
// [package]
// name = "df-wasm-analytics"
// crate-type = ["cdylib"]
//
// [dependencies]
// datafusion   = { version = "43", default-features = false, features = ["parquet"] }
// object_store = { version = "0.11", features = ["http"] }
// wasm-bindgen = "0.2"
// tokio        = { version = "1", features = ["rt", "macros"] }
// wee_alloc    = "0.4"  # smaller allocator for WASM

use datafusion::prelude::*;
use datafusion::execution::runtime_env::RuntimeEnv;
use datafusion::execution::context::{SessionConfig, SessionState};
use object_store::http::HttpBuilder;
use std::sync::Arc;
use wasm_bindgen::prelude::*;

#[global_allocator]
static ALLOC: wee_alloc::WeeAlloc = wee_alloc::WeeAlloc::INIT;

#[wasm_bindgen]
pub async fn run_query(sql: String, parquet_url: String) -> Result<String, JsValue> {
    // 1. Small session config — WASM memory budget is tight
    let config = SessionConfig::new()
        .with_batch_size(1024)                   // small batches for tight RAM
        .with_target_partitions(1);              // no parallelism in single-threaded WASM

    let runtime = Arc::new(RuntimeEnv::default());
    let state   = SessionState::new_with_config_rt(config, runtime);
    let ctx     = SessionContext::new_with_state(state);

    // 2. Register the HTTP-backed ObjectStore for the Parquet URL
    let http_store = HttpBuilder::new()
        .with_url(&parquet_url)
        .build()
        .map_err(|e| JsValue::from_str(&e.to_string()))?;
    ctx.runtime_env().register_object_store(
        &url::Url::parse(&parquet_url).unwrap(),
        Arc::new(http_store),
    );

    // 3. Register the Parquet file
    ctx.register_parquet("t", &parquet_url, ParquetReadOptions::default())
        .await
        .map_err(|e| JsValue::from_str(&e.to_string()))?;

    // 4. Run the user's SQL
    let df = ctx.sql(&sql).await.map_err(|e| JsValue::from_str(&e.to_string()))?;
    let batches = df.collect().await.map_err(|e| JsValue::from_str(&e.to_string()))?;

    // 5. Serialise result to JSON for the JS caller
    let json = record_batches_to_json(&batches);
    Ok(json)
}

fn record_batches_to_json(batches: &[arrow::record_batch::RecordBatch]) -> String {
    // (elided — use arrow-json for real code)
    format!("{{\"batches\": {}}}", batches.len())
}

Step-by-step trace.

Layer	Config	Purpose
Compilation target	wasm32-unknown-unknown	run in browser
Allocator	wee_alloc	smaller than the default (~7 KB vs ~40 KB)
Batch size	1024	tight RAM budget in the browser
Parallelism	target_partitions = 1	single-threaded WASM
ObjectStore	HTTP-backed	fetch Parquet from any URL
Async runtime	Tokio single-threaded	fits wasm-bindgen-futures
Result encoding	JSON	interop with JS caller

After the WASM module compiles (~4-6 MB gzipped in 2026), it loads in a browser and can run arbitrary SQL over Parquet fetched via HTTP. Memory footprint stays under 128 MB for TPC-H SF1-shaped workloads. There is no server round-trip; the browser is the query engine. Alternative candidates evaluated: DuckDB-WASM (mature; C++ compiled to WASM; slightly bigger footprint), sqlite-wasm (only handles SQLite files), Polars-JS (DataFrame API only, no SQL surface).

Output:

Metric	Value
WASM binary size	~5 MB gzipped
Cold-start latency	~200 ms (WASM instantiate + Tokio startup)
Warm query latency (TPC-H Q1 SF1)	~800 ms (HTTP-fetch dominates)
Peak RAM	~80 MB for SF1
Alternative: DuckDB-WASM	~8 MB gzipped, ~1.4× faster on Q1
Trade-off	Rust ecosystem interop + smaller footprint vs raw perf

Why this works — concept by concept:

• DataFusion compiles to WASM — Rust's wasm32-unknown-unknown target plus wasm-bindgen produces a browser-loadable module. The Tokio single-threaded runtime + wasm-bindgen-futures provides the async substrate.
• HTTP-backed ObjectStore — object_store::http lets DataFusion fetch Parquet files over HTTP with range-request support. Parquet's page indexes and column pruning turn a "download the whole file" into "download only the pages we need" — this is what makes browser-side Parquet queries feasible.
• Small batch size + single-partition — WASM has tight RAM budgets (browsers cap memory at ~2-4 GB per tab). Small batches keep peak RAM bounded; single-partition avoids the complexity of pretending to have a thread pool in single-threaded WASM.
• wee_alloc — the default Rust allocator is ~40 KB in a WASM module. wee_alloc is ~7 KB with slightly slower allocation — the right trade for browser-loaded modules where size dominates.
• Cost — ~5 MB WASM binary (vs ~8 MB for DuckDB-WASM). Slightly slower per-query than DuckDB-WASM (1.3-1.5× on TPC-H). Full Rust ecosystem interop (Arrow, Parquet, ObjectStore) — you can share code with your Rust server. The choice comes down to "Rust ecosystem interop matters" vs "raw browser perf matters" — pick DataFusion for the former, DuckDB-WASM for the latter. O(bytes fetched) I/O plus O(rows returned) execution; both dominated by network, not CPU.

Python
Topic — data-analysis
Data analysis and Parquet problems for practice

Practice →

SQL
Topic — sql
SQL over Parquet + columnar interview problems

Practice →

---

Cheat sheet — DataFusion recipes

• Hello-world SQL over Parquet in Rust. use datafusion::prelude::*; inside a #[tokio::main] function; let ctx = SessionContext::new(); allocates the session; ctx.register_parquet("orders", "path.parquet", ParquetReadOptions::default()).await?; wires the Parquet file into the catalog as a ListingTable; let df = ctx.sql("SELECT ...").await?; runs the whole plan lifecycle (parser → plan builder → optimizer → physical planner) and returns a DataFrame; df.show().await?; triggers volcano execution and prints the RecordBatches. Everything happens inside one Rust process; no cluster, no JVM, no driver. Add datafusion = "43" and tokio = { version = "1", features = ["full"] } to Cargo.toml.
• Hello-world SQL from Python. pip install datafusion pyarrow. In Python: from datafusion import SessionContext; ctx = SessionContext(); ctx.register_parquet("orders", "path.parquet"); df = ctx.sql("SELECT ... FROM orders"); df.show(). Convert to Arrow via df.to_arrow_table(); convert to pandas via .to_pandas() on the resulting Arrow table. This is the right tool for notebook analytics, sanity-checking SQL before wiring into Rust, and teaching DataFusion concepts to Python-first team members.
• Custom TableProvider skeleton. Implement schema() (return the Arrow SchemaRef), scan(state, projection, filters, limit) (return an Arc<dyn ExecutionPlan> producing RecordBatches — MemoryExec::try_new(...) is the shortcut for in-memory data), supports_filters_pushdown(filters) (return Vec<TableProviderFilterPushDown> — Exact if you'll apply the filter yourself, Unsupported to let the planner do it, Inexact if you apply a superset), and statistics() (row-count and column min/max hints for the optimizer). Register with ctx.register_table("name", Arc::new(YourProvider)). This is the load-bearing plug for any custom storage source.
• Custom OptimizerRule skeleton. impl OptimizerRule for MyRule { fn name(&self) -> &str { "my_rule" } fn try_optimize(&self, plan: &LogicalPlan, config: &dyn OptimizerConfig) -> Result<Option<LogicalPlan>> { /* match narrow pattern; return Some(new_plan) if fired, None if not */ } }. Insert into the session's optimizer rule vector at a specific position (before or after built-ins matters). Cardinal rule: never rewrite optimistically — return None when your match is uncertain. The optimizer's fixed-point loop assumes each rule either reduces plan complexity or leaves it unchanged; violate that and you'll hang the optimizer.
• Custom UDF / UDAF / UDWF registration. Scalar UDF: let f = create_udf("plus_one", vec![DataType::Int64], DataType::Int64, Volatility::Immutable, Arc::new(|args| { /* per-batch fn */ })); then ctx.register_udf(f);. UDAF: create_udaf(name, input_types, return_type, volatility, accumulator_factory, state_type) — implement Accumulator::state(), update_batch(), merge_batch(), evaluate(). UDWF: same as UDAF plus a window-frame contract. Every extension surfaces through the same SQL as built-in functions; users can't tell built-in from custom from the SQL side.
• Ballista distributed config. cargo install ballista-scheduler ballista-executor. Run scheduler: ballista-scheduler --config ballista-scheduler.toml. Run executor: ballista-executor --config ballista-executor.toml. Client: use ballista::prelude::*; let ctx = BallistaContext::remote("scheduler-host", 50050, BallistaConfig::default()).await?; — same ctx.sql(...) API as embedded DataFusion, but planning is centralised in the scheduler and execution is fanned out to executors. Substrait-encoded plans on the wire. For production, run scheduler + N executors on K8s via the Ballista Helm chart.
• Comet Spark accelerator config. spark-submit --packages org.apache.datafusion:datafusion-comet-spark3.5_2.12:0.4.0 --conf spark.plugins=org.apache.spark.CometPlugin --conf spark.comet.enabled=true --conf spark.comet.exec.enabled=true .... Spark plans normally through Catalyst; Comet's CometExecRule inspects each stage and swaps to CometNativeExec when every operator is supported. Unsupported stages fall back to Spark's JVM executor transparently. Tune batch size upward before the JVM ⇄ native boundary to amortise JNI overhead.
• InfluxDB 3 IOx architecture reference. Ingestion writes into Arrow2-backed in-memory buffer; buffer flushes to Parquet in object storage; query layer routes through DataFusion which reads buffer + Parquet, applies pushdown, returns Arrow2 RecordBatches to InfluxQL / SQL / FlightSQL surface. IOx-specific contributions: time-series optimizer rules (rollup pushdown, time-bucket rewrite), custom TableProvider for the buffer + Parquet tiered store, custom UDAFs for time-series aggregations. The engine itself is unmodified DataFusion.
• DataFusion vs DuckDB vs Polars decision matrix. SQL + Rust embedding + plan-tree extensibility → DataFusion. Best single-node SQL perf, C++ ok, no embedding → DuckDB. DataFrame-first Python + Rust perf under the hood → Polars. Distributed Rust → Ballista/Sail (DataFusion under the hood). Distributed JVM ecosystem → Spark. Federated across many sources → Trino. Real-time OLAP at billions of rows → ClickHouse. Print this on a sticky note; use it in every interview.
• Interview probe checklist. (1) Name Arrow2 as the in-memory format; (2) walk the four-stage plan lifecycle (parser → LogicalPlan → optimizer → physical plan → RecordBatch stream); (3) name at least five optimizer rules (PushDownFilter, PushDownProjection, SimplifyExpressions, JoinReorder, EliminateFilter); (4) name TableProvider / UDF / UDAF / UDWF / OptimizerRule / custom ExecutionPlan as the extension points; (5) position DataFusion on the shared-substrate axis versus Velox (C++) and Catalyst (JVM); (6) name at least three downstream projects (InfluxDB 3, Comet, Ballista, GreptimeDB, ROAPI, Cube.dev). All six unprompted = senior signal.
• Deployment sizing rules of thumb. Embedded (single binary): ~5-50 MB RAM per session; batch size 8192; target partitions = CPU count. Analytics service: ~200 MB RAM per query; scale by concurrent-query count. Ballista cluster: scheduler 1× (~500 MB), executors N× (~1-4 GB each); scale N by expected concurrent queries × per-query memory. Comet on Spark: no additional memory (piggybacks Spark executor pods); native memory is Arrow-backed off-heap.
• Version pinning cheat sheet. DataFusion release cadence is monthly-ish; API changes are common between majors. Pin to datafusion = "43" (or your current major) and expect to review changelogs on upgrades. Arrow2 is at arrow = "53" and moves in step. datafusion-python version matches the underlying Rust crate. Ballista and Comet trail DataFusion by 1-2 releases as they integrate the changes. For production, use Cargo.lock and staged upgrades; do not float ~43 in production.
• Debugging with EXPLAIN VERBOSE. For any query that's slow or produces the wrong shape, ctx.sql("EXPLAIN VERBOSE " + your_sql).await?.show().await?; prints the plan after every optimizer pass. Look for (a) which rules fired, (b) whether the final TableScan has full_filters=[...] (filter pushdown worked) and projection=[...] (projection pruning worked), (c) whether the physical plan chose HashJoin vs SortMergeJoin vs NestedLoopJoin. Nine out of ten "why is my query slow" bugs are answered by "the filter didn't push down because your TableProvider returned Unsupported."
• Substrait interop. DataFusion supports serialising and deserialising LogicalPlan and ExecutionPlan via Substrait — the emerging cross-engine IR. This is the bridge that lets Ballista ship plans between scheduler and executor without picking a proprietary format, and (in principle) lets a Substrait-emitting frontend target DataFusion for execution. use datafusion_substrait::logical_plan::producer::to_substrait_plan; use datafusion_substrait::logical_plan::consumer::from_substrait_plan; For cross-engine portability, Substrait is the medium of exchange.
• Feature flags to know. datafusion crate features: crypto_expressions, encoding_expressions, regex_expressions, unicode_expressions, avro, compression, pyarrow (for datafusion-python interop). Default features include parquet, arrow, and the common expression sets; strip features aggressively for WASM builds to shrink binary size. See Cargo.toml of datafusion for the current list.

Frequently asked questions

What is Apache DataFusion in one sentence?

Apache DataFusion is a Rust-native, Arrow2-based, columnar query engine that exposes both SQL and DataFrame APIs, ships as an embeddable library rather than a server, and has become the shared substrate under dozens of vendor products — InfluxDB 3.0 (IOx), Comet (Spark's Rust-native execution accelerator), Ballista (distributed DataFusion on Kubernetes), Sail (distributed Rust engine), GreptimeDB, ROAPI, Cube.dev, ParadeDB, Coralogix — because writing a correct SQL parser plus a rule-based optimizer plus a volcano-style executor plus a Parquet reader plus a Tokio-integrated async I/O layer once, in a memory-safe language, is a five-year investment nobody wants to repeat. The plan lifecycle runs SQL / DataFrame → logical plan → rule-based optimizer → physical plan → execution stream, with Arrow2 RecordBatch as the unit of exchange between operators. Every extension point interviewers probe — TableProvider, UDF, UDAF, UDWF, OptimizerRule, custom ExecutionPlan — hooks into a specific stage of that lifecycle, and knowing which is the difference between an architect who has embedded DataFusion and a candidate who has only read the docs.

DataFusion also sits on the shared-substrate axis in the wider industry: Rust → DataFusion, C++ → Velox (Meta), JVM → Catalyst (Spark) and Calcite (Trino / Flink). Every language ecosystem has converged on a small number of query-engine substrates in 2024-2026 because building the same engine from scratch inside every vendor turned out to be a losing engineering strategy; sharing the substrate means bugs are fixed once, optimizer rules are contributed once, and vendors compete on storage / distribution / connectors / UX rather than on whether their WHERE clause pushdown works. DataFusion is the Rust-language answer in that trend.

DataFusion vs DuckDB — when do I pick each?

Pick DataFusion when the requirement includes any of: (a) your product is written in Rust and you want the engine embedded in your binary (no separate process, no JVM, no cluster); (b) you need plan-tree extensibility — custom TableProvider, custom OptimizerRule, custom ExecutionPlan, custom UDF/UDAF/UDWF; (c) you want to be part of the shared-substrate ecosystem that includes InfluxDB 3.0, Comet, Ballista, GreptimeDB, ROAPI, and dozens of other vendors; (d) you need a viable path to distributed execution via Ballista or Sail (both use DataFusion as the engine); or (e) you're compiling to WASM (DataFusion is one of the smallest full-SQL engines available for the browser). The Rust-embedding requirement is the single most-common decisive factor.

Pick DuckDB when: (a) your product is written in C++ (or you're calling DuckDB from Python / R / Node / Go via the mature client libraries); (b) you want the best single-node SQL performance on Parquet — DuckDB benchmarks 15-30% faster than DataFusion on most TPC-H queries in 2026; (c) you want the more mature CLI and Python UX (DuckDB has invested heavily in developer ergonomics); or (d) you don't need Rust-native embedding, extensibility beyond the built-in operators, or a distributed path. DuckDB is the pragmatic pick for "just give me the fastest single-node SQL engine over Parquet," and its ecosystem (duckdb-wasm, dbt-duckdb, duckdb-motherduck) is broader in 2026. Both engines are 10-50× faster than Spark local mode; the choice between them is a language and extensibility trade-off, not a performance argument.

What is Ballista and how does it relate to DataFusion?

Ballista is the "Spark for DataFusion" — a Kubernetes-native distributed runner that partitions a DataFusion physical plan across worker pods. The naming has historically confused everyone because Ballista was originally the parent project (a distributed compute framework) and DataFusion was its query-engine sub-project; today DataFusion is the load-bearing engine and Ballista is one of two shipped distributed wrappers around it (the other being Sail). A Ballista deployment consists of a scheduler pod (which runs the DataFusion planner and coordinates stages) and N executor pods (which run DataFusion natively and shuffle intermediate results). Plans are serialised on the wire via Substrait, an emerging cross-engine IR that gives Ballista optionality for future frontends.

Ballista is production-viable in 2026 for scoped analytics workloads — TB-scale Parquet queries, distributed SQL over object-storage-backed lakes, batch-oriented ETL. It is not a drop-in Spark replacement: Ballista's connector ecosystem, streaming maturity, cluster-management story, and 15 years of tuning are all behind Spark's. Pick Ballista when your team is Rust-native, your workload fits its scope (batch analytics over columnar storage), and you cannot tolerate a JVM footprint. Pick Spark when you need the mature ecosystem (Delta Lake, Structured Streaming, MLlib, GraphX, hundreds of connectors), when your workload sits inside an existing Spark deployment, or when your team's expertise is JVM-based. Both share the "distributed volcano-style query engine" pattern; they differ in language, ecosystem depth, and deployment shape.

Why did InfluxDB 3.0 pick DataFusion?

InfluxDB 3.0 (also known as IOx internally) rebuilt its query layer on Rust + Arrow + DataFusion + Parquet in 2020-2022, replacing the Go-based InfluxDB 2 engine that Paul Dix's team had shipped previously. The public rationale, documented in KubeCon 2022 keynotes and multiple InfluxData tech-talks 2023-2024, has three prongs. First, columnar storage was the right primitive for time-series data — moving from row-based to columnar unlocked 100× ingest throughput and unlimited cardinality (fixing InfluxDB 2's cardinality-blowup problem). Second, Arrow was the right in-memory format for the columnar substrate — Arrow's ecosystem interop meant InfluxDB 3.0 gets Parquet, FlightSQL, and cross-language interop essentially for free. Third, DataFusion was the right query engine — it exposed the full SQL surface, the plan-tree extensibility for time-series-specific optimizer rules, and the Rust-native embedding that let InfluxDB 3.0 ship as a single static binary without a JVM.

The shared-substrate argument dominated the decision. Building a bespoke query engine would have cost the team 5+ engineer-years and produced a commodity output; adopting DataFusion freed that budget for the differentiators — the time-series-specific storage layout, the ingestion pipeline that flushes buffers to Parquet in object storage, and the InfluxQL compatibility layer. Time-series-specific optimizations (rollup pushdown, time-bucket aggregation rewrites, tag-filter pruning) are contributed to DataFusion where they generalise and kept in-house where they're InfluxDB-specific. This is the reference case study for the shared-substrate architecture pattern — cited in every senior interview that probes "why would a vendor pick a shared substrate over building their own?"

How does Comet accelerate Spark with DataFusion?

Comet is Apache DataFusion Comet — a Spark accelerator that swaps Spark's JVM physical operators for DataFusion's native Rust operators on a per-stage basis. Published in SIGMOD 2024 by the Apple team, it's the load-bearing example of the JVM ⇄ native handoff pattern for the shared-substrate world. The mechanism: a Spark query plans normally through Catalyst; before physical execution, Comet's CometExecRule inspects each stage; if every operator in the stage has a DataFusion equivalent (Scan, Filter, Project, HashAggregate, Sort, Limit, etc.), Comet wraps the stage in CometNativeExec, which marshals input Arrow batches across the JVM ⇄ native boundary via JNI + Arrow's C Data Interface, executes the stage in Rust via DataFusion's operators, and marshals output batches back to the JVM. Non-supported stages (custom Scala UDFs, unsupported operators) fall back to Spark's regular executor transparently.

The performance win, per the SIGMOD paper's TPC-H benchmarks, is 2-5× per-stage speedup for CPU-bound stages, with the win coming from Arrow's vectorised execution and Rust's lack of JVM garbage-collection pauses. The bridge overhead (JNI + Arrow marshalling) is fixed per batch, so Comet's tuning story is largely "make batches bigger before crossing the boundary" — small batches see the bridge overhead eat the native execution win. Comet is the case study for "shared substrate reduces engineering scope" — Comet's team wrote the plan-rewriting bridge and the JNI marshalling; every physical operator, the whole optimizer, the Arrow execution — all came from DataFusion. Two-person team, Spark accelerator shipped. That's the shared-substrate payoff at scale.

Is DataFusion production-ready in 2026?

Yes, unambiguously. DataFusion is production-deployed at scale by InfluxData (InfluxDB 3.0 IOx serves production time-series workloads for thousands of customers), Coralogix (log observability at petabyte scale), GreptimeDB (time-series + IoT), ROAPI (Parquet-to-REST services), Cube.dev (semantic layer for BI tools), ParadeDB (Postgres-compatible analytics), and the Apple Comet team (Spark acceleration in production Spark clusters). The API surface is stable across the 43+ releases; the release cadence is monthly-ish; and the contributor base spans dozens of downstream vendors. There is no meaningful "will it stick around" risk in 2026 — the shared-substrate pattern has locked in enough vendor investment that DataFusion is now infrastructure.

The caveats: (a) the release cadence means API changes between majors are common, so pin your Cargo.toml version and review changelogs on upgrades; (b) the ecosystem's connector coverage lags Trino / Spark in some areas (federated queries across many source systems); (c) distributed execution via Ballista / Sail is production-viable but does not have Spark's cluster-management ecosystem depth; (d) cost-based optimization is experimental in mainline (rule-based is the default), so join-order decisions on complex multi-way joins may need manual hints. None of these disqualifies DataFusion for its target workloads — Rust-native embedded queries, single-node analytics, distributed scoped workloads, and shared-substrate downstream products. For the target use cases, it is the mature, production-ready pick in 2026.

Practice on PipeCode

• Drill the SQL practice library → for the columnar-analytics, query-optimizer, plan-tree, and window-function problems senior interviewers love when they open with DataFusion / Trino / Spark trade-offs.
• Rehearse on the ETL practice library → for the Parquet-scan, filter-pushdown, projection-pruning, and rollup-pushdown patterns that DataFusion's optimizer applies automatically and that senior DEs are expected to be able to reason about explicitly.
• Sharpen the streaming axis with the streaming practice library → for the time-series, InfluxDB 3.0 IOx, Comet Spark-accelerator, and log-observability workloads where DataFusion sits as the shared substrate.
• Anchor query-engine design intuition on the design practice library → for the "build vs pick substrate," "custom TableProvider skeleton," "custom OptimizerRule when to fire," and "shared-runtime governance ring" scenarios that come up in senior data-platform interviews.
• Stack the prerequisites against PipeCode's broader 450+ data-engineering catalogue to anchor the four-axis DataFusion positioning card (language, execution model, extensibility, deployment shape) against real graded inputs and the shared-substrate architectural pattern that dominates modern data-platform decisions.

Lock in apache datafusion muscle memory

Docs explain the engine. PipeCode drills explain the decision — when DataFusion's Rust-native embedding wins over DuckDB, when its plan-tree extensibility wins over Polars, when Ballista's distributed shape wins over Spark, when Comet's JVM ⇄ native bridge earns its overhead, when InfluxDB 3.0's shared-substrate bet compounds year after year. Pipecode.ai is Leetcode for Data Engineering — pattern-first practice tuned for the production trade-offs senior data engineers actually face when choosing query engines.

Practice SQL problems →
Practice design problems →