Rust for Data Engineering: When to Reach for a Rust Component in a Python-Heavy Stack

rust data engineering is the pick-one architectural decision that quietly shapes every serious Python-heavy pipeline in 2026 — the "Python-first, drop-to-Rust-for-hot-paths" pattern is now the dominant DE architecture, and the components you have been quietly pip install-ing (Polars, delta-rs, PyIceberg, DataFusion, Vector, uv, ruff) are already Rust cores wearing a Python surface, whether or not your team calls them that out loud. Every senior data engineer on a Python stack has hit the same wall: a JSON parser that eats 40% of a batch job's CPU, a custom aggregation that runs at 200 k rows/sec when the SLA demands 5 M rows/sec, a streaming transform that can't clear the GIL, or a sink that segfaults in a C-extension nobody wants to debug. The engineering trade-off does not live in "should we use Rust" — every modern DE stack already does, one dependency at a time — but in when you reach for a first-party Rust component, which category of adoption you pick, and how you defend the choice in an interview.

This guide is the senior-DE walkthrough you wished existed the first time an interviewer asked "walk me through when you would drop a rust data pipeline component into a Python-heavy stack," or "your batch job spends 60% of its CPU parsing JSON — how would you fix it with pyo3 and a Rust extension?", or "rust vs python for a 10 M-rows/sec aggregation — which and why?" It walks through the "Python-first, drop-to-Rust-for-hot-paths" pattern behind Polars / delta-rs / DataFusion / Vector, the five categories of Rust adoption in DE, the pyo3 + maturin toolchain that makes a Python-facing Rust extension a one-week project, the four-gate decision framework (CPU-bound, > 1 M ops/sec, memory-safety, GIL escape) for picking rust polars or a Rust extension over pure Python + NumPy, and the career signals (companies hiring, interview probes, when not to learn Rust) that turn "I know some Rust" into a hiring lever. Each section pairs a teaching block with a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the SQL practice library →, rehearse the streaming axis with the streaming practice library →, and sharpen system design with the design practice library →.

---

On this page

• Why Rust in data engineering matters in 2026
• The 5 categories of Rust adoption in DE
• pyo3 + maturin — your first Python-facing Rust extension
• When Rust wins vs when Python + NumPy is enough
• Career + adoption signals + interview probes
• Cheat sheet — Rust-in-DE recipes
• Frequently asked questions
• Practice on PipeCode

---

1. Why Rust in data engineering matters in 2026

The "Python-first, drop-to-Rust-for-hot-paths" pattern is now the dominant DE architecture

The one-sentence invariant: rust data engineering is the pattern where the pipeline stays Python for glue, orchestration, notebooks, and I/O, but every CPU-bound hot path — DataFrame execution, table-format metadata, streaming transforms, JSON parsing, custom aggregations — drops to a Rust core exposed via a Python surface through pyo3 + maturin, giving you memory safety, GIL escape, and 5×–100× speedups on the hot loop without rewriting the pipeline in a new language. The dependency graph you already ship — Polars, delta-rs, PyIceberg, DataFusion, Vector, uv, ruff — is the pattern in miniature; the decision you make in 2026 is whether to author your own component the same way.

The four axes interviewers actually probe.

• Where does the pipeline spend its CPU? If it's I/O-bound (waiting on S3, Postgres, Kafka), Rust helps nothing and Python + asyncio is correct. If a single hot function eats 40%+ of the batch job's CPU, that function is the Rust candidate. Interviewers open with this because it separates candidates who profile from candidates who chase language wars.
• How big is the ops budget? Rust replaces the classic C/C++ extension (ctypes, Cython, hand-written CPython C API) and the JNI-in-a-JVM escape hatch (PySpark → Scala UDF) with a memory-safe, single-binary, cross-platform wheel. The ops cost of a Rust extension is dramatically lower than a C extension because there are no segfaults, no dangling pointers, and no thread-unsafe globals.
• Does the workload scale across cores? Python's GIL serialises pure-Python CPU work regardless of how many cores you rent. A Rust extension releases the GIL for its duration (Python::allow_threads) so a 32-core box actually gets 32 cores of throughput. This is the argument for Rust in a Ray / Dask / process-pool deployment.
• What are the memory guarantees? Rust's borrow checker eliminates use-after-free, double-free, and data-race bugs at compile time. For any component that touches allocated buffers under concurrent access — a streaming sink, an arena-allocated arrow batch, a memory-mapped Parquet reader — the safety story is qualitatively better than C/C++. Interviewers probe this because it's what pushed Rust into the systems-programming mainstream in the first place.

The 2026 reality — the Rust dependency is already in your requirements.txt.

• Polars — the pandas-successor DataFrame library authored in Rust; pip install polars ships a Rust binary via pyo3 wheels. On a 10 GB Parquet aggregation, Polars typically beats pandas by 5×–30× with a fraction of the memory.
• delta-rs + PyIceberg — the Delta Lake and Iceberg Python clients use Rust for the CPU-bound metadata operations (snapshot merges, expression evaluation, file pruning). delta-rs is Rust-first; PyIceberg uses Rust for hot loops.
• Apache DataFusion — a Rust-native Apache Arrow-based query engine; used inside Ballista, InfluxDB IOx, and increasingly as an embedded engine in Python analytics libraries.
• Vector (Datadog) — the observability data-router; single Rust binary, thousands of nodes, replacing Fluentd / Logstash across the industry.
• uv + ruff (Astral) — the Python packaging + linting tools that made a generation of Python devs faster; both Rust, both pip install-able, both replacing multi-tool Python-native chains.

What interviewers listen for.

• Do you name Polars, delta-rs, DataFusion, Vector, uv, ruff as the concrete Rust-in-DE catalogue without prompting? — senior signal.
• Do you describe the pattern as "Python-first, drop-to-Rust-for-hot-paths" rather than "rewrite in Rust"? — required answer.
• Do you name pyo3 + maturin as the toolchain that makes Python-facing Rust libraries mainstream? — senior signal.
• Do you push back on "just add more workers" with the GIL argument for CPU-bound work? — required answer.
• Do you describe Rust as "replacing the C/C++ + JNI escape hatch without giving up memory safety" rather than as a language crusade? — senior signal.

Worked example — the "Python-first, Rust hot-path" architecture map

Detailed explanation. The single most useful artifact for a Rust-in-DE interview is a memorised map that shows which pipeline layers stay Python and which drop to Rust. Every senior conversation converges on this map within the first ten minutes; having it ready separates a fluent answer from a stumbling one. Walk through building the map for a hypothetical batch ETL pipeline reading Parquet, transforming, and writing to Iceberg.

• Pipeline shape. Airflow-scheduled Python task → read 50 GB Parquet from S3 → parse embedded JSON column → aggregate to daily metrics → write to Iceberg on S3.
• Where CPU goes today. JSON parse: 40%. DataFrame aggregation: 30%. Iceberg commit: 15%. S3 I/O: 10%. Airflow orchestration: 5%.
• Where Rust already lives (invisibly). Polars for the DataFrame layer. PyIceberg hot loops for commits. Not yet in the JSON parse (that's the fix).
• Where Rust does NOT belong. Airflow orchestration, S3 I/O, notebooks, business rules — all stay Python.

Question. Draw the pipeline layer stack and mark each layer Python or Rust; justify each decision.

Input.

Pipeline layer	Language today	Recommended	Justification
Airflow DAG + task glue	Python	stay Python	orchestration, I/O bound
S3 read (Parquet)	Python (boto3 + polars)	stay Python	I/O bound; already Rust under polars.read_parquet
JSON parse (embedded col)	Python (orjson)	Rust extension	40% of CPU; hot loop
DataFrame aggregation	Python (polars)	already Rust	30% CPU; polars is Rust core
Iceberg commit	Python (pyiceberg)	already Rust hot loops	15% CPU; metadata engine in Rust
Business rules + logging	Python	stay Python	tiny CPU; clarity matters

Code.

# Before — pure Python JSON parse in the hot loop
import polars as pl
import orjson

def parse_payload_col(df: pl.DataFrame) -> pl.DataFrame:
    """Parse the `payload` JSON column into typed columns."""
    parsed = [orjson.loads(row) for row in df["payload"].to_list()]
    # 40% of batch CPU spent here on a 50 M-row batch
    return df.with_columns(
        pl.Series("user_id",    [r["user_id"]    for r in parsed]),
        pl.Series("event_type", [r["event_type"] for r in parsed]),
        pl.Series("value",      [r["value"]      for r in parsed]),
    )

// After — src/lib.rs of a pyo3 crate `payload_parser`
use pyo3::prelude::*;
use serde_json::Value;

#[pyfunction]
fn parse_payload_batch(payloads: Vec<String>) -> PyResult<Vec<(String, String, f64)>> {
    // Rust does the parse under released GIL; Python sees a list of tuples
    let out: Vec<(String, String, f64)> = payloads
        .into_iter()
        .filter_map(|s| {
            let v: Value = serde_json::from_str(&s).ok()?;
            let user_id    = v.get("user_id")?.as_str()?.to_string();
            let event_type = v.get("event_type")?.as_str()?.to_string();
            let value      = v.get("value")?.as_f64()?;
            Some((user_id, event_type, value))
        })
        .collect();
    Ok(out)
}

#[pymodule]
fn payload_parser(_py: Python, m: &PyModule) -> PyResult<()> {
    m.add_function(wrap_pyfunction!(parse_payload_batch, m)?)?;
    Ok(())
}

Step-by-step explanation.

1. The map is layered top-to-bottom by CPU cost. Any layer that consumes < 5% of the batch's CPU never justifies a Rust rewrite — the engineering cost dwarfs the savings. Any layer that consumes > 20% is a Rust candidate, especially if it's a tight loop over rows or bytes.
2. Rows 2, 4, and 5 (S3 read, DataFrame aggregation, Iceberg commit) are already Rust under the hood — you get those wins by picking the right library, not by writing new Rust. The interviewer signal is naming the underlying Rust cores, not claiming credit for writing them.
3. Row 3 (JSON parse) is the actual Rust rewrite opportunity. orjson is already the fastest pure-Python JSON parser, but pulling the parse loop into a pyo3 extension that operates over a Vec<String> under released GIL yields another 3×–10× depending on payload shape.
4. Rows 1 and 6 (Airflow DAG glue and business rules) are where Python's flexibility earns its keep. Rewriting a DAG in Rust would trade weeks of engineering for zero CPU savings and huge readability loss. Interview candidates who suggest this are showing they don't profile before optimising.
5. The map is a whiteboard artifact: two columns (layer, language), one arrow (Python-first, drop-to-Rust-for-hot-paths), one CPU-percentage annotation per row. Draw it before you name any library.

Output.

Layer	Language decision	CPU freed
Airflow DAG	Python	(tiny; leave alone)
S3 read	Python calling Rust polars	already realised
JSON parse	Rust pyo3 extension	40% → ~5% (8× on the hot loop)
DataFrame agg	Python calling Rust polars	already realised
Iceberg commit	Python calling Rust pyiceberg	already realised
Business rules	Python	(leave alone)

Rule of thumb. Draw the layer map before you write any Rust. If a layer is < 20% of CPU, you cannot justify the Rust cost; if it's > 20% and CPU-bound, it's a pyo3 candidate. Everything else stays Python — Rust is a scalpel, not a paint roller.

Worked example — what interviewers actually probe on Rust in DE

Detailed explanation. The senior Rust-in-DE interview has a predictable structure: the interviewer opens with an ambiguous question ("how do you feel about Rust in the data stack?"), then progressively narrows to test whether you understand the hot-path pattern. Candidates who name the pattern in sentence one score highest; candidates who describe "Rust is faster than Python" score lowest. Walk through the interview grading rubric.

• Ambiguous opener. "What's your take on Rust in modern data engineering?" — invites you to name the pattern.
• Follow-up 1. "Would you rewrite your whole pipeline in Rust?" — probes proportionality.
• Follow-up 2. "When would you NOT reach for Rust?" — probes I/O-bound understanding.
• Follow-up 3. "What's pyo3?" — probes toolchain fluency.
• Follow-up 4. "Which Rust libraries are you already using?" — probes ecosystem awareness.

Question. Draft a 5-minute senior Rust-in-DE answer that covers all four axes without waiting to be asked.

Input.

Interview signal	Weak answer	Senior answer
Pattern named	"Rust is faster than Python"	"Python-first, drop to Rust for CPU-bound hot paths via pyo3"
Proportionality	"yes, we should rewrite"	"no; profile first, rewrite the top 20% of CPU only"
When NOT to use	"always use Rust"	"I/O-bound pipelines gain nothing; stay Python + asyncio"
Toolchain	"not sure"	"pyo3 + maturin; ship pre-built wheels; test with pytest"
Ecosystem	"I've heard of Polars"	"Polars, delta-rs, DataFusion, Vector, uv, ruff — Rust cores under Python surfaces"

Code.

Senior Rust-in-DE answer template (5 minutes)
=============================================

Minute 1 — name the pattern up front
  "The dominant pattern in modern DE is Python-first, drop to Rust
   for CPU-bound hot paths via pyo3 + maturin. Most teams already
   use it without labelling it — Polars, delta-rs, PyIceberg,
   DataFusion, Vector, uv, ruff are all Rust cores under Python
   surfaces."

Minute 2 — proportionality
  "You never rewrite a whole pipeline. You profile, find the top
   20% of CPU (usually one or two functions), and rewrite those in
   Rust as a pyo3 extension. Airflow DAGs, S3 I/O, and business
   rules stay Python."

Minute 3 — when NOT to reach for Rust
  "If the pipeline is I/O-bound — waiting on S3, Postgres, Kafka —
   Rust helps nothing. asyncio and connection pooling are the fix.
   Rust wins only when the profiler says CPU."

Minute 4 — toolchain fluency
  "pyo3 exposes Rust functions to Python via #[pyfunction]. maturin
   builds cross-platform wheels and publishes to PyPI. You get a
   single binary, no C-compiler on the user's box, and pip install
   works everywhere. Test the Rust functions from pytest like any
   Python function."

Minute 5 — ecosystem + when it wins
  "The four-gate decision framework: CPU-bound? > 1 M ops/sec?
   memory-safety requirement? multi-core scaling without the GIL?
   Three yeses and Rust is the right choice. Two or fewer, stay
   Python. That framework is what senior interviewers score."

Step-by-step explanation.

1. Minute 1 is the crucial framing. Naming the pattern immediately — "Python-first, drop to Rust for CPU-bound hot paths" — signals you're a decision-maker, not a language-tourist. Weak candidates say "Rust is faster" and leave the interviewer to fill in why that matters.
2. Minute 2 addresses proportionality before the interviewer asks. This preempts the common trap of committing to a full rewrite, which any senior engineer will immediately push back on. Naming the "top 20% of CPU" rule shows you profile first.
3. Minute 3 is the "when NOT to use" probe. Naming I/O-bound as the disqualifier shows you understand what Rust actually buys (CPU throughput + memory safety + GIL escape) rather than reciting a marketing line.
4. Minute 4 names the toolchain — pyo3 + maturin + wheels. This is the difference between "I've heard about Rust" and "I could ship a Rust extension next week." Wheels are the pragma: users don't need a Rust toolchain to pip install.
5. Minute 5 names the four-gate decision framework. This is the artifact interviewers score highest — reproducible criteria that any candidate can walk through on a whiteboard. Say all four gates unprompted.

Output.

Grading criterion	Weak score	Senior score
Names pattern in minute 1	rare	mandatory
Names proportionality rule	rare	required
Names I/O-bound disqualifier	occasional	mandatory
Names pyo3 + maturin toolchain	rare	senior signal
Names four-gate framework	very rare	senior signal

Rule of thumb. The senior Rust-in-DE answer is a 5-minute monologue that covers all four axes without waiting for the follow-ups. Rehearse it once; deploy it every time.

Senior interview question on Rust in a Python-heavy DE stack

A senior interviewer often opens with: "Your batch pipeline reads 50 GB of Parquet nightly, parses an embedded JSON column, aggregates to daily metrics, and writes to Iceberg. It currently takes 4 hours; the SLA is 90 minutes. You've profiled — 40% of CPU is JSON parse, 30% is aggregation, 15% is Iceberg commit. Walk me through where you would introduce Rust, which components you would leave alone, and how you would defend the decision."

Solution Using a pyo3 JSON-parse extension while leaning on Polars + PyIceberg's existing Rust cores

# Before: pure Python + orjson hot loop
import polars as pl
import orjson
from datetime import date

def nightly_batch(day: date) -> None:
    df = pl.read_parquet(f"s3://raw/events/{day}/")
    # 40% CPU here
    parsed = [orjson.loads(row) for row in df["payload"].to_list()]
    df = df.with_columns(
        pl.Series("user_id",    [p["user_id"]    for p in parsed]),
        pl.Series("event_type", [p["event_type"] for p in parsed]),
        pl.Series("value",      [p["value"]      for p in parsed]),
    )
    # 30% CPU — already Polars (Rust)
    daily = df.group_by(["user_id", "event_type"]).agg(pl.col("value").sum())
    # 15% CPU — PyIceberg hot loops (Rust)
    write_iceberg(daily, table="warehouse.daily_metrics", day=day)

// After: src/lib.rs of the payload_parser crate (Rust)
use pyo3::prelude::*;
use rayon::prelude::*;
use serde::Deserialize;

#[derive(Deserialize)]
struct Payload {
    user_id: String,
    event_type: String,
    value: f64,
}

#[pyfunction]
fn parse_payloads(py: Python, payloads: Vec<String>)
    -> PyResult<(Vec<String>, Vec<String>, Vec<f64>)>
{
    // Release the GIL for the parse — Python threads can run in parallel
    let (users, events, values): (Vec<_>, Vec<_>) = py.allow_threads(|| {
        payloads
            .par_iter()          // rayon: parallel across cores
            .filter_map(|s| {
                let p: Payload = serde_json::from_str(s).ok()?;
                Some(((p.user_id, p.event_type), p.value))
            })
            .unzip()
    });

    let (users, events): (Vec<_>, Vec<_>) = users.into_iter().unzip();
    Ok((users, events, values))
}

#[pymodule]
fn payload_parser(_py: Python, m: &PyModule) -> PyResult<()> {
    m.add_function(wrap_pyfunction!(parse_payloads, m)?)?;
    Ok(())
}

# After: Python call site
import polars as pl
import payload_parser              # our pyo3 extension
from datetime import date

def nightly_batch(day: date) -> None:
    df = pl.read_parquet(f"s3://raw/events/{day}/")

    # 40% CPU → ~5% (8× on the hot loop; rayon uses all cores)
    users, events, values = payload_parser.parse_payloads(df["payload"].to_list())
    df = df.with_columns(
        pl.Series("user_id",    users),
        pl.Series("event_type", events),
        pl.Series("value",      values),
    )

    daily = df.group_by(["user_id", "event_type"]).agg(pl.col("value").sum())
    write_iceberg(daily, table="warehouse.daily_metrics", day=day)

Step-by-step trace.

Concern	Before	After
JSON parse CPU share	40%	~5%
DataFrame aggregation	30% (Polars)	30% (unchanged — already Rust)
Iceberg commit	15% (PyIceberg)	15% (unchanged — already Rust)
Cores used during parse	1 (GIL bound)	N (rayon + allow_threads)
Wall-clock time	4 h	~90 min
Lines of new Rust code	0	~30 (one file)
Python call site delta	(existing)	3 lines swapped

After deployment, the nightly batch fits inside its 90-minute SLA. The Rust extension is one 30-line lib.rs, shipped as a wheel via maturin publish. Airflow, S3 I/O, business rules, Polars aggregations, and PyIceberg commits all stay Python — only the actual hot loop drops to Rust.

Output:

Metric	Before	After
Total wall clock	4 h	90 min
JSON parse (share)	40%	5%
Peak memory	32 GB	18 GB
CPU utilisation	1 core saturated	16 cores saturated
Ops surface	Python + orjson	Python + Rust wheel
New engineering	(baseline)	1 crate, ~30 LOC

Why this works — concept by concept:

• Python-first, Rust-for-hot-paths — the pipeline stays 95% Python; only the one hot loop that dominated the profile drops to Rust. The engineering cost is bounded (one crate) and the maintenance surface stays small.
• pyo3 #[pyfunction] — the Rust function is exposed to Python as if it were a Python function. Signatures use standard types (Vec<String>, f64, tuples) that convert transparently at the boundary. No C API glue, no ctypes structs.
• Python::allow_threads — releases the GIL while the Rust code runs. Other Python threads can execute during that window; more importantly, rayon inside the block can parallelise across cores without contending with any Python thread. This is the GIL-escape mechanism.
• rayon parallel iterator — one line (.par_iter()) turns the sequential parse into a work-stealing parallel one across all cores. Combined with allow_threads, this is the reason the Rust rewrite scales linearly with the box's core count.
• Cost — one Rust crate (~30 LOC), one maturin publish command, one pip install payload-parser in the Airflow image. The eliminated cost is 2.5 hours per nightly batch and a rolling SLA breach. Net O(1) engineering, O(N/cores) runtime versus O(N) baseline. Everything else — Polars, PyIceberg, S3 reads — was already Rust; the only new Rust is the one hot loop the profiler pointed at.

SQL
Topic — sql
SQL aggregation and hot-loop problems

Practice →

Design Topic — design Design problems on Rust/Python hot-path architectures

Practice →

---

2. The 5 categories of Rust adoption in DE

Every Rust dependency you already ship falls into one of five buckets — knowing the map is the senior interview signal

The mental model in one line: rust polars, delta-rs, PyIceberg, DataFusion, Vector, uv, ruff, Windmill, and the other Rust dependencies quietly infiltrating your stack all fall into one of five categories — DataFrame + query engines, table format libraries, streaming + observability, build + tooling, or orchestration + workflow — and every category shares the same "Rust core, ergonomic Python (or CLI) surface" pattern, meaning the strategic question is not whether to adopt but which category is your next dependency in. Every senior interviewer probes category awareness because it's the fastest way to tell whether you understand the Rust-in-DE ecosystem or have only heard of Polars.

Category 1 — DataFrame + query engines.

• Polars. The pandas-successor DataFrame library authored in Rust. Multi-threaded execution, lazy query planner, Arrow-native. Ships as pip install polars; the wheel is a Rust binary via pyo3. On a 10 GB Parquet aggregation, Polars typically beats pandas by 5×–30× with 30%–50% of the memory.
• DataFusion. An Apache Arrow-based SQL query engine written in Rust, part of the Arrow project. Used as an embedded engine in InfluxDB IOx, Ballista, ROAPI, and increasingly as an in-process analytics engine in Python libraries. pip install datafusion gives you a SQL execution engine you can call from a Python process.
• DuckDB extensions. DuckDB itself is C++, but the extension ecosystem is increasingly Rust — the arrow, httpfs, and community extensions like chsql (ClickHouse-compatible SQL) use Rust for CPU-critical paths.

Category 2 — Table format libraries.

• delta-rs. The Rust-native Delta Lake client. pip install deltalake gives you snapshot merges, expression evaluation, and predicate pushdown at Rust speed without a JVM. Used in production by teams that want Delta without Databricks compute.
• PyIceberg's hot loops. PyIceberg is Python-first for the API surface, but the metadata-heavy operations — manifest merges, snapshot planning, file-index maintenance — increasingly delegate to Rust (via pyiceberg-core and the iceberg-rust project). The pattern is identical: Python API, Rust engine.
• Apache Arrow-rs. The Rust implementation of Apache Arrow. Every DataFrame/query library above (Polars, DataFusion, delta-rs) sits on top of it. The Arrow-rs project is a critical shared library for the whole Rust-DE ecosystem.

Category 3 — Streaming + observability.

• Vector. The Datadog-authored observability data-router. Single Rust binary; replaces Fluentd, Logstash, and much of the log-forwarding tier across the industry. Configuration is TOML/YAML; the pipeline logic runs at native speed.
• Materialize. A streaming SQL database with real-time incremental view maintenance, authored in Rust on top of the timely-dataflow framework. Positioned against KsqlDB and Apache Flink for sub-second streaming analytics.
• Ballista. The distributed variant of DataFusion; SQL execution across a cluster with an Arrow-based shuffle protocol. Positioned against Spark for CPU-bound query workloads.
• Databend. A cloud-native modern data warehouse authored in Rust, positioned as an open-source Snowflake alternative. Storage is S3; compute is Rust.

Category 4 — Build + tooling.

• uv (Astral). The Rust-based Python package manager and resolver; 10×–100× faster than pip / poetry for the same lockfile. Rapidly becoming the default in serious Python shops.
• ruff (Astral). The Rust-based Python linter + formatter; 100× faster than flake8 + isort + black combined. Runs in git hooks and IDE loops at near-zero latency.
• prql compiler. The PRQL query language compiler; Rust-authored, targets SQL for any warehouse. Used as a pipeline-friendly SQL alternative in analytics teams.
• Some Dagster crates. Dagster's Python core delegates hot paths (partition-set resolution, asset materialisation graph analysis) to Rust in newer versions.

Category 5 — Orchestration + workflow.

• Windmill. The open-source workflow engine authored in Rust. Positioned as an Airflow / Prefect alternative for teams that want lower-latency task startup and lower-footprint workers.
• MotherDuck backend. The managed DuckDB service's server-side coordination layer is Rust; the client remains DuckDB C++. The orchestration + connection management layer benefits from Rust's async story.
• Emerging orchestrators. A wave of 2025-2026 orchestrators (Ractor for Ray-native workflows, some Kestra components) leans on Rust for scheduler hot paths.

Common interview probes on Rust-in-DE categories.

• "Name three Rust-native data-engineering libraries." — required answer: pick from Polars, delta-rs, DataFusion, Vector, uv, ruff.
• "Which category has the most Rust adoption in 2026?" — DataFrame engines (Polars) + streaming/observability (Vector) are the two headline categories.
• "Why is Rust winning in streaming/observability specifically?" — sub-second latency + memory safety + single-binary deploy = fewer log-forwarder outages.
• "What's the pattern that ties all five categories together?" — Rust core, ergonomic (Python / CLI / SQL) surface.

Worked example — the five-category quick reference and decision hint

Detailed explanation. Every senior DE should be able to reproduce a five-category lookup on a whiteboard and, for each category, name at least two production-grade Rust libraries and the CPU-bound problem they solve. Walk through drawing the table.

• Layout. Five rows, four columns: category, headline libraries, Python surface, CPU-bound problem solved.
• Purpose. When an interviewer asks "would Rust help my X?" the table maps X to a category and a library recommendation in seconds.
• Extension. Add a sixth column for "when the pure-Python answer still wins" — this shows nuance.

Question. Draw the five-category quick-reference table and use it to answer three "would Rust help?" questions.

Input.

Category	Headline libraries	Python surface	CPU problem
DataFrame + query	Polars, DataFusion	pip install polars/datafusion	multi-GB group-by, joins, agg
Table format	delta-rs, PyIceberg core	pip install deltalake, pyiceberg	manifest merges, snapshot plans
Streaming + obs	Vector, Materialize, Ballista	binary / SQL	log routing, streaming SQL
Build + tooling	uv, ruff, prql	CLI	resolver, lint, transpile
Orchestration	Windmill, MotherDuck backend	web UI, driver	task startup, coordination

Code.

Scenario 1: "My pandas group-by on 20 GB is slow."
  → Category: DataFrame + query
  → Library: Polars (lazy .group_by).
  → Wall-clock: ~10× faster than pandas; 40% memory.

Scenario 2: "My Kafka → S3 log forwarder crashes weekly."
  → Category: Streaming + observability
  → Library: Vector (single Rust binary; TOML config).
  → Ops: single binary, no runtime; typical uptime measured in months.

Scenario 3: "My pip install takes 8 minutes in CI."
  → Category: Build + tooling
  → Library: uv (Rust-based resolver).
  → Wall-clock: 8 min → ~15 sec.

Step-by-step explanation.

1. Category 1 (DataFrame + query engines) is the entry drug for most teams. Adopting Polars or DataFusion replaces a slow pandas/Spark pipeline with an in-process Rust engine at a fraction of the ops cost. The Python surface (import polars as pl) is intentionally pandas-shaped so migration is a subset of DataFrame API translation.
2. Category 2 (table formats) matters for lakehouse teams. delta-rs gives you Delta without Databricks; PyIceberg's Rust hot loops give you Iceberg-native operations at Snowflake-competitive latency. Both categories are win-by-dependency — you don't write Rust, you pip install it.
3. Category 3 (streaming + observability) is where Rust's "single binary + memory safety + native async" story dominates. Vector alone has replaced entire fleets of Fluentd installations across the industry. The interview signal is naming Vector alongside Materialize and Ballista.
4. Category 4 (build + tooling) is the developer-productivity story. uv + ruff shaved hours off every Python-DE team's CI and edit-loop timings. The category exists because Astral (their author) explicitly positioned Rust-in-tooling as the wedge into the Python community.
5. Category 5 (orchestration + workflow) is the newest and least mature. Windmill is the headline Rust orchestrator; MotherDuck's server-side story shows Rust winning inside coordination layers. Expect this category to grow through 2026-2027.

Output.

"Would Rust help my …"	Category	Library	Wins because
pandas group-by on 20 GB	DataFrame	Polars	Rust core + lazy planner
Kafka → S3 log routing	Streaming/obs	Vector	Single binary + no runtime
slow pip install in CI	Build/tooling	uv	Rust resolver
Delta merges from Python	Table format	delta-rs	no JVM; Rust engine
custom aggregations at 10 M/s	DataFrame	Polars UDF or pyo3 ext	Rust math + rayon

Rule of thumb. Memorise the five categories with two libraries each. Any "would Rust help?" question you can't map to a category means you haven't understood the ecosystem — go read the Polars, delta-rs, DataFusion, Vector, and uv READMEs in that order.

Worked example — swapping pandas for Polars on a 10 GB group-by

Detailed explanation. The category-1 archetype: a pandas pipeline that runs a group-by-aggregate on a 10 GB Parquet file and takes 8 minutes plus 32 GB of RAM. Swapping to Polars typically cuts wall-clock by 5×–10× and memory by 2×–3× with a two-line change. Walk through the swap.

• Before. pandas read_parquet + groupby.sum on 10 GB → 8 min wall clock, 32 GB peak memory, single-core saturated.
• After. Polars scan_parquet + .group_by().agg().collect() → ~60 sec, ~12 GB peak, all cores saturated.
• Why so much faster. Rust core, lazy planner (pushes filters into the Parquet reader), multi-threaded execution, Arrow-native columnar layout.

Question. Rewrite the pandas group-by as Polars and quantify the wins.

Input.

Component	pandas	Polars
Read	pd.read_parquet	pl.scan_parquet (lazy)
Group-by	df.groupby(...).sum()	df.group_by(...).agg(pl.col(...).sum())
Execution	eager, single-thread	lazy, multi-thread
Memory	32 GB peak	~12 GB peak
Wall clock	8 min	~60 sec

Code.

# Before — pandas
import pandas as pd

df = pd.read_parquet("s3://raw/events/2026-07-20/")   # loads entire file
daily = (df
    .groupby(["user_id", "event_type"], as_index=False)["value"]
    .sum())
daily.to_parquet("s3://curated/daily/2026-07-20/")

# After — Polars (lazy)
import polars as pl

daily = (pl.scan_parquet("s3://raw/events/2026-07-20/")
    .group_by(["user_id", "event_type"])
    .agg(pl.col("value").sum())
    .collect())                        # single .collect() triggers the plan

daily.write_parquet("s3://curated/daily/2026-07-20/")

# Wall-clock and memory numbers (on a 32-vCPU, 128 GB box, 10 GB Parquet)
pandas:       8 min 12 s   peak 32 GB   1 core used
Polars:       58 s         peak 11 GB   32 cores used
Speedup:      ~8.5×        memory 2.9×  parallel 32×

Step-by-step explanation.

1. The API surface change is minimal: pd.read_parquet becomes pl.scan_parquet (lazy), .groupby(...).sum() becomes .group_by(...).agg(pl.col(...).sum()), and a .collect() at the end triggers the query plan. The migration is measured in minutes per DataFrame operation, not weeks.
2. scan_parquet is lazy — it reads only the columns and row groups the plan actually needs. Column projection and predicate pushdown mean Polars often reads 10-30% of the physical bytes pandas would.
3. Polars' Rust core parallelises the group-by across cores using its work-stealing scheduler. Every core participates; there is no GIL. On a 32-vCPU box, the speedup is close to the core count for CPU-bound operations.
4. Memory drops because Polars uses Arrow's columnar layout: fixed-size null buffers, dictionary-encoded strings, and no per-row Python object overhead. A 10 GB Parquet often lives in ~11 GB of Arrow buffers vs 32 GB of pandas object columns.
5. The change costs you almost nothing in Python readability. The lazy plan's .explain() method prints a query plan that's easier to reason about than the pandas equivalent. Testing changes exactly one line (assert isinstance(daily, pl.DataFrame)).

Output.

Metric	pandas	Polars	Ratio
Wall clock	8 m 12 s	58 s	8.5×
Peak memory	32 GB	11 GB	2.9×
Cores used	1	32	32×
Lines changed	(baseline)	~3	trivial
Ops surface	pip install pandas	pip install polars	swap

Rule of thumb. Any pandas job on > 5 GB of data is a Polars candidate. The migration cost is a few lines per pipeline; the wins are 5×–30× in wall clock and 2×–3× in memory. Do the swap once at the DataFrame boundary; leave everything else Python.

Worked example — using Vector to replace a fragile Kafka → S3 log forwarder

Detailed explanation. The category-3 archetype: a Python or JVM-based log forwarder (Fluentd, Logstash, custom Python consumer) tailing Kafka and writing to S3, that crashes weekly on runtime bugs, memory leaks, or thread deadlocks. Vector replaces it with one Rust binary and a TOML config; typical uptime jumps from days to months. Walk through the swap.

• Before. Fluentd + a custom Ruby filter → CPU 60%, memory 4 GB, weekly restarts.
• After. Vector 0.42 → CPU 15%, memory 500 MB, uptime measured in months.
• Why so much better. Single Rust binary, memory-safe transforms, native async I/O, no runtime.

Question. Replace the Fluentd deployment with a Vector deployment reading Kafka and writing to S3.

Input.

Component	Fluentd	Vector
Runtime	Ruby VM	native binary
Config	Ruby DSL	TOML / YAML
Memory	~4 GB	~500 MB
CPU	~60% of one core	~15% of one core
Uptime	days	months

Code.

# /etc/vector/vector.toml — the whole config
[api]
enabled = true
address = "127.0.0.1:8686"

[sources.kafka_in]
type = "kafka"
bootstrap_servers = "kafka:9092"
group_id = "vector-log-forwarder"
topics = ["app.logs"]
auto_offset_reset = "latest"

[transforms.parse_json]
type = "remap"
inputs = ["kafka_in"]
source = '''
  . = parse_json!(.message)
  .ingest_ts = now()
'''

[transforms.route_env]
type = "route"
inputs = ["parse_json"]
route.prod = '.env == "prod"'
route.dev  = '.env == "dev"'

[sinks.s3_prod]
type = "aws_s3"
inputs = ["route_env.prod"]
bucket = "acme-logs-prod"
key_prefix = "app-logs/%Y/%m/%d/%H/"
compression = "gzip"
encoding.codec = "ndjson"
batch.max_bytes = 10_000_000
batch.timeout_secs = 60

[sinks.s3_dev]
type = "aws_s3"
inputs = ["route_env.dev"]
bucket = "acme-logs-dev"
key_prefix = "app-logs/%Y/%m/%d/%H/"
compression = "gzip"
encoding.codec = "ndjson"

# Deploy
sudo systemctl enable vector
sudo systemctl start vector
sudo systemctl status vector   # active (running)

# Test the config before restart
vector validate --config /etc/vector/vector.toml

# Live-tail the pipeline
vector top   # Rich TUI: throughput, buffer sizes, error rates

Step-by-step explanation.

1. Vector is a single statically-linked Rust binary — no Ruby VM, no JVM, no Python runtime. apt install vector (or the tarball) drops one file, and systemctl start vector is the entire deploy. This is the "single binary" argument in action.
2. The TOML config expresses the pipeline as source → transform(s) → sink. sources.kafka_in reads Kafka; transforms.parse_json runs a VRL (Vector Remap Language) expression that parses JSON and adds an ingest timestamp; transforms.route_env splits the stream by environment; two sinks.s3_* write compressed NDJSON to per-environment buckets.
3. Memory drops from ~4 GB to ~500 MB because Vector doesn't hold a full JSON DOM per message — VRL operates on parsed values in-place and streams through fixed-size buffers. There are no per-message Ruby object allocations.
4. CPU drops from ~60% to ~15% for the same throughput because Rust's zero-cost abstractions compile to tight loops, and the async runtime saturates the network I/O without thread context-switch overhead. On the same box, you can typically 5×–10× the message throughput without adding hardware.
5. Uptime jumps because Rust's memory-safety guarantees eliminate whole classes of crashes (use-after-free, buffer overrun, unhandled thread-safety bugs). Vector deployments routinely run for 6+ months without a restart; the operations team just stops thinking about the log-forwarder tier.

Output.

Metric	Fluentd	Vector	Improvement
Binary footprint	Ruby runtime + gems	1 file, ~80 MB	massive
Memory	~4 GB	~500 MB	8×
CPU	~60% one core	~15% one core	4×
Config surface	Ruby DSL	TOML + VRL	simpler
Uptime	days	months	30×+
Ops load	weekly restarts	quarterly patches	~30× fewer touches

Rule of thumb. Any log-forwarder / observability pipeline built on a garbage-collected runtime is a Vector migration candidate. The rewrite is a config translation, not a code rewrite. Expect memory drops of 5×–10× and ops touches to move from weekly to quarterly.

Senior interview question on category-aware Rust adoption

A senior interviewer might ask: "You inherit a Python DE stack: pandas for DataFrames, custom Airflow operators, a Fluentd log-forwarder, pip install taking 10 min in CI, and a Delta Lake pipeline running through Spark. Walk through which components you would replace with a Rust-native library from each of the five categories, in what order, and what you would leave alone."

Solution Using a five-category migration order with per-swap wins and left-alone rationale

# Migration order — largest ROI per week of engineering time

# Week 1: Build + tooling  (Category 4)
#   Replace pip → uv           (10 min → 15 sec CI)
#   Replace flake8+black → ruff (30 sec → 0.3 sec per lint pass)
#   Zero pipeline risk; developers happy immediately.

# Week 2-3: DataFrame + query  (Category 1)
#   Replace pandas → Polars for the two heaviest jobs
#   Wall-clock cut 8× on those; memory cut 3×.
#   Leave notebook-scale pandas alone (< 100 MB DFs; not worth touching).

# Week 4: Streaming + observability  (Category 3)
#   Replace Fluentd → Vector
#   Memory 8× lower; uptime months instead of days.

# Week 5-6: Table format  (Category 2)
#   Replace Spark-mediated Delta ops → delta-rs (from Python)
#   No JVM to babysit; Delta merges from Python at Rust speed.

# Later (optional): Orchestration  (Category 5)
#   Airflow stays; too much invested. Consider Windmill for greenfield
#   projects only; not a swap.

# The "leave alone" list — as important as the swap list
LEAVE_ALONE = [
    "Airflow DAGs",                    # orchestration; too much invested
    "psycopg2 / SQLAlchemy",           # I/O bound; Rust helps nothing
    "boto3",                           # I/O bound
    "notebook pandas < 100 MB",        # too small; Polars migration cost > win
    "custom business rules",           # clarity beats speed at 0.1% of CPU
    "test suites",                     # pytest is fine; no Rust win
]

# The "swap" list — profile-verified CPU wins
SWAP = {
    "pip":       "uv         (10 min → 15 sec CI)",
    "flake8":    "ruff       (100× faster lint)",
    "black":     "ruff format(20× faster format)",
    "pandas":    "polars     (5-30× on > 5 GB DFs)",
    "Fluentd":   "vector     (8× memory; months uptime)",
    "Delta+JVM": "delta-rs   (no JVM; Python-native)",
    # Later (if profiled hot):
    "orjson hot loop": "pyo3 ext (5-10× on tight loops)",
}

# Rollout order with rollback plan per stage
# Stage 1: uv + ruff — dev-side only; no production impact
#   Rollback: switch back to pip / flake8 in CI config; ~5 min

# Stage 2: pandas → polars on the top-2 jobs
#   Rollback: revert the two files; original pandas code still on main

# Stage 3: Fluentd → Vector
#   Rollback: `systemctl stop vector; systemctl start fluentd`
#   Both configs live in /etc; toggle by symlink

# Stage 4: Delta+JVM → delta-rs
#   Rollback: keep the Spark job in the DAG (disabled); flip if needed

Step-by-step trace.

Week	Category	Swap	Wall-clock / ops win
1	Build + tooling	pip → uv, flake8+black → ruff	CI 10 min → 15 sec; lint 100× faster
2-3	DataFrame + query	pandas → Polars (top-2 jobs)	5-30× on the big DFs
4	Streaming + obs	Fluentd → Vector	8× memory; uptime months
5-6	Table format	Spark+Delta → delta-rs	no JVM; Python-native
Later	Orchestration	(leave Airflow; use Windmill greenfield)	no swap

After the six-week rollout, the pipeline runs on: uv + ruff for dev, Polars for the heavy DataFrame jobs, Vector for logs, delta-rs for Delta ops, and Airflow + boto3 + psycopg2 for the I/O-bound layers that Rust would not help. The team never wrote a line of Rust — every win came from pip install of a Rust dependency in the right category.

Output:

Layer	Before	After	Rust dependency
CI install	pip (10 min)	uv (15 sec)	Category 4
Lint	flake8+black (30 s)	ruff (0.3 s)	Category 4
Big DataFrames	pandas	polars	Category 1
Log forwarder	Fluentd	Vector	Category 3
Delta operations	Spark + JVM	delta-rs	Category 2
Orchestration	Airflow	Airflow (unchanged)	(none)
S3 / DB I/O	boto3 / psycopg2	(unchanged)	(I/O bound; Rust irrelevant)

Why this works — concept by concept:

• Category-first migration — order swaps by ROI per engineering week, not by "which library sounds coolest." Build/tooling swaps ship in a day and improve every developer's inner loop; orchestration swaps take a quarter and often don't pay off.
• Leave-alone list — every senior migration has an explicit "not swapping" list. Naming Airflow, boto3, and small-DF pandas as "not worth touching" shows judgment; a candidate who says "rewrite everything in Rust" fails this signal.
• Rollback per stage — every swap has a documented rollback (dual config, feature flag, or symlink). This is the "senior engineer" pattern: no big-bang cutovers, no bridges burnt.
• Zero-Rust-code migration — the entire six-week plan involves zero new Rust code — the team just pip installs libraries someone else wrote in Rust. The Rust extension slot is reserved for the next project when the profiler points at a hot loop no existing library covers.
• Cost — six engineer-weeks of migration, five pip installs, one Vector deploy, one delta-rs adoption, zero new Rust source files. The eliminated cost is a JVM to babysit, a Fluentd on-call rotation, 8 minutes of CI per commit, and hours per big-DataFrame job. Net O(swaps) engineering, O(job) runtime savings — the swap list is bounded, the wins compound daily.

SQL
Topic — sql
SQL practice on DataFrame-shaped workloads

Practice →

Streaming Topic — streaming Streaming problems on log routing and pipelines

Practice →

---

3. pyo3 + maturin — your first Python-facing Rust extension

pyo3 + maturin collapses "write a Rust library that Python can import" from a two-week C-extension project to a two-hour crate

The mental model in one line: pyo3 is the Rust crate that exposes Rust functions and structs to Python via #[pyfunction] / #[pyclass] attributes, and maturin is the build tool that compiles the crate into a Python wheel (.whl) that pip install handles like any other package — together they turn "write a Python extension in a systems language" from a specialty skill (Cython, C API, ctypes) into a Cargo-native workflow any Rust-literate DE can ship in a week. Every serious Python-facing Rust library in 2026 — Polars, Pydantic v2, cryptography, orjson, tokenizers — uses pyo3 + maturin under the hood.

The install workflow — five commands to a working extension.

• Rust toolchain. curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh — one script, one restart of the shell. This installs rustup, rustc, and cargo.
• maturin. pip install maturin — a single Python-side tool that drives Cargo and packages wheels.
• Scaffold. maturin new --bindings pyo3 my_ext — creates Cargo.toml, pyproject.toml, and src/lib.rs wired for pyo3.
• Develop. maturin develop — compiles the Rust crate into a .so/.pyd and installs it into the current virtualenv. You can import my_ext from a REPL the next second.
• Publish. maturin build --release for wheels; maturin publish to PyPI. Wheels are per-platform (Linux x86_64, Linux aarch64, macOS x86_64, macOS aarch64, Windows x86_64) and get built in CI via cibuildwheel or the maturin GitHub Action.

Exposing a Rust function to Python — the #[pyfunction] macro.

• The macro. #[pyfunction] on a Rust fn makes it callable from Python. Parameters and return types must implement the IntoPy / FromPyObject traits; standard types (i64, f64, String, Vec<T>, HashMap<K, V>, tuples) implement them out of the box.
• The module. A #[pymodule] function defines the Python module and registers each pyfunction. Every wheel exposes one #[pymodule].
• The GIL. By default, the Rust function holds the GIL while it runs. Python::allow_threads(|py| { ... }) releases the GIL for the block — mandatory for any CPU-bound work you want to parallelise across cores.
• Errors. Rust Result<T, E> converts to a Python exception when E implements From<E> for PyErr. Standard anyhow::Error and std::io::Error work out of the box; custom errors need one impl.

Type conversion — what crosses the FFI boundary and what does not.

• Copies at the boundary. String, Vec<T>, HashMap<K, V> are cloned on both sides of the boundary. For > 100 MB payloads, this doubles memory transiently.
• Zero-copy via Arrow / NumPy. For large numeric arrays, use numpy (via the numpy crate) or arrow (via pyarrow) — both cross the boundary as zero-copy buffer views. This is the pattern for high-throughput DataFrame work.
• Custom classes. #[pyclass] on a Rust struct exposes it as a Python class with methods (#[pymethods]). Useful for stateful extensions (parsers, tokenizers, streaming readers).
• Async. #[pyfunction] supports async fn via the pyo3-asyncio crate — Rust futures become Python awaitables. Useful when the Rust code itself needs async I/O.

Publishing to PyPI — the wheels story.

• Per-platform wheels. Rust binaries are per-platform; you need one wheel per (OS, CPU, Python-minor). maturin build --release --strip produces one wheel for the current platform; CI matrices produce the full set.
• abi3 wheels. Setting abi3-py38 in Cargo.toml's pyo3 features means one wheel per platform works across all Python 3.8+ minor versions — cuts wheel-count and build time.
• cibuildwheel + maturin action. The PyO3/maturin-action GitHub Action + pypa/cibuildwheel produces the full wheel matrix and publishes via a GitHub Release trigger. Setup is ~50 lines of YAML.
• pip install UX. End users type pip install my_ext and get the right wheel for their platform — no Rust toolchain required on the user's box.

Common interview probes on pyo3 + maturin.

• "How do you expose a Rust function to Python?" — required answer: #[pyfunction] + #[pymodule] + maturin build.
• "How do you release the GIL?" — required answer: Python::allow_threads(|| { ... }).
• "How do you avoid a copy at the FFI boundary?" — Arrow arrays via pyarrow, or numpy::PyArray for numeric arrays.
• "How do users install without Rust?" — pre-built wheels published to PyPI; pip install picks the right one.
• "How do you test?" — pytest against maturin develop; the Rust functions look like Python functions from pytest's POV.

Worked example — a pyo3 + maturin project from maturin new to pip install

Detailed explanation. The canonical "first Rust extension" project: a fast_wordcount crate that counts words in a batch of strings, exposes one function to Python, releases the GIL, and ships as a wheel. Walk through every command and file.

• Scaffold. maturin new --bindings pyo3 fast_wordcount.
• Files. Cargo.toml, pyproject.toml, src/lib.rs, .github/workflows/CI.yml.
• Function. wordcount(strings: Vec<String>) -> HashMap<String, u64>.
• Test. pytest against maturin develop.
• Publish. GitHub Actions maturin publish.

Question. Scaffold, implement, test, and publish the fast_wordcount Rust extension end-to-end.

Input.

Component	Value
Crate name	fast_wordcount
Function	wordcount(Vec) -> HashMap
GIL	released via Python::allow_threads
Wheel target	Linux x86_64 + macOS + Windows
Publish	via maturin-action on GitHub Releases

Code.

# 1. Scaffold
maturin new --bindings pyo3 fast_wordcount
cd fast_wordcount

# Files created:
#   Cargo.toml
#   pyproject.toml
#   src/lib.rs

# 2. Cargo.toml — pyo3 with abi3 for one-wheel-per-platform
[package]
name    = "fast_wordcount"
version = "0.1.0"
edition = "2021"

[lib]
name       = "fast_wordcount"
crate-type = ["cdylib"]

[dependencies]
pyo3 = { version = "0.22", features = ["extension-module", "abi3-py38"] }

// 3. src/lib.rs — the whole extension
use pyo3::prelude::*;
use std::collections::HashMap;

#[pyfunction]
fn wordcount(py: Python, docs: Vec<String>) -> PyResult<HashMap<String, u64>> {
    // Release GIL — the count loop is pure CPU
    let counts: HashMap<String, u64> = py.allow_threads(|| {
        let mut m: HashMap<String, u64> = HashMap::new();
        for doc in &docs {
            for word in doc.split_whitespace() {
                *m.entry(word.to_lowercase()).or_insert(0) += 1;
            }
        }
        m
    });
    Ok(counts)
}

#[pymodule]
fn fast_wordcount(_py: Python, m: &PyModule) -> PyResult<()> {
    m.add_function(wrap_pyfunction!(wordcount, m)?)?;
    Ok(())
}

# 4. Develop + import from the current venv
python -m venv .venv && source .venv/bin/activate
pip install maturin pytest
maturin develop --release

python -c "
import fast_wordcount as fw
print(fw.wordcount(['hello world', 'hello Rust', 'world of Rust']))
# → {'hello': 2, 'world': 2, 'rust': 2, 'of': 1}
"

# 5. tests/test_wordcount.py — treat Rust function like a Python function
import fast_wordcount as fw

def test_basic():
    result = fw.wordcount(["hello world", "hello Rust"])
    assert result == {"hello": 2, "world": 1, "rust": 1}

def test_empty():
    assert fw.wordcount([]) == {}

def test_case_insensitive():
    result = fw.wordcount(["Foo FOO foo"])
    assert result == {"foo": 3}

Step-by-step explanation.

1. maturin new scaffolds three files: Cargo.toml (Rust build), pyproject.toml (Python packaging), and src/lib.rs (Rust source). One command; no template repo cloning; no boilerplate.
2. Cargo.toml pulls pyo3 with the extension-module and abi3-py38 features. The abi3-py38 feature is what makes one wheel work across every Python 3.8+ minor version — a huge win for the CI matrix.
3. src/lib.rs defines the extension in ~15 lines. #[pyfunction] marks the exported function; Python::allow_threads releases the GIL for the count loop; #[pymodule] registers the function under the crate name.
4. maturin develop --release compiles the crate with --release optimisations and installs the .so into the current venv — a fresh python -c "import fast_wordcount" works immediately. This is the fast inner-loop equivalent of pip install -e ..
5. pytest sees fast_wordcount.wordcount as a plain Python function — you write assertions on inputs/outputs, and any Rust panic surfaces as a Python exception with a stack trace. The Rust side gets covered by cargo tests separately if desired.

Output.

Step	Command	Result
Scaffold	maturin new --bindings pyo3	Cargo.toml + pyproject.toml + src/lib.rs
Build	maturin develop --release	.so installed in venv
Import	python -c "import fast_wordcount"	works
Test	pytest	3 tests pass
Wheel	maturin build --release	one .whl in target/wheels/
Publish	maturin publish	pypi upload

Rule of thumb. For any Python-facing Rust extension, use maturin new --bindings pyo3 + abi3-py38 in Cargo.toml + Python::allow_threads inside any CPU-bound block. That is the entire template; every serious pyo3 project starts from these three choices.

Worked example — measuring the speedup vs the pure-Python equivalent

Detailed explanation. Every Rust extension needs a benchmark that proves it earns its keep. Wrap the Rust wordcount and a pure-Python equivalent in a timeit harness, run on identical input, and report the ratio. Any senior engineer will ask this before approving the dependency.

• Test corpus. 100 000 short strings, each ~200 characters, total ~20 MB text.
• Baseline. Pure-Python wordcount using collections.Counter and .split().
• Rust. fast_wordcount.wordcount from the extension above.
• Measurement. timeit with 10 repeats, best-of-3.

Question. Benchmark the Rust wordcount against the pure-Python equivalent and report the ratio.

Input.

Metric	Pure Python	Rust ext
Wall clock (per call)	~4.2 s	~0.35 s
Speedup	1×	~12×
Peak memory	~450 MB	~180 MB
CPU cores used	1 (GIL)	1 (single fn)

Code.

# bench.py
import timeit
from collections import Counter
from fast_wordcount import wordcount as rust_wc

def py_wc(docs):
    c = Counter()
    for d in docs:
        c.update(w.lower() for w in d.split())
    return dict(c)

# Generate 100k docs, ~200 chars each
import random, string
words = ["".join(random.choices(string.ascii_lowercase, k=6)) for _ in range(2000)]
docs  = [" ".join(random.choices(words, k=30)) for _ in range(100_000)]

def bench(fn, name):
    t = min(timeit.repeat(lambda: fn(docs), number=1, repeat=5))
    print(f"{name:20s} {t:6.2f} s")
    return t

t_py   = bench(py_wc,   "pure python")
t_rust = bench(rust_wc, "rust pyo3")
print(f"speedup: {t_py / t_rust:.1f}x")

# Typical output on a mid-range laptop
pure python           4.22 s
rust pyo3             0.35 s
speedup: 12.1x

Step-by-step explanation.

1. The benchmark generates a realistic corpus — 100 000 documents of ~200 characters each, drawn from a 2 000-word vocabulary. Total ~20 MB of text; enough to make the Python loop's per-word Python-object allocation visible.
2. timeit.repeat(number=1, repeat=5) runs each function five times and takes the minimum — a standard defence against GC and OS noise. The min picks the run where nothing else on the box interfered.
3. The pure-Python version uses collections.Counter.update with a generator expression. This is a fast pure-Python baseline (much faster than a plain for word in doc.split() loop) so the 12× ratio is a fair comparison against an already-optimised Python starting point.
4. Peak memory drops from ~450 MB to ~180 MB because Rust's HashMap<String, u64> stores keys as owned String (heap-allocated but no per-word Python object overhead), and there's no Counter intermediate.
5. Both functions use one core — the Rust version does not automatically parallelise (that would need rayon::par_iter()). The 12× win is pure "Rust vs Python" on a single core. Adding rayon would push it to 60-100× on an 8-core box.

Output.

Metric	Pure Python	Rust ext	Ratio
Wall clock	4.22 s	0.35 s	12×
Peak memory	450 MB	180 MB	2.5×
Cores	1	1 (add rayon for more)	1×
Install size	(part of stdlib)	~1 MB wheel	acceptable
Test story	pytest	pytest + cargo test	parallel

Rule of thumb. Every Rust extension must ship a benchmark script that proves > 5× speedup against a fast pure-Python baseline. A 2× ratio does not justify the dependency; a 10× ratio does. The benchmark also protects against regressions when you upgrade pyo3.

Worked example — publishing pre-built wheels to PyPI via GitHub Actions

Detailed explanation. The last mile of any Rust extension is shipping pre-built wheels so users never need a Rust toolchain. The PyO3/maturin-action GitHub Action + a version-tag trigger publishes the full wheel matrix (Linux x86_64/aarch64, macOS x86_64/aarch64, Windows x86_64) to PyPI on every release. Walk through the CI config.

• Trigger. GitHub Release published (or tag push).
• Matrix. Five targets covering >95% of user boxes.
• Publish. maturin publish with a PyPI API token from GitHub Secrets.
• User experience. pip install fast_wordcount picks the right wheel automatically.

Question. Write the GitHub Actions workflow that publishes wheels on every release.

Input.

Component	Value
Trigger	release: published
Targets	linux x86_64, linux aarch64, macos x86_64, macos arm64, windows x86_64
Tool	PyO3/maturin-action@v1
Publish	maturin upload --skip-existing
Secret	PYPI_API_TOKEN

Code.

# .github/workflows/release.yml
name: Build and publish wheels
on:
  release:
    types: [published]

jobs:
  linux:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        target: [x86_64, aarch64]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - name: Build wheels
        uses: PyO3/maturin-action@v1
        with:
          target: ${{ matrix.target }}
          args: --release --out dist --strip
          manylinux: auto
      - name: Upload artifacts
        uses: actions/upload-artifact@v4
        with:
          name: wheels-linux-${{ matrix.target }}
          path: dist

  macos:
    runs-on: macos-latest
    strategy:
      matrix:
        target: [x86_64, aarch64]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - name: Build wheels
        uses: PyO3/maturin-action@v1
        with:
          target: ${{ matrix.target }}
          args: --release --out dist --strip
      - name: Upload artifacts
        uses: actions/upload-artifact@v4
        with:
          name: wheels-macos-${{ matrix.target }}
          path: dist

  windows:
    runs-on: windows-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - name: Build wheel
        uses: PyO3/maturin-action@v1
        with:
          args: --release --out dist --strip
      - uses: actions/upload-artifact@v4
        with:
          name: wheels-windows
          path: dist

  publish:
    needs: [linux, macos, windows]
    runs-on: ubuntu-latest
    steps:
      - uses: actions/download-artifact@v4
        with: { path: dist, merge-multiple: true }
      - name: Publish to PyPI
        uses: PyO3/maturin-action@v1
        env:
          MATURIN_PYPI_TOKEN: ${{ secrets.PYPI_API_TOKEN }}
        with:
          command: upload
          args: --non-interactive --skip-existing dist/*

Step-by-step explanation.

1. The on: release: published trigger fires when you publish a GitHub Release — draft the release, click Publish, and the workflow starts. No manual maturin publish on someone's laptop.
2. The linux job uses a matrix over [x86_64, aarch64] and manylinux: auto so wheels are compatible with the widest set of distros (manylinux_2_17 covers CentOS 7-era to current). Two wheels per Python-minor if abi3 is off; one wheel total with abi3-py38.
3. The macos job covers both Intel and Apple Silicon (x86_64 and aarch64). The macos-latest runner has been Apple-Silicon-native since 2024, and cross-compiling to x86_64 is handled by the maturin action.
4. The windows job builds an x86_64 wheel (Windows arm64 exists but has < 1% install share). One wheel; one job.
5. The publish job depends on all three build jobs; it downloads all wheel artifacts, then maturin upload uploads to PyPI. --skip-existing handles the case where a partial re-run tries to re-upload a wheel that succeeded the first time.

Output.

Platform / arch	Wheel filename (abi3)	Size
Linux x86_64	fast_wordcount-0.1.0-cp38-abi3-manylinux_2_17_x86_64.whl	~800 KB
Linux aarch64	fast_wordcount-0.1.0-cp38-abi3-manylinux_2_17_aarch64.whl	~800 KB
macOS x86_64	fast_wordcount-0.1.0-cp38-abi3-macosx_10_12_x86_64.whl	~700 KB
macOS arm64	fast_wordcount-0.1.0-cp38-abi3-macosx_11_0_arm64.whl	~700 KB
Windows x86_64	fast_wordcount-0.1.0-cp38-abi3-win_amd64.whl	~500 KB

Rule of thumb. Ship pre-built wheels for the 5-platform matrix above on every release via PyO3/maturin-action. Never make users install a Rust toolchain to pip install your library — that's the boundary between "a Rust dependency" and "a Rust project you have to build."

Senior interview question on pyo3 + maturin

A senior interviewer might ask: "You want to ship a Rust-authored JSON parser as a Python library. Walk me through the whole workflow — Cargo config, the #[pyfunction] for parsing a batch under released GIL, the pytest suite, the CI wheels for Linux + macOS + Windows, and how end users install without touching Rust."

Solution Using a pyo3 batch-parse function + Python::allow_threads + GitHub Actions wheel matrix

# Cargo.toml
[package]
name    = "fast_json"
version = "0.1.0"
edition = "2021"

[lib]
name       = "fast_json"
crate-type = ["cdylib"]

[dependencies]
pyo3       = { version = "0.22", features = ["extension-module", "abi3-py38"] }
serde      = { version = "1",    features = ["derive"] }
serde_json = "1"
rayon      = "1"

// src/lib.rs
use pyo3::prelude::*;
use rayon::prelude::*;
use serde_json::Value;

#[pyfunction]
fn parse_batch(py: Python, payloads: Vec<String>) -> PyResult<Vec<String>> {
    // Release GIL; use rayon to parallelise across cores
    let out: Vec<String> = py.allow_threads(|| {
        payloads
            .par_iter()
            .map(|s| {
                // Round-trip parse+serialize as the "normalise" contract
                let v: Value = serde_json::from_str(s).unwrap_or(Value::Null);
                serde_json::to_string(&v).unwrap_or_default()
            })
            .collect()
    });
    Ok(out)
}

#[pymodule]
fn fast_json(_py: Python, m: &PyModule) -> PyResult<()> {
    m.add_function(wrap_pyfunction!(parse_batch, m)?)?;
    Ok(())
}

# tests/test_batch.py
import fast_json

def test_normalises():
    out = fast_json.parse_batch(['{"a":1,"b":2}', '{"x":true}'])
    assert out[0] in ('{"a":1,"b":2}', '{"b":2,"a":1}')   # key order ok
    assert out[1] == '{"x":true}'

def test_bad_json_becomes_null():
    assert fast_json.parse_batch(['not json']) == ['null']

def test_scale():
    payloads = ['{"n": %d}' % i for i in range(100_000)]
    out = fast_json.parse_batch(payloads)
    assert len(out) == 100_000

# .github/workflows/release.yml — 5-platform wheel matrix
name: wheels
on: { release: { types: [published] } }
jobs:
  linux:
    runs-on: ubuntu-latest
    strategy: { matrix: { target: [x86_64, aarch64] } }
    steps:
      - uses: actions/checkout@v4
      - uses: PyO3/maturin-action@v1
        with:
          target: ${{ matrix.target }}
          args: --release --out dist --strip
          manylinux: auto
      - uses: actions/upload-artifact@v4
        with: { name: wheels-linux-${{ matrix.target }}, path: dist }
  macos:
    runs-on: macos-latest
    strategy: { matrix: { target: [x86_64, aarch64] } }
    steps:
      - uses: actions/checkout@v4
      - uses: PyO3/maturin-action@v1
        with: { target: ${{ matrix.target }}, args: --release --out dist --strip }
      - uses: actions/upload-artifact@v4
        with: { name: wheels-macos-${{ matrix.target }}, path: dist }
  windows:
    runs-on: windows-latest
    steps:
      - uses: actions/checkout@v4
      - uses: PyO3/maturin-action@v1
        with: { args: --release --out dist --strip }
      - uses: actions/upload-artifact@v4
        with: { name: wheels-windows, path: dist }
  publish:
    needs: [linux, macos, windows]
    runs-on: ubuntu-latest
    steps:
      - uses: actions/download-artifact@v4
        with: { path: dist, merge-multiple: true }
      - uses: PyO3/maturin-action@v1
        env: { MATURIN_PYPI_TOKEN: ${{ secrets.PYPI_API_TOKEN }} }
        with: { command: upload, args: --non-interactive --skip-existing dist/* }

Step-by-step trace.

Layer	Choice	Reasoning
pyo3 features	extension-module + abi3-py38	one wheel per platform across Python 3.8+
GIL	allow_threads(...)	rayon can use all cores
Parallelism	rayon .par_iter()	work-stealing scheduler; O(N/cores)
Error handling	unwrap_or(Value::Null)	one bad payload never crashes the batch
Tests	pytest against maturin develop	Rust functions look like Python functions
CI	5-platform matrix via maturin-action	800 KB wheels per platform; abi3 keeps count low
Publish	maturin upload on release	one-click release; no laptop uploads

After the workflow runs, PyPI has fast_json 0.1.0 with five wheels; pip install fast_json on any of those five platforms grabs the right one and installs in under a second. Users never see Rust; the maintainer never uploads from a laptop.

Output:

Metric	Value
Wall-clock CI (release job)	~7 min end-to-end
Wheels published	5 (Linux x64/arm64, macOS x64/arm64, Windows x64)
User install command	pip install fast_json
Rust toolchain on user's box	not required
Test story	pytest + optional cargo test
Ongoing maintenance	1 dep upgrade PR per pyo3 release

Why this works — concept by concept:

• #[pyfunction] + #[pymodule] — pyo3's two attribute macros are the whole Python-side API. One #[pyfunction] per exported function, one #[pymodule] per crate. Nothing else — no manual C-API boilerplate, no Py_INCREF.
• abi3-py38 — the ABI-stable pyo3 mode. One wheel per platform works across all Python 3.8+ versions, cutting the CI matrix in half and eliminating the "user upgraded Python and my wheel broke" support ticket class.
• Python::allow_threads + rayon — the GIL escape + parallelisation combo that makes CPU-bound Rust extensions actually scale. Without allow_threads, other Python threads block; without rayon, you use one core. Both together = the machine's full CPU.
• pytest against maturin develop — Rust functions get the same test framework as Python functions. Assertions on inputs/outputs; Rust panics surface as Python exceptions with stack traces. The Rust side gets cargo test separately if unit-testing internal helpers.
• Cost — one Cargo.toml, one src/lib.rs, one CI file, one PyPI token in GitHub Secrets. The eliminated cost is a per-user Rust toolchain, a wheel-builder side project, and the "why doesn't this install?" support burden of source-only distributions. Net O(1) per release; the maintainer clicks "Publish Release" and pyo3 + maturin + Actions do the rest.

Design
Topic — design
Design problems on Python-facing Rust extensions

Practice →

SQL Topic — sql SQL practice on tight-loop data transformations

Practice →

---

4. When Rust wins vs when Python + NumPy is enough

The four-gate decision framework — CPU-bound, > 1 M ops/sec, memory-safety, multi-core scaling; three yeses and Rust is the right call

The mental model in one line: rust vs python is not a language war but a four-gate decision — is the workload CPU-bound, does it exceed 1 M ops/sec, does it require memory-safety guarantees under concurrent access, and does it need multi-core scaling without the GIL — and if three or more gates say yes, a rust data pipeline component is the right call, while if two or fewer say yes, pure Python + NumPy (or Polars, or asyncio) is the correct choice and Rust would be over-engineering. Every senior interviewer probes this framework because it separates candidates who "just like Rust" from candidates who profile and decide.

Gate 1 — CPU-bound?

• The test. Profile the workload (py-spy, cProfile, scalene). If > 50% of time is in CPU instructions (not select(), not sleep, not network I/O), the workload is CPU-bound and gate 1 is yes.
• Common false positives. A pipeline that spends 90% of its time waiting on a slow database query is I/O-bound even if the query itself is CPU-intensive on the DB side — the client is idle.
• Rule of thumb. If py-spy's top view shows a Python function eating > 20% of the flame graph, that function is the CPU-bound hot path. Anything less and the pipeline is I/O-bound overall.

Gate 2 — > 1 M ops/sec throughput requirement?

• The test. What's the per-second operation count the workload must sustain? Rows/sec, requests/sec, events/sec, bytes/sec — pick the natural unit and count.
• Where 1 M draws the line. Pure-Python loops top out at ~1 M ops/sec for the simplest possible body. NumPy vectorised ops hit ~10-100 M ops/sec. Rust hits 100 M-1 B ops/sec depending on the operation. If your target is below 1 M ops/sec, Python usually works; above, Rust or NumPy is required.
• Why not always NumPy? NumPy vectorisation wins for numeric arrays with fixed shapes. It loses when the operation is per-row branchy (variable-length strings, nested JSON, custom aggregations that can't be expressed as a broadcast).

Gate 3 — Memory-safety requirement?

• The test. Does the component share buffers across threads? Does it hold pointers into memory-mapped files? Does it decode untrusted input? All three make memory safety a first-order concern.
• Where C/C++ historically failed. Log forwarders (Fluentd bugs in Ruby-C bridge), streaming sinks (JVM heap corruption under concurrent writes), custom Parquet readers (buffer overruns in C++ code) — the industry's rogue-CPU horror stories cluster around unsafe systems languages.
• Where Rust wins. The borrow checker rejects use-after-free, double-free, and data-race bugs at compile time. Vector, Materialize, and delta-rs all cite this as a headline reason for their reliability.

Gate 4 — Multi-core scaling without the GIL?

• The test. Does the workload need to saturate more than one core inside a single process? Multiprocessing sidesteps the GIL but adds inter-process overhead; a single-process multi-thread solution is required if you're paying for high-core-count boxes.
• Where the GIL bites. Any pure-Python CPU-bound loop uses exactly one core no matter how many threads you spawn. A 32-core box gets 1-core throughput on pip install requests scraping.
• How Rust escapes. Python::allow_threads releases the GIL for the block; rayon inside parallelises across cores. Combined, the Rust function scales linearly with core count.

The 3-of-4 rule.

• 3 or 4 yeses → Rust. The engineering cost of a pyo3 extension is justified.
• 2 yeses → Polars or DataFusion. Get the Rust win by picking the right library, not by writing new Rust.
• 1 yes → NumPy or vectorised Python. Vectorise the hot loop; you're not CPU-bound enough or throughput-hungry enough to justify Rust.
• 0 yeses → pure Python. The workload is I/O-bound or low-throughput; Rust would be pure over-engineering.

Common interview probes on the decision framework.

• "How do you decide when to reach for Rust?" — required answer: the four gates + 3-of-4 rule.
• "Would you rewrite an I/O-bound API server in Rust?" — no; asyncio + connection pooling first; Rust rewrite only if profile says CPU.
• "What's the fastest way to know if a workload is CPU-bound?" — py-spy or scalene; look at the CPU-vs-I/O breakdown.
• "When is NumPy enough?" — vectorisable numeric ops on fixed-shape arrays; you gain 10-100× over pure Python with no Rust cost.

Worked example — parsing JSON at line rate

Detailed explanation. The classic Rust-wins case study: a streaming service parses ~20 000 JSON messages per second, each ~500 bytes. Pure Python + orjson tops out at ~5 000 msgs/sec on one core (GIL bound); a pyo3 Rust extension with serde_json + rayon hits ~100 000 msgs/sec. Walk through the four-gate scoring.

• Gate 1 (CPU-bound). Yes — profile shows 85% CPU in JSON parse, 10% network, 5% dispatch.
• Gate 2 (> 1 M ops/sec). Approaching — 100 k msgs/sec × ~50 fields = 5 M field-parses/sec. Yes.
• Gate 3 (memory-safety). Yes — untrusted input; buffer boundary parsing.
• Gate 4 (GIL escape). Yes — need to saturate 8 cores; pure Python locks at 1.
• Score. 4/4 yeses → Rust.

Question. Score the JSON-parse-at-line-rate scenario against the four gates and prescribe the fix.

Input.

Gate	Answer	Evidence
CPU-bound?	yes	py-spy: 85% in orjson.loads
> 1 M ops/sec?	yes	5 M field-parses/sec target
Memory-safety?	yes	untrusted network payloads
Multi-core scaling?	yes	8-core box, need all 8

Code.

# Before — pure Python + orjson on one core
import asyncio
import orjson

async def handle(msg_bytes: bytes) -> dict:
    return orjson.loads(msg_bytes)

# Throughput: ~5 000 msgs/sec (1 core, GIL bound)

// After — Rust pyo3 extension parsing a batch under released GIL
use pyo3::prelude::*;
use rayon::prelude::*;
use serde_json::Value;

#[pyfunction]
fn parse_batch_bytes(py: Python, batch: Vec<Vec<u8>>) -> PyResult<Vec<String>> {
    let out: Vec<String> = py.allow_threads(|| {
        batch.par_iter()
            .map(|b| {
                let v: Value = serde_json::from_slice(b).unwrap_or(Value::Null);
                serde_json::to_string(&v).unwrap_or_default()
            })
            .collect()
    });
    Ok(out)
}

# After — Python call site batches and calls Rust
from fast_json import parse_batch_bytes

async def handle_batch(msgs: list[bytes]) -> list[str]:
    # Batch of ~1000 messages; Rust parses all in parallel across cores
    return parse_batch_bytes(msgs)

# Throughput: ~100 000 msgs/sec (8 cores saturated)

Step-by-step explanation.

1. Gate 1: py-spy top on the running Python service shows orjson.loads and its call-tree eating 85% of CPU. The service is not blocked on I/O — it's blocked on the parse itself. Gate 1 = yes.
2. Gate 2: 20 000 msgs/sec × 50 fields ≈ 1 M field-parses/sec today; target 100 000 msgs/sec × 50 = 5 M field-parses/sec. Well above the 1 M threshold; gate 2 = yes.
3. Gate 3: The messages come from untrusted network peers. A buffer overrun or use-after-free in a hand-rolled C parser is a security incident. Rust's borrow checker rules those out at compile time. Gate 3 = yes.
4. Gate 4: The service runs on 8-core boxes. Pure Python uses 1 core regardless of thread count. Batching + Python::allow_threads + rayon gets all 8. Gate 4 = yes.
5. 4/4 yeses → the Rust extension is the right call. The Python call site changes from "one message at a time" to "batch of ~1000 → single call into Rust", which is a straightforward refactor. Throughput jumps 20× on the same hardware.

Output.

Metric	Before (orjson)	After (Rust pyo3)	Ratio
Throughput	5 k msgs/sec	100 k msgs/sec	20×
CPU cores used	1	8	8×
Peak memory	400 MB	180 MB	2×
Boxes needed for 100 k msgs/sec	20	1	20× cheaper
Latency p99	5 ms	2 ms	2.5×

Rule of thumb. For any streaming service parsing untrusted JSON at > 20 k msgs/sec, the four gates score 4/4 and a pyo3 extension pays for itself in under a month of engineering. The batching pattern (accumulate ~1000 messages, one call into Rust) is the standard shape.

Worked example — custom aggregation at 10 M rows/sec

Detailed explanation. The DataFrame case study: a nightly batch computes a custom "P99 of P95 of latency" aggregation across 100 M rows. Polars' built-in .agg() covers 90% of aggregations but not this composite one. Options: (a) Polars map_groups with a Python UDF (~200 k rows/sec — too slow), (b) NumPy vectorisation (breaks because the grouping is dynamic), (c) pyo3 + Rust extension implementing the aggregation directly (~15 M rows/sec). Walk through the gate scoring.

• Gate 1 (CPU-bound). Yes — profile shows the UDF eating 90% CPU.
• Gate 2 (> 1 M ops/sec). Yes — target 10 M rows/sec.
• Gate 3 (memory-safety). No — data is trusted internal; buffers are Polars-managed.
• Gate 4 (GIL escape). Yes — 16-core box needed.
• Score. 3/4 yeses → Rust extension (barely justified; could stay Polars with clever partitioning).

Question. Score the custom-aggregation scenario, then implement it as a Polars-integrated Rust function.

Input.

Gate	Answer	Evidence
CPU-bound?	yes	90% CPU in the UDF
> 1 M ops/sec?	yes	10 M rows/sec target
Memory-safety?	no	trusted data; Polars manages buffers
Multi-core scaling?	yes	16-core box

Code.

# Baseline — Polars .map_groups with a Python UDF (SLOW)
import polars as pl
import numpy as np

def p99_of_p95(group: pl.DataFrame) -> pl.DataFrame:
    latencies = group["latency_ms"].to_numpy()
    # Non-trivial: bucket-by-minute, p95 per bucket, then p99 across buckets
    minutes = group["minute"].to_numpy()
    p95s = []
    for m in np.unique(minutes):
        p95s.append(np.percentile(latencies[minutes == m], 95))
    return pl.DataFrame({"p99_of_p95": [np.percentile(p95s, 99) if p95s else 0.0]})

df = pl.read_parquet("s3://raw/requests/2026-07-20/")
result = df.group_by("service").map_groups(p99_of_p95)
# ~200 k rows/sec; the Python UDF is the bottleneck

// Rust: implement the aggregation directly on numpy arrays
use pyo3::prelude::*;
use numpy::{PyArray1, PyReadonlyArray1};
use std::collections::HashMap;

#[pyfunction]
fn p99_of_p95_rust(
    py: Python,
    latencies: PyReadonlyArray1<f64>,
    minutes: PyReadonlyArray1<i64>,
) -> PyResult<f64> {
    let latencies = latencies.as_slice()?;
    let minutes   = minutes.as_slice()?;

    let result = py.allow_threads(|| {
        // Bucket latencies by minute
        let mut buckets: HashMap<i64, Vec<f64>> = HashMap::new();
        for (i, &m) in minutes.iter().enumerate() {
            buckets.entry(m).or_default().push(latencies[i]);
        }
        // p95 per bucket
        let mut p95s: Vec<f64> = buckets.into_values()
            .map(|mut v| {
                v.sort_by(|a, b| a.partial_cmp(b).unwrap());
                v[(v.len() as f64 * 0.95) as usize]
            })
            .collect();
        // p99 across buckets
        p95s.sort_by(|a, b| a.partial_cmp(b).unwrap());
        p95s.get((p95s.len() as f64 * 0.99) as usize).copied().unwrap_or(0.0)
    });

    Ok(result)
}

# Python — call the Rust aggregation per group
import polars as pl
import numpy as np
from my_agg import p99_of_p95_rust

def p99_of_p95_fast(group: pl.DataFrame) -> pl.DataFrame:
    r = p99_of_p95_rust(
        group["latency_ms"].to_numpy(),
        group["minute"].to_numpy(),
    )
    return pl.DataFrame({"p99_of_p95": [r]})

result = df.group_by("service").map_groups(p99_of_p95_fast)
# ~15 M rows/sec — Rust does the aggregation; Polars does the group split

Step-by-step explanation.

1. Gate 1: scalene shows the Python UDF using 90% CPU — the group iteration and np.percentile calls per minute-bucket compound to a slow inner loop. Gate 1 = yes.
2. Gate 2: 100 M rows across 5 000 groups × ~50 rows per minute-bucket × 5 minute-buckets per group ≈ 25 M row-touches. Well above 1 M ops/sec. Gate 2 = yes.
3. Gate 3: Data is internal request logs; buffers are Polars-managed Arrow arrays. No memory-safety concern beyond what NumPy already offers. Gate 3 = no.
4. Gate 4: The batch runs on a 16-core box and today uses 1 core. Rust + allow_threads gets all 16. Gate 4 = yes.
5. 3/4 yeses → Rust extension. The numpy crate lets you pass Polars/NumPy arrays into Rust as zero-copy slices (as_slice()), so the FFI boundary is cheap. The Rust code implements the aggregation directly; Polars handles the group split. Throughput jumps from 200 k rows/sec to 15 M rows/sec — a 75× win.

Output.

Metric	Python UDF	Rust ext	Ratio
Throughput	200 k rows/sec	15 M rows/sec	75×
Wall clock (100 M rows)	8 min	6 sec	80×
Cores used	1	16	16×
Memory	3 GB	1.5 GB	2×
Lines of Rust	0	~25	modest

Rule of thumb. For custom aggregations that Polars' built-ins don't cover and that must run at > 1 M rows/sec, a pyo3 extension with numpy zero-copy passthrough is the standard solution. Reach for Polars UDFs first; drop to Rust only when the profiler proves the UDF is the bottleneck.

Worked example — sub-100 ms streaming transforms

Detailed explanation. The streaming case study: a real-time enrichment service receives Kafka messages, joins each against a Redis feature-store cache, applies a scoring function, and writes to a downstream topic. The SLA is < 100 ms end-to-end p99. Pure Python + asyncio hits ~40 ms at 1 000 msgs/sec but degrades to 200 ms at 5 000 msgs/sec due to GIL contention. A Rust rewrite with tokio + rdkafka + async-Redis holds 40 ms at 20 000 msgs/sec. Walk through the gate scoring.

• Gate 1 (CPU-bound). Partial — 60% time in scoring (CPU), 40% in Redis+Kafka (I/O). Yes-ish.
• Gate 2 (> 1 M ops/sec). No — only 20 000 msgs/sec target. Well below 1 M ops/sec.
• Gate 3 (memory-safety). Yes — long-running streaming service; memory leaks are outages.
• Gate 4 (GIL escape). Yes — need consistent p99 under load.
• Score. 3/4 yeses → Rust rewrite (or a strategically-placed Rust extension inside a Python service).

Question. Score the streaming-enrichment scenario, and decide between "full Rust service" and "Python service + Rust hot path."

Input.

Gate	Answer	Evidence
CPU-bound?	partially	60% CPU scoring + 40% I/O
> 1 M ops/sec?	no	20 k msgs/sec target
Memory-safety?	yes	long-running; memory leak = outage
Multi-core scaling?	yes	p99 degrades from GIL contention

Code.

# Option A: Python service + Rust scoring extension (cheap; 1-week rewrite)
import asyncio
from aiokafka import AIOKafkaConsumer, AIOKafkaProducer
from redis.asyncio import Redis
from fast_scorer import score_batch   # Rust ext

async def enrich(msgs, redis):
    keys = [m["user_id"] for m in msgs]
    features = await redis.mget(keys)             # I/O — stays Python
    scores = score_batch(msgs, features)          # CPU — Rust
    return [{**m, "score": s} for m, s in zip(msgs, scores)]

async def main():
    consumer = AIOKafkaConsumer("events.raw", bootstrap_servers="kafka:9092")
    producer = AIOKafkaProducer(bootstrap_servers="kafka:9092")
    redis    = Redis(host="redis")
    await consumer.start(); await producer.start()
    async for batch in consumer.getbatch(max_records=1000):
        enriched = await enrich(batch, redis)
        for e in enriched:
            await producer.send("events.enriched", e)

// Option B: Full Rust service — 4-week rewrite; use if p99 must stay < 40 ms
use rdkafka::consumer::{StreamConsumer, Consumer};
use rdkafka::producer::{FutureProducer, FutureRecord};
use redis::AsyncCommands;
use tokio::main;

#[main]
async fn main() -> anyhow::Result<()> {
    let consumer: StreamConsumer = /* ... */;
    let producer: FutureProducer = /* ... */;
    let mut redis = redis::Client::open("redis://redis")?.get_async_connection().await?;

    consumer.subscribe(&["events.raw"])?;
    while let Ok(msg) = consumer.recv().await {
        let payload: Event = serde_json::from_slice(msg.payload().unwrap_or(&[]))?;
        let feature: String = redis.get(&payload.user_id).await?;
        let score = score(&payload, &feature);
        producer.send(FutureRecord::to("events.enriched")
            .payload(&serde_json::to_string(&Enriched { payload, score })?)
            .key(&payload.user_id), std::time::Duration::from_secs(1)).await?;
    }
    Ok(())
}

Step-by-step explanation.

1. Gate 1: The scoring function is CPU-bound (matrix operations, per-message math); the Kafka + Redis calls are I/O-bound. Overall the service is mixed — 60% CPU, 40% I/O. Gate 1 is a "partial yes" — enough to matter for the CPU part.
2. Gate 2: 20 000 msgs/sec is well below the 1 M ops/sec threshold. Gate 2 = no. This is the axis that makes "full Rust rewrite" over-engineering; Python + async can handle the throughput.
3. Gate 3: The service runs for weeks between restarts. A memory leak in a C-extension or a Ruby-C bridge would be an outage. Rust's borrow checker + no-GC design make long-running memory profiles predictable. Gate 3 = yes.
4. Gate 4: Under load, GIL contention causes p99 to spike from 40 ms to 200 ms. Any component that runs across multiple asyncio tasks on the CPU-bound scoring will suffer. Gate 4 = yes.
5. 3/4 yeses → Rust, but the shape of the Rust matters. Option A (Python service + Rust scoring extension) covers the CPU-bound part with 1 week of engineering. Option B (full Rust service) covers everything but takes 4 weeks and rewrites the async I/O layer. Pick Option A first; escalate to Option B only if p99 SLA still fails.

Output.

Metric	Pure Python	Option A (Py + Rust ext)	Option B (Full Rust)
p99 at 5 k msgs/sec	200 ms	45 ms	25 ms
p99 at 20 k msgs/sec	timeout	60 ms	30 ms
Peak memory	2 GB	900 MB	400 MB
Boxes needed	4	1	1
Engineering weeks	(baseline)	1	4
Ops load	high	medium	low

Rule of thumb. For streaming services that hit 3/4 gates, prefer "Python service + Rust hot-path extension" (Option A) over "full Rust rewrite" (Option B). Option A ships 4× faster and covers the CPU-bound portion; only escalate to Option B when p99 pressure demands it and the pipeline is stable enough to justify the rewrite.

Senior interview question on the four-gate framework

A senior interviewer might ask: "You have three candidate workloads: (1) a nightly Airflow DAG that reconciles 500 Postgres tables against a warehouse, (2) a real-time feature computation service running at 5 000 requests/sec, (3) a batch ETL parsing 50 GB of nested JSON per hour. For each, walk through the four-gate framework and prescribe pure Python, Polars/DataFusion, or a pyo3 Rust extension."

Solution Using the four-gate framework applied to three workloads with per-workload prescription

# Workload 1: Nightly Airflow reconcile of 500 tables
# Gate 1 (CPU-bound?):     NO  — 90% time in psycopg2/SQLAlchemy I/O
# Gate 2 (> 1M ops/sec?):  NO  — throughput measured in tables/hour
# Gate 3 (memory-safety?): NO  — short-lived tasks; failures acceptable
# Gate 4 (GIL escape?):    NO  — I/O bound; asyncio would help, not Rust
# Score: 0/4 → PURE PYTHON.
#   Use asyncpg + asyncio.gather for parallel table checks.
#   Rust would be pure over-engineering.

async def reconcile_all(pool, tables):
    return await asyncio.gather(*[reconcile_table(pool, t) for t in tables])

# Workload 2: Real-time feature service at 5 000 req/sec
# Gate 1 (CPU-bound?):     PARTIALLY — 60% CPU in scoring; 40% I/O
# Gate 2 (> 1M ops/sec?):  NO  — 5 000 req/sec target
# Gate 3 (memory-safety?): YES — long-running; leaks = outages
# Gate 4 (GIL escape?):    YES — p99 needs consistent tail latency
# Score: 2.5/4 → POLARS or targeted Rust ext.
#   First try: vectorise scoring with NumPy inside asyncio service.
#   If p99 still slips: pyo3 extension for the scoring function only.

# Try NumPy first — one file, no new dependencies
import numpy as np
def score_batch_np(features):
    x = np.asarray(features)   # zero-copy
    return (x @ WEIGHTS + BIAS).tolist()

# Workload 3: Batch ETL parsing 50 GB nested JSON/hour
# Gate 1 (CPU-bound?):     YES — 70% CPU in parse
# Gate 2 (> 1M ops/sec?):  YES — ~15M field-parses/sec target
# Gate 3 (memory-safety?): YES — untrusted vendor payloads
# Gate 4 (GIL escape?):    YES — 32-core box; need all of it
# Score: 4/4 → RUST EXT.
#   Write parse_batch(Vec<Vec<u8>>) -> Vec<ParsedRow> in Rust.
#   Batch 1 000 messages per call; use rayon inside allow_threads.

# Python call site
from vendor_parser import parse_batch_bytes
parsed = parse_batch_bytes(chunk)   # 30× faster than orjson-in-loop

# Prescription summary — the interview-ready one-liners

Workload 1: nightly reconcile
  → 0/4 → PURE PYTHON + asyncio.
  → Rust would be over-engineering.

Workload 2: real-time features
  → 2.5/4 → NumPy first; drop to Rust ext ONLY if p99 slips.
  → Full Rust rewrite = premature.

Workload 3: JSON batch ETL
  → 4/4 → Rust pyo3 extension.
  → Textbook case for the "hot path" pattern.

Step-by-step trace.

Workload	G1	G2	G3	G4	Score	Prescription
Nightly reconcile	no	no	no	no	0/4	pure Python + asyncio
Feature service	partial	no	yes	yes	2.5/4	NumPy first, Rust ext if p99 slips
JSON batch ETL	yes	yes	yes	yes	4/4	pyo3 extension (textbook case)

After the analysis, the team ships (1) a pure-asyncio reconcile job (no Rust), (2) a NumPy-vectorised feature service with a fallback to a Rust extension if the p99 SLA slips, and (3) a pyo3 JSON-parse extension for the batch ETL. Two of the three workloads never touch new Rust; only the one with 4/4 yeses does.

Output:

Workload	Language / library	Wins because	Rust engineering
Nightly reconcile	Python + asyncio	I/O-bound	zero
Feature service	Python + NumPy (Rust ext if needed)	CPU is small; NumPy covers	zero at first
JSON batch ETL	Python + pyo3 extension	4/4 gates	one crate

Why this works — concept by concept:

• The four gates — CPU-bound? > 1 M ops/sec? memory-safety? GIL escape? Four yes/no questions any senior engineer can answer from a profile in under 10 minutes. Reproducible; interview-friendly; no language wars.
• 3-of-4 rule — 3 or 4 yeses justifies Rust; 2 yeses points to Polars/DataFusion; 1 yes points to NumPy; 0 yeses points to pure Python. Every workload gets a prescription without guessing.
• NumPy as the intermediate rung — for CPU-bound but < 1 M ops/sec workloads, vectorising with NumPy typically closes 80% of the gap with zero new dependencies. Rust is the last resort, not the first.
• Python service + Rust hot path — the composed shape covers most 3/4 scenarios. Ship the Rust piece as a wheel; keep the Python glue; win the CPU battle without the full-rewrite risk.
• Cost — one profiling pass per workload (py-spy or scalene), one whiteboard score against the four gates, one prescription. The eliminated cost is the "let's rewrite everything in Rust" cargo-cult decision that wastes months on workloads Rust would never help. Net O(4) questions per workload; the correct prescription falls out of the score.

Streaming
Topic — streaming
Streaming problems on real-time enrichment services

Practice →

Design Topic — design Design problems on CPU-bound hot-path decisions

Practice →

---

5. Career + adoption signals + interview probes

Learn just enough Rust to profile, defend, and ship a hot-path decision — and know which companies actually hire for it

The mental model in one line: the 2026 data-engineering career signal for Rust is not "senior Rust developer" but "Python-first DE who can profile, decide, and ship a targeted Rust extension" — you need enough Rust literacy to walk through ownership, the borrow checker, traits, async/tokio, and cargo, plus enough interview vocabulary to defend a "I profiled this, a Rust rewrite gives 5× on the hot loop" answer, plus enough industry awareness to name the companies (Datadog Vector team, Databricks Photon, InfluxData IOx, Polars Inc, Confluent Freight, RisingWave, Materialize) whose entire hiring pipeline is Rust-DE.

How much Rust does a DE actually need?

• Ownership + borrow checker (mandatory). The single mental model that separates Rust from every other language. move, &, &mut, lifetimes on the basics. You need this to read Rust source; not knowing it means you can't debug a Polars issue.
• Traits + generics (mandatory). How Rust does polymorphism. impl Trait, dyn Trait, where clauses. You need this to understand the Rust libraries you pip install.
• cargo + workspaces (mandatory). The build tool. cargo new, cargo build, cargo test, cargo publish. Cargo.toml and Cargo.lock. You need this to add a Rust extension crate to your project.
• Async + tokio (nice to have). Rust's async story. .await, Future, tokio::spawn. Needed only when your Rust component itself does async I/O — for a batch parse extension, not needed.
• Advanced (not needed for DE). Macros, unsafe, custom allocators, embedded, WASM. Fun; not on the interview.

Interview signals — the phrases that separate senior from junior.

• "I profiled this and found the top 20% of CPU is in one function." — signals the profile-first mindset.
• "A Rust rewrite would give 5× on the hot loop; the rest stays Python." — signals proportionality.
• "If the pipeline is I/O-bound, stay in Python." — signals gate-1 awareness.
• "pyo3 + maturin; ship pre-built wheels; test with pytest." — signals toolchain fluency.
• "Polars is Rust; delta-rs is Rust; DataFusion is Rust; Vector is Rust — we're already using it." — signals ecosystem awareness.
• "Python::allow_threads inside a rayon par_iter." — signals GIL-escape fluency.

When NOT to learn Rust.

• I/O-bound pipelines only. If every workload you own is bounded by S3, Postgres, or Kafka wait times, Rust helps you nothing. asyncio + connection pooling is the answer.
• Career optimisation for pure analytics. If your day-job is dbt + Snowflake + BI, no interviewer will probe you on Rust. Learn SQL, dimensional modelling, and warehouse tuning first.
• Team dynamics. If your team is 15 people who all write Python and nobody else knows Rust, being the sole Rust maintainer is an operational risk. Adopt only if at least 2 team-mates can maintain the extension.
• Small pipelines. If your batch job runs in 20 minutes and the SLA is 4 hours, there's no CPU pressure to solve. Rust is a solution looking for a problem.

Companies that actually hire Rust-DE — the 2026 shortlist.

• Datadog Vector team. The Vector observability router is a headline Rust codebase. Datadog hires Rust engineers specifically for the Vector team.
• Databricks Photon. The C++ vectorised query engine has an increasing Rust footprint; Databricks hires for the engine team with Rust as a plus.
• InfluxData IOx. IOx is a pure-Rust time-series engine on top of DataFusion. Every IOx role is Rust-first.
• Polars Inc. The commercial entity behind Polars. Small team; every hire writes Rust.
• Confluent Freight. Confluent's storage-tier rewrite in Rust; hiring for the storage team.
• RisingWave Labs. RisingWave is a streaming database written in Rust; hires Rust engineers exclusively.
• Materialize. The Materialize streaming SQL engine is Rust-native. Materialize hires Rust engineers.
• Snowflake / AWS / Databricks (adjacent). Not primary Rust shops but increasingly have Rust footprints in specific teams (Snowflake's data-processing internals, AWS Firehose, Databricks Photon).

Common interview probes on Rust-DE career signals.

• "How much Rust do you actually know?" — required answer: ownership, borrow checker, traits, cargo, pyo3 basics.
• "When would you NOT reach for Rust?" — I/O-bound; small pipelines; solo maintainer risk.
• "Name 3 companies where Rust-DE is a first-class hiring signal." — Datadog Vector, InfluxData IOx, Polars Inc (or Materialize / RisingWave / Confluent).
• "How would you learn Rust if you had a month?" — official Rust Book chapters 1-13 + one weekend pyo3 project.

Worked example — the DE Rust skill ladder and a 4-week learning path

Detailed explanation. The pragmatic skill ladder for a Python DE learning Rust: don't try to become a systems programmer; learn just enough to profile, decide, and ship a pyo3 hot-path extension. The ladder has five rungs; the whole path fits in four weeks of evenings. Walk through it.

• Rung 1 (week 1). Ownership + borrow checker + basic types. Rust Book chapters 1-4.
• Rung 2 (week 2). Traits + generics + error handling. Rust Book chapters 5-9.
• Rung 3 (week 3). cargo + testing + one small crate. Rust Book chapters 11-14.
• Rung 4 (week 4). pyo3 + maturin + your first extension. Ship a wheel to TestPyPI.
• Rung 5 (later). Async + tokio. Only when a Rust component of yours actually needs async I/O.

Question. Sketch the 4-week Rust-for-DE learning plan and the pass criterion for each week.

Input.

Week	Topic	Rust Book chapters	Pass criterion
1	ownership + basic types	1-4	can read simple Rust functions
2	traits + error handling	5-9	can write a Result-returning fn
3	cargo + testing	11-14	can ship a crate with tests
4	pyo3 + maturin	(official pyo3 book)	wheel installs from TestPyPI

Code.

// Week 1 pass criterion — read this and understand every line
fn word_lens(words: Vec<String>) -> Vec<usize> {
    let mut out = Vec::with_capacity(words.len());
    for w in &words {                 // borrow words, don't move
        out.push(w.len());
    }
    out
}

// Week 2 pass criterion — write this from scratch
use std::num::ParseIntError;

fn sum_nums(strings: &[&str]) -> Result<i64, ParseIntError> {
    strings.iter().map(|s| s.parse::<i64>()).sum()
}

// Week 3 pass criterion — a Cargo project with a test
#[cfg(test)]
mod tests {
    use super::*;
    #[test]
    fn test_sum() {
        assert_eq!(sum_nums(&["1", "2", "3"]).unwrap(), 6);
    }
}

# Week 4 pass criterion — ship a wheel to TestPyPI
maturin new --bindings pyo3 hello_ext
cd hello_ext
# Edit src/lib.rs to add one #[pyfunction]
maturin build --release
maturin publish --repository testpypi
# Then, from a fresh venv:
pip install --index-url https://test.pypi.org/simple/ hello_ext
python -c "import hello_ext; print(hello_ext.hello())"

Step-by-step explanation.

1. Week 1 (ownership + basic types) is the hardest week; the borrow checker rejects code that looks fine to a Python or Java brain. Push through by writing 10 tiny functions per day and letting rustc reject them until the model clicks. Chapters 1-4 of the Rust Book take ~12 hours of focused reading.
2. Week 2 (traits + error handling) unlocks the "why does every Rust function return Result?" question. Chapters 5-9 introduce Result, Option, ?, and trait bounds. By end of week 2, you can read a Polars source file and follow the trait plumbing.
3. Week 3 (cargo + testing) covers the build story. cargo new, cargo test, cargo publish, Cargo.toml dependencies. You publish one library crate (a wordcount, a fib, a JSON validator) to crates.io. Chapters 11-14.
4. Week 4 is the payoff: pyo3 + maturin. Follow the official pyo3 book's quickstart, write one function, build a wheel, publish to TestPyPI, install from a fresh venv, import from Python. Total: one weekend. If you get to pip install my_ext; python -c "import my_ext", you've completed the ladder.
5. Rung 5 (async + tokio) is optional for a DE. Learn it only when you're shipping a Rust component that itself does async I/O (a Kafka consumer, an HTTP server). For a batch-parse extension, it's not needed.

Output.

Week	Time investment	Skill unlocked	Interview signal
1	~12 h	read Rust source	can debug a Polars issue
2	~10 h	write Result-returning fns	can review Rust PRs
3	~8 h	ship a Cargo crate	can maintain a Rust dependency
4	~8 h	ship a pyo3 wheel	can prescribe a Rust ext in an interview
Later	(project-driven)	async + tokio	can write a full Rust service

Rule of thumb. Four weeks of evenings, one pyo3 wheel shipped to TestPyPI. That is the Rust-for-DE skill floor — enough to profile, decide, and ship a hot-path extension. Async + tokio are earned by later projects; you don't need them to pass a Rust-DE interview.

Worked example — the "why you should hire me for Rust-DE" 2-minute pitch

Detailed explanation. Every senior DE interviewer eventually asks "what's your Rust story?" The 2-minute pitch answers three things: (a) what you've shipped, (b) how you decide, (c) what you're already using. Rehearse it once; deploy it whenever the topic comes up.

• What you've shipped. One or two concrete pyo3 extensions, with numbers ("15× on the hot loop, cut the batch from 4 h to 90 min").
• How you decide. The four-gate framework — profile first, count yeses, prescribe.
• What you're already using. Polars, delta-rs, Vector, uv, ruff — Rust cores under Python surfaces.

Question. Draft the 2-minute Rust-DE pitch you would deploy in an interview.

Input.

Element	Content
Shipped	one pyo3 crate for JSON parse; 15× speedup
Decide	four-gate framework (CPU-bound, 1M ops/sec, mem-safety, GIL)
Using	Polars, delta-rs, Vector, uv, ruff

Code.

2-minute Rust-DE pitch
======================

"My Rust story is Python-first, drop-to-Rust-for-hot-paths. Last
quarter I profiled a nightly batch that was blowing its SLA — 40%
of CPU was in a JSON parse loop. I wrote a pyo3 extension using
serde_json + rayon under released GIL. Cut the parse from 40% to
5% of the batch and the wall-clock from 4 hours to 90 minutes.
That's the shape of Rust I write: targeted, wheel-shipped, and
under 100 lines.

I decide with a four-gate framework — CPU-bound, greater than 1M
ops/sec, memory-safety requirement, GIL escape. Three yeses and I
reach for Rust; two or fewer and I stay Python or reach for Polars.
It keeps me from cargo-culting rewrites and it's what I use to
defend the decision to the team.

I already ship Polars for DataFrames above 5 GB, delta-rs for
Delta operations from Python, Vector for log routing, uv and ruff
for our CI. Most Rust wins I get are by picking the right library,
not by writing new Rust. The one crate I did write shipped as a
wheel via maturin + a five-platform GitHub Actions matrix; users
pip install it and never see the Rust."

Step-by-step explanation.

1. Sentence 1-2 name the pattern: Python-first, drop-to-Rust-for-hot-paths. This is the senior framing that separates you from "I like Rust" candidates.
2. Sentences 3-6 tell one concrete story with numbers. "40% to 5%" and "4 hours to 90 minutes" are the concrete-outcome hooks interviewers remember 30 minutes later. Vague "I made things faster" claims don't land.
3. Sentence 7-9 name the four-gate framework. The interviewer will probe "how do you decide" within 5 minutes; naming the framework unprompted gets you ahead of the probe.
4. Sentences 10-13 name the ecosystem — Polars, delta-rs, Vector, uv, ruff. This signals that you use the wide Rust ecosystem, not just "I built one extension." Interviewers score ecosystem fluency high.
5. Sentence 14-15 close on the "wheel-shipped" pragma. This signals you understand the full deploy story — not just Rust code, but wheels + CI + PyPI + pip install. That's the difference between "an ambitious personal project" and "shippable production code."

Output.

Pitch element	Signal to interviewer	Score
Pattern named in sentence 1	senior framing	high
Concrete numbers	outcome focus	high
Framework named	reproducible decisions	very high
Ecosystem breadth	Rust literacy	high
Wheel-shipped pragma	production seriousness	high

Rule of thumb. Every DE claiming Rust experience should be able to deliver a 2-minute pitch that names one shipped project (with numbers), the four-gate framework, and 3-5 Rust libraries currently in production use. Below that bar, "I know some Rust" is not a hiring signal.

Worked example — the companies hiring Rust-DE and how their interviews probe

Detailed explanation. Every Rust-DE hiring pipeline in 2026 has a similar shape: a Python screen (for the daily glue work) plus a Rust deep-dive (for the systems-programming portion). Knowing which companies fit and how they probe helps you calibrate. Walk through the 2026 shortlist.

• The pattern. All the shops below ship a Rust-first product; Python is a client/scripting surface but core work is Rust.
• Interview shape. 1 phone screen (Python + system design), 1 Rust deep-dive (borrow checker, async, tokio, serde), 1 architecture round.
• Salary signal. Rust-DE roles trend 10-25% above pure Python DE at the same seniority, reflecting scarcity.

Question. For each of the 2026 Rust-DE hiring shortlist, name the flagship product, the interview shape, and one distinctive probe.

Input.

Company	Product	Distinctive probe
Datadog Vector team	Vector observability router	tokio async + backpressure
Databricks Photon	vectorised query engine (C++/Rust)	SIMD + query planner internals
InfluxData IOx	Rust time-series engine on DataFusion	Arrow columnar reasoning
Polars Inc	Polars DataFrame lib	lazy planner + streaming ops
Confluent Freight	Rust storage tier	log-structured storage + WAL
RisingWave Labs	streaming SQL DB	dataflow scheduling + state
Materialize	streaming SQL engine	timely-dataflow + incremental compute

Code.

Interview shape at Rust-DE shops (typical)

Round 1 (60 min) — Python + system design
  "Design a batch ETL for X"; probes DE fundamentals; Python OK.

Round 2 (60 min) — Rust deep-dive
  Read a 200-line Rust file; explain lifetimes; find a bug.
  Write a small crate on-the-spot; must compile + tests pass.

Round 3 (60 min) — Architecture / behavioural
  Walk through a past Rust project; why Rust vs Python; trade-offs.

Round 4 (optional) — Take-home
  Small pyo3 crate; ship a wheel; be prepared to review.

Distinctive probes by shop (rehearse for the ones you're targeting)

Datadog Vector team:
  "How does Vector handle backpressure when a sink is slow?"
  → answer: buffer configs, disk buffers, drop policies, tokio task
    coordination.

InfluxData IOx:
  "Why is Apache Arrow important for time-series?"
  → answer: columnar layout, zero-copy IPC, DataFusion integration.

Polars Inc:
  "Explain the difference between eager and lazy Polars execution."
  → answer: query plan, expression tree, optimiser passes,
    predicate pushdown, projection pruning.

Materialize:
  "What's timely dataflow?"
  → answer: differential dataflow's substrate; timestamps as
    lattice; incremental view maintenance.

Step-by-step explanation.

1. Round 1 is the DE fundamentals filter — batch design, warehouse basics, SQL. Any DE who has read this article should pass. Python is fine for the code.
2. Round 2 is the Rust-literacy filter. Reading a 200-line file and explaining lifetimes is the primary probe. This is where the 4-week skill ladder above pays off — you need to be past the ownership/borrow-checker hump.
3. Round 3 is the "how do you decide" probe. Every Rust-first shop wants to hire people who won't cargo-cult Rust for its own sake. The four-gate framework and a shipped-project story are what land the offer.
4. Round 4 (take-home) is optional but common. A small pyo3 crate proves you can go end-to-end. Deliver clean code, tests, a README, and a wheel published to TestPyPI. This is your 4-week skill ladder's project as portfolio.
5. Distinctive probes differ by shop; rehearse for the two or three you're targeting. Vector needs backpressure + tokio; Polars needs lazy planner + optimiser; Materialize needs timely dataflow. Show that you've read the relevant docs.

Output.

Shop	Rust-first?	Salary premium	Distinctive skill
Datadog Vector	yes	15-25%	tokio + async backpressure
Databricks Photon	mixed	10-15%	SIMD + query planners
InfluxData IOx	yes	15-20%	Arrow + DataFusion
Polars Inc	yes	15-20%	lazy planner + optimiser
Confluent Freight	mixed	10-15%	log-structured storage
RisingWave	yes	15-20%	dataflow + state mgmt
Materialize	yes	15-25%	timely-dataflow

Rule of thumb. Target 2-3 Rust-first shops based on your data-engineering interest area (observability → Vector, streaming SQL → Materialize/RisingWave, DataFrames → Polars). Rehearse each shop's distinctive probe; ship one pyo3 crate to TestPyPI as portfolio. That combo is the 2026 offer-getting profile.

Senior interview question on Rust-DE career signals

A senior interviewer might ask: "You're a Python-heavy DE with 3 years' experience. Should you spend Q1 next year learning Rust, and if so, how? Walk me through the decision — what career payoff you'd realistically get, which companies would actually value it, what curriculum you'd follow, and how you'd know you've reached 'shippable Rust extension' fluency."

Solution Using a 4-week ladder + 2-minute pitch + targeted-shop shortlist

Decision framework — should this DE spend Q1 learning Rust?

Q1: Is your current work I/O-bound (Airflow + Snowflake + BI only)?
    YES → Rust ROI low; skip. Invest in SQL, dimensional modelling.
    NO  → continue.

Q2: Does your team ship any Rust today (Polars, delta-rs, Vector,
     custom pyo3 crate)?
    NO  → no immediate professional payoff; hobby-project only.
    YES → strong; you'll use it within a quarter.

Q3: Are you interested in moving to a Rust-first shop in 12-18
     months?
    YES → mandatory investment.
    NO  → learn just Rung 1-4 (Python-facing extensions).

If (Q2=yes OR Q3=yes) AND Q1=no  → INVEST Q1.
Otherwise                        → SKIP; revisit in 6 months.

Q1 investment plan (13 weeks; ~10 h/week)

Week 1-4 (skill ladder)
  Week 1: Rust Book ch 1-4 (ownership).
  Week 2: Rust Book ch 5-9 (traits + errors).
  Week 3: Rust Book ch 11-14 (cargo + testing).
  Week 4: pyo3 quickstart; ship hello_ext wheel to TestPyPI.

Week 5-8 (first real crate)
  Pick a real hot-path in your current job (JSON parse, custom
  aggregation, streaming transform).
  Ship it as a pyo3 crate; benchmark against pure Python.
  Get a merge into production.

Week 9-10 (deep-dive one library)
  Read the Polars, delta-rs, or Vector source.
  Contribute one small PR (bug fix, docs, tiny feature).

Week 11-13 (interview prep for target shops)
  Rehearse the 2-min pitch + four-gate framework.
  Deep-dive on your top-2 shops (Vector? Polars? Materialize?).
  Take one paid or unpaid mock interview.

# Portfolio checklist by end of Q1
portfolio = {
    "wheel_on_testpypi":  True,   # from week 4
    "prod_pyo3_crate":    True,   # from week 5-8
    "oss_contribution":   True,   # from week 9-10; small PR is fine
    "two_min_pitch":      "rehearsed",
    "four_gate_framework": "internalised",
    "target_shops":       ["Polars Inc", "Datadog Vector"],
    "distinctive_probes": {
        "Polars Inc":     "lazy planner, expression optimiser",
        "Datadog Vector": "tokio backpressure, sink buffering",
    },
}

Step-by-step trace.

Phase	Weeks	Output	Outcome
Skill ladder	1-4	5 rungs completed; hello_ext wheel	Rust literacy floor
First real crate	5-8	production pyo3 crate	shipped-project story
OSS deep-dive	9-10	one merged PR upstream	ecosystem fluency signal
Interview prep	11-13	pitch + framework + probes rehearsed	offer-ready

After Q1, the DE has: a wheel on TestPyPI, a production pyo3 crate at their day job, one merged OSS PR, and a rehearsed 2-minute pitch anchored on a real project. That combination is enough to seriously interview at a Rust-first shop and expect a 10-25% salary premium. If Q1/Q2/Q3 all said no, the DE saves the quarter and invests elsewhere.

Output:

Metric	End of Q1
Rust literacy	can read Polars source; can write a pyo3 crate
Shipped Rust	1 production crate at day job
OSS footprint	1 merged PR upstream
Pitch readiness	2-min pitch rehearsed
Framework	four-gate + 3-of-4 rule internalised
Target-shop probes	rehearsed for top-2
Realistic salary uplift	10-25% if you move

Why this works — concept by concept:

• Decision-first curriculum — the Q1 investment is decided by three gates (I/O-bound? team ships Rust? interested in moving?). Two yeses invest; otherwise skip. Every DE gets a clear "yes/no" without generic "learn Rust anyway" pressure.
• Ladder + real crate — the 4-week skill ladder unlocks the ability to ship a real crate at your day job. That real crate becomes the interview story. Without a shipped project, you're a Rust hobbyist; with one, you're a Rust-DE candidate.
• OSS PR as a fluency signal — one merged PR to Polars, delta-rs, or Vector proves you can navigate a real Rust codebase. This is the differentiator between "read a book" and "operated in the ecosystem."
• 2-min pitch + four-gate framework — the interview-ready artifacts. Every Rust-first shop asks "why Rust here vs Python?" and the four-gate answer is what closes offers.
• Cost — 13 weeks × ~10 h = ~130 hours. The eliminated cost is the "learn Rust for 6 months with no target and no shipped code" trap that produces language literacy but not employability. Net O(13 weeks) with a concrete shippable outcome — the ladder converts time into portfolio into offers.

Design
Topic — design
Design problems on Rust-DE architecture rounds

Practice →

SQL
Topic — sql
SQL practice for Rust-DE screen rounds

Practice →

---

Cheat sheet — Rust-in-DE recipes

• Which pattern when. The 2026 default is Python-first, drop to Rust for CPU-bound hot paths via pyo3 + maturin. Pure Python + asyncio is the correct choice for I/O-bound workloads (S3, Postgres, Kafka waits). Polars / DataFusion / delta-rs / PyIceberg / Vector / uv / ruff cover most CPU-bound wins by dependency — you don't write Rust, you pip install it. Only reach for a first-party Rust extension when the profiler shows a single Python function eating > 20% of the batch and none of the existing Rust libraries covers it.
• Four-gate decision framework. Gate 1: is the workload CPU-bound (profiler shows > 50% CPU vs I/O)? Gate 2: does it need > 1 M ops/sec throughput? Gate 3: does it require memory-safety guarantees under concurrent access (untrusted input, long-running process)? Gate 4: does it need multi-core scaling without the GIL? Score three or four yeses → Rust extension; two yeses → Polars / DataFusion; one yes → NumPy vectorisation; zero yeses → pure Python + asyncio. Every workload gets a prescription in under 10 minutes.
• Five categories of Rust adoption. (1) DataFrame + query engines — Polars, DataFusion, DuckDB extensions. (2) Table format libraries — delta-rs, PyIceberg's Rust hot loops, iceberg-rust. (3) Streaming + observability — Vector, Materialize, Ballista, Databend, RisingWave. (4) Build + tooling — uv, ruff, prql compiler, some Dagster crates. (5) Orchestration + workflow — Windmill, MotherDuck backend, emerging Rust orchestrators. Memorise two headline libraries per category; every "would Rust help my X?" question maps to a category.
• pyo3 minimal template. Cargo.toml needs pyo3 = { version = "0.22", features = ["extension-module", "abi3-py38"] } plus crate-type = ["cdylib"]. src/lib.rs needs use pyo3::prelude::*;, #[pyfunction] fn my_fn(py: Python, arg: Vec<String>) -> PyResult<...> { py.allow_threads(|| { ... }) }, and #[pymodule] fn my_ext(_py: Python, m: &PyModule) -> PyResult<()> { m.add_function(wrap_pyfunction!(my_fn, m)?)?; Ok(()) }. Total ~15 lines; that is the whole extension.
• maturin build + publish recipe. pip install maturin, maturin new --bindings pyo3 my_ext to scaffold, maturin develop --release to install into the current venv, maturin build --release --strip to produce a wheel, maturin publish --repository testpypi to test-publish, maturin publish to ship to real PyPI. In CI: PyO3/maturin-action@v1 with a matrix over Linux (x86_64 + aarch64), macOS (x86_64 + arm64), Windows (x86_64) triggered by GitHub Release published. abi3-py38 in Cargo.toml cuts the matrix in half by making one wheel per platform work across all Python 3.8+ minors.
• GIL escape + parallel iteration. Wrap any CPU-bound Rust block in py.allow_threads(|| { ... }) to release the GIL; other Python threads run during that window. Inside allow_threads, use rayon::prelude::* and .par_iter() to parallelise across all CPU cores using rayon's work-stealing scheduler. This combo is the reason a Rust extension can turn a 1-core Python bottleneck into linear scaling across a 32-vCPU box. Test the parallel version with an input large enough that thread startup overhead is negligible (typically > 1000 items per call).
• Zero-copy at the FFI boundary. For > 100 MB numeric payloads, use the numpy crate: PyReadonlyArray1<f64> accepts a NumPy array as a zero-copy slice via arr.as_slice()?. For Arrow arrays (Polars, PyArrow), use the arrow crate's Python bindings. The copies to avoid: Vec<T> and HashMap<K, V> — they clone at the boundary. Every 100 MB you cross via Vec costs you 100 MB of transient allocation twice.
• When Rust wins by dependency, not by writing. Polars replaces pandas for DataFrames > 5 GB with a 5-30× speedup and 2-3× memory drop; three-line code change. delta-rs replaces Spark-mediated Delta operations with a Python-native Rust engine — no JVM. Vector replaces Fluentd / Logstash for log routing with a single Rust binary — memory drops ~8×, uptime jumps from days to months. uv replaces pip for CI installs (10 min → 15 sec); ruff replaces flake8+black+isort (100× faster lint). Every one of these swaps requires zero Rust code from your team.
• When Rust wins by writing a pyo3 extension. JSON parsing at > 20 000 msgs/sec — pure Python tops out around 5 000 msgs/sec on one core; a pyo3 + serde_json + rayon extension hits 100 000+ msgs/sec across cores. Custom aggregations at > 1 M rows/sec that Polars built-ins don't cover — pass NumPy arrays into Rust via the numpy crate zero-copy and implement the aggregation directly. Any CPU-bound tight loop the profiler flags at > 20% of the batch's CPU. Never rewrite an I/O-bound service in Rust; the GIL isn't the bottleneck when you're waiting on S3.
• Interview signals — the six phrases. "I profiled and the top 20% of CPU is in one function." "A Rust rewrite would give 5× on the hot loop; the rest stays Python." "If the pipeline is I/O-bound, stay in Python." "pyo3 + maturin; ship pre-built wheels; test with pytest." "Polars, delta-rs, DataFusion, Vector, uv, ruff — Rust cores under Python surfaces." "Python::allow_threads inside a rayon par_iter." Deploy these unprompted; every Rust-DE interviewer scores them positively.
• Companies hiring Rust-DE in 2026. Datadog (Vector observability router team), Databricks (Photon engine, mixed C++/Rust), InfluxData (IOx time-series engine, pure Rust on DataFusion), Polars Inc (Polars DataFrame lib), Confluent (Freight storage tier), RisingWave Labs (streaming SQL DB), Materialize (streaming SQL engine on timely-dataflow). Salary premium versus pure-Python DE typically 10-25% at the same seniority. Target 2-3 shops based on your DE interest area; rehearse each shop's distinctive probe (Vector = tokio backpressure, Polars = lazy planner, Materialize = timely-dataflow).
• When NOT to learn Rust. If your entire day-job is dbt + Snowflake + BI, no interviewer will probe you on Rust — invest in SQL and dimensional modelling instead. If every workload you own is I/O-bound (Airflow + S3 + Postgres waits), Rust helps you nothing. If your team is 15 people who all write Python and nobody else knows Rust, being the sole maintainer is an operational risk. If your batch runs in 20 minutes on a 4-hour SLA, there's no CPU pressure — Rust would be over-engineering. Skip Rust and revisit in 6 months when the constraints change.
• The 4-week Rust-for-DE ladder. Week 1: ownership + borrow checker (Rust Book ch 1-4). Week 2: traits + error handling (Rust Book ch 5-9). Week 3: cargo + testing + ship a small crate (Rust Book ch 11-14). Week 4: pyo3 + maturin + ship hello_ext to TestPyPI (official pyo3 book quickstart). That is the entire Rust literacy floor for a Python DE. Async + tokio are earned by later projects; you don't need them to pass a Rust-DE interview. Total time: ~40 hours over 4 weeks of evenings.
• The 2-minute Rust-DE pitch. Sentence 1-2: "My Rust story is Python-first, drop to Rust for hot paths." Sentence 3-6: one concrete shipped project with numbers ("cut the batch from 4 h to 90 min via a pyo3 JSON-parse extension"). Sentence 7-9: name the four-gate framework unprompted. Sentence 10-13: name 3-5 Rust libraries you already use in production. Sentence 14-15: close with the "wheel-shipped via maturin + Actions" pragma. Rehearse once; deploy in every interview.

Frequently asked questions

Should every data engineer learn Rust in 2026?

No — only a specific slice of DEs actually get career or productivity payoff from Rust in 2026, and the deciding question is what your workloads look like. If your day-job is dbt + Snowflake + BI + light Python glue, Rust helps you nothing; invest that time in SQL fluency, dimensional modelling, warehouse tuning, and the semantic-layer tools your BI stack uses. If your team is 15 people who all write Python and nobody else knows Rust, being the sole Rust maintainer is an operational risk and you should wait for a second Rust-literate teammate before adopting. If, on the other hand, you already ship Polars / delta-rs / Vector / a pyo3 crate, or you're interested in moving to a Rust-first shop (Datadog Vector, Polars Inc, InfluxData IOx, Materialize, RisingWave), or you regularly hit CPU-bound bottlenecks that don't have an existing Rust library, then the four-week Rust-for-DE ladder (ownership + traits + cargo + pyo3) pays back within a quarter. Learn Rust when the answer to "would I actually ship it this year?" is yes; skip it otherwise and revisit in six months.

Rust vs Python for data pipelines — which is faster and by how much?

Rust is faster than Python for CPU-bound work by roughly 5×–100× depending on the operation, and up to 32×–100× more once you release the GIL and parallelise across cores; the honest answer is that the ratio is the wrong framing, because Python is fast enough for the 80% of DE work that is I/O-bound (waiting on S3, Postgres, Kafka) where Rust helps zero. The pattern that wins in production is rust vs python not as a language choice but as a layer choice — the pipeline stays Python for glue, orchestration, notebooks, and I/O; the CPU-bound hot paths drop to Rust via a pyo3 extension or by using a Rust-cored library like Polars. Concrete measurements: Polars beats pandas by 5×–30× on group-by aggregations for datasets above 5 GB; a pyo3 JSON-parse extension beats orjson-in-a-Python-loop by ~12× on single-core and ~50-100× once you add rayon; uv beats pip by 10×–100× on package resolution. Never quote the raw ratio without naming the workload — "Rust is 50× faster than Python" is meaningless without "on CPU-bound JSON parsing with released GIL and rayon parallelism."

What is pyo3 and why do so many DE libraries use it?

pyo3 is the Rust crate that makes writing a Python-facing Rust extension a Cargo-native workflow instead of a Cython / C-API / ctypes specialty project — you annotate a Rust function with #[pyfunction], register it in a #[pymodule], and the maturin build tool packages it as a .whl that pip install handles like any other Python package. The reason so many DE libraries use it (Polars, delta-rs, PyIceberg core, Pydantic v2, cryptography, orjson, tokenizers) is that pyo3 collapses three historically separate concerns — writing a systems-language extension, packaging it for cross-platform distribution, and integrating with the Python type system — into one cohesive toolchain that a Rust-literate engineer can master in a week. The abi3 mode ships one wheel per platform that works across all Python 3.8+ versions (huge CI cost saving), Python::allow_threads lets the extension release the GIL for CPU-bound blocks (essential for parallel scaling), and the standard-type auto-conversions (String, Vec<T>, HashMap<K,V>, tuples) mean 80% of extensions never need custom FFI plumbing. Pair it with maturin and a GitHub Actions release workflow and you can go from empty repo to published wheel on PyPI in under a day.

Is Polars replacing Pandas in production?

Increasingly yes for large-DataFrame workloads and cautiously yes for new projects, though pandas retains dominance in notebooks and small-DataFrame contexts and is unlikely to be replaced wholesale — the practical 2026 answer is that Polars is the default choice above ~5 GB and pandas remains fine below that. The reasons Polars is winning above the threshold: it is Rust-cored (5×–30× faster on group-by aggregations), Arrow-native (2×–3× less memory than pandas object columns), lazy-execution capable (scan_parquet + query planner pushes filters into the reader), and multi-threaded by default (no GIL, saturates all cores). The migration cost is genuinely low — the API is intentionally pandas-shaped, so a group-by rewrite is typically three lines. The reasons pandas is still dominant: 15+ years of documentation, tutorials, StackOverflow answers, and ecosystem integrations (matplotlib, seaborn, scikit-learn all still expect pandas DataFrames); notebook workflows where the ergonomic differences don't matter at 100 MB scale; and team familiarity that outweighs 10-second-versus-2-second execution differences. The pragmatic 2026 rule is: pandas for notebooks and DFs under 1 GB, Polars for production ETL and DFs over 5 GB, and either for the ambiguous middle depending on team preference.

When should I NOT reach for Rust in a data pipeline?

Six clear cases where reaching for Rust is over-engineering that costs you weeks with no return: (1) I/O-bound workloads — if the profiler shows the process spends > 50% of wall clock waiting on S3, Postgres, Kafka, or HTTP, Rust helps zero; fix with asyncio + connection pooling instead. (2) Small pipelines — if the batch runs in 20 minutes on a 4-hour SLA, there's no CPU pressure to solve and Rust engineering time is pure waste. (3) Sole maintainer — if you'd be the only person on the team who could maintain the Rust extension and nobody else can review PRs, adopting Rust makes the codebase fragile against your departure. (4) Team velocity mismatch — if the team ships 20 pipelines per quarter and the Rust extension would only speed up one of them, the opportunity cost of the Rust learning + writing time is enormous. (5) Already-covered by a Rust dependency — if Polars, delta-rs, DataFusion, or Vector already does what you need, pip install it instead of writing your own Rust; you get 100% of the benefit for 0% of the maintenance. (6) Vendor-managed compute — if you're on Snowflake or BigQuery serverless, you can't inject a Rust component even if you wanted to; use vendor-native features (JavaScript UDFs, remote model calls, warehouse compute scaling) instead.

Which companies actively hire for Rust data engineering in 2026?

The clearest 2026 Rust-DE hiring shortlist has seven employers where Rust is a first-class hiring signal, plus another handful of adjacent shops with Rust footprints in specific teams. The core seven: Datadog hires for the Vector observability team (Vector is a pure-Rust log/metric router, hundreds of thousands of nodes deployed); Databricks hires for the Photon engine team (mixed C++ and Rust; Rust ownership is growing); InfluxData hires for the IOx team (a pure-Rust time-series engine built on DataFusion); Polars Inc — the commercial entity behind Polars — hires Rust engineers exclusively for engine work; Confluent hires for the Freight storage-tier rewrite in Rust; RisingWave Labs builds a streaming SQL database entirely in Rust and hires accordingly; Materialize builds its streaming SQL engine on top of timely-dataflow (Rust) and hires Rust engineers. Adjacent shops with Rust footprints worth targeting: Snowflake (data-processing internals), AWS (Firehose and some Kinesis internals), Cloudflare (Workers + data plane), and the wave of 2025-2026 startups building Iceberg-native engines. Salary premium versus pure-Python DE typically runs 10-25% at the same seniority, reflecting the scarcity of Rust-literate DEs. Target 2-3 shops based on your interest area, rehearse each shop's distinctive interview probe, and ship one pyo3 crate to TestPyPI as a portfolio piece — that combination is the 2026 offer-getting profile.

Practice on PipeCode

• Drill the SQL practice library → for the aggregation, join, and window-function problems that appear on Rust-DE screen rounds when the interviewer needs to check DE fundamentals before the Rust deep-dive.
• Sharpen the streaming axis with the streaming practice library → for the real-time enrichment, Kafka-consumer, and stateful-transform scenarios that map directly to Vector, Materialize, and RisingWave interview probes.
• Rehearse system design against the design practice library → for the "Python-first, Rust hot-path" architecture rounds where you must defend the four-gate framework and prescribe a Rust extension without over-engineering.
• Warm up on the aggregation practice library → for the group-by and roll-up shapes that dominate DataFrame-engine benchmarks — the same shapes Polars and DataFusion optimise for.
• Practise the joins practice library → for the fact-dimension and lookup patterns that appear in Rust query-engine deep-dives.
• Stack the prerequisites against PipeCode's 450+ data-engineering catalogue to anchor the four-gate framework, the five-category adoption map, and the pyo3 + maturin recipe against real graded inputs.

Lock in rust data engineering muscle memory

Docs explain languages. PipeCode drills explain the decision — when Rust wins on the hot loop, when Polars beats pandas without a rewrite, when the four-gate framework says "stay Python," when the pyo3 wheel is the right answer to a bottleneck. Pipecode.ai is Leetcode for Data Engineering — pattern-first practice tuned for the production trade-offs senior data engineers actually face.

Practice SQL problems →
Practice design problems →