Why I stopped using pandas for everything

Aug 2026

Polars, DuckDB, and the rise of query engines that make pandas feel like a relic. A practical migration guide with real benchmarks on a 10M row dataset.

For years, pandas was my default. It was the Swiss Army knife of data work in Python—flexible, familiar, and good enough. Then the datasets grew. Notebooks started freezing. Memory errors became routine. A “simple” group-by on 10 million rows turned into a coffee break. I spent more time waiting than thinking.

In 2025–2026 I finally stopped treating pandas as the answer for everything. I moved the heavy lifting to Polars and DuckDB. The difference is not incremental—it’s architectural. Here’s what changed, the numbers, and a practical path to migrate without rewriting your entire life.

The pandas problem at scale

Pandas is single-threaded by default, eager (it materializes every intermediate step), and memory-hungry. It shines for exploration under a few hundred thousand or low-millions of rows, especially when you lean on its rich ecosystem. Beyond that, you hit the wall:

• High memory overhead (often 5–10× the raw data size).
• Sequential execution that leaves most of your cores idle.
• No real query optimizer—every chain of operations pays the full cost.

Modern alternatives were designed differently. Polars is a Rust-based DataFrame library with lazy evaluation, multi-threading, and Apache Arrow under the hood. DuckDB is an embedded analytical (OLAP) database that speaks excellent SQL, streams Parquet/CSV directly, and can spill to disk. Both treat your data more like a query engine than a bag of Python objects.

Benchmarks on a 10M-row dataset

I ran a consistent set of operations on a synthetic 10-million-row sales dataset (mixed numeric, categorical, and datetime columns, stored as Parquet, ~1–2 GB). Hardware: mid-range laptop / desktop with 16 cores. Times are approximate averages from repeated runs (cold-ish cache). Numbers align with multiple independent 2025–2026 comparisons.

Typical results (seconds):

Operation	Pandas	Polars (lazy)	DuckDB
Read Parquet	8.0	1.2	0.9
Filter + Select	4.5	0.4	0.3
GroupBy + Aggregations	12.0	0.8	0.7
Left join to 200k dim table	25.0	3.5	2.8
Multi-step pipeline	45.0	4.5	3.8

Peak memory was roughly 4.8 GB for pandas, 1.2 GB for Polars, and ~0.8 GB for DuckDB.

On this size, Polars and DuckDB are routinely 5–15× faster (sometimes more on joins and complex pipelines) while using a fraction of the RAM. At 100M+ rows the gap widens further; pandas often OOMs while the others continue (Polars via streaming, DuckDB via out-of-core execution).

Caveats apply—results vary with schema, selectivity, hardware, and exact versions—but the direction is consistent across independent tests.

Practical migration guide

You do not need a big-bang rewrite. Migrate the hot paths first.

1. Start with Polars if you love DataFrames

The API is intentionally close to pandas, so muscle memory transfers quickly. Key differences:

• No Index (everything is explicit columns—this is a feature).
• Prefer expressions and the lazy API for real speed.
• .apply() is strongly discouraged; use vectorized expressions or map_batches only when necessary.

Common patterns side-by-side:

# Reading
# pandas
df = pd.read_parquet("data.parquet")
# Polars (eager)
df = pl.read_parquet("data.parquet")
# Polars (lazy — preferred for bigger data)
lf = pl.scan_parquet("data.parquet")

# Filter
# pandas
df[df["amount"] > 100]
# Polars
df.filter(pl.col("amount") > 100)

# New column
# pandas
df["total"] = df["price"] * df["qty"]
# Polars
df = df.with_columns((pl.col("price") * pl.col("qty")).alias("total"))

# GroupBy
# pandas
df.groupby("region").agg(revenue=("amount", "sum"), count=("id", "count"))
# Polars
df.group_by("region").agg(
    pl.col("amount").sum().alias("revenue"),
    pl.len().alias("count")
)

# Lazy pipeline example (this is where the magic happens)
result = (
    pl.scan_parquet("sales/*.parquet")
    .filter(pl.col("status") == "completed")
    .group_by(["region", "sku"])
    .agg(pl.col("amount").sum().alias("revenue"))
    .collect(streaming=True)   # for data larger than RAM
)

Polars also interops cleanly with Arrow, so you can hand results back to pandas or scikit-learn when needed with near-zero copy.

2. Reach for DuckDB when SQL is clearer or data lives on disk

DuckDB shines when you want to query Parquet/CSV/JSON files directly, run complex joins/window functions, or keep an SQL-first mental model. It works great as a “SQL layer” on top of pandas or Polars DataFrames too.

import duckdb

# Query files directly — no full load
result = duckdb.sql("""
    SELECT region, sku,
           SUM(amount) AS revenue,
           COUNT(*) AS orders
    FROM read_parquet('sales/*.parquet')
    WHERE status = 'completed'
    GROUP BY 1, 2
    ORDER BY revenue DESC
""").df()   # or .pl() for Polars DataFrame

# Or operate on an existing DataFrame
duckdb.sql("SELECT * FROM df WHERE amount > 100").df()

DuckDB’s optimizer and vectorized execution make multi-table analytics feel almost free compared with the equivalent pandas code.

3. Hybrid workflows (the real 2026 stack)

Most of my pipelines now look like this:

1. Ingest / heavy transforms → Polars (lazy) or DuckDB.
2. Final small result → convert to pandas only if a downstream library still requires it.
3. Exploration notebooks → start in Polars or DuckDB; drop to pandas only for the last visualization step if needed.

Arrow makes the hand-offs cheap.

When to keep using pandas

• Tiny data (< ~100k–500k rows) and rapid interactive exploration.
• Heavy reliance on the broader ecosystem (certain statsmodels, older plotting helpers, etc.).
• Legacy code that is stable, well-tested, and not performance-critical.

For everything else—especially production ETL, feature pipelines, or any recurring job on multi-million-row data—Polars or DuckDB will usually feel like a relief.

Closing thoughts

Pandas taught a generation how to think about tabular data in Python. That achievement remains. But the constraints that made sense in 2015 no longer match the hardware or the data volumes of 2026. Polars gives you a faster, safer DataFrame experience. DuckDB gives you a powerful embedded query engine that treats files as tables.

You don’t have to abandon pandas entirely. Just stop using it for everything. Migrate the painful parts first, measure, and enjoy getting your evenings back.

The tools have moved on. Your workflows can too.