Polars vs Pandas: the three differences that matter

Every Polars post opens with a benchmark where it beats pandas by a factor of ten. That number is real on some workload somewhere, and it is the least useful thing about the library.

Three differences actually cause the gap. Once you know them, you can predict when Polars will help you and when it will not, which is worth more than a chart.

1. It builds the plan before it does the work

Pandas is eager. Each line runs when it is read:

df = pd.read_parquet("transactions.parquet")   # whole file, all columns
df = df[df["amount"] > 0]                      # walk it, allocate a new frame
df["fee"] = df["amount"] * 0.029               # walk it again
df = df.drop_duplicates(["transaction_id"])    # and again

The file is fully loaded before the first filter, including columns you never mention again. Every step allocates.

Polars, given scan_ instead of read_, does nothing at all until you ask:

result = (
    pl.scan_parquet("transactions.parquet")
    .filter(pl.col("amount") > 0)
    .with_columns((pl.col("amount") * 0.029).alias("fee"))
    .unique(subset=["transaction_id"])
    .collect()          # only now does anything run
)

By the time collect fires, Polars knows the whole pipeline, so it can do two things pandas structurally cannot:

• Projection pushdown. Your query mentions three columns, so it reads three columns off disk. The other forty are never touched.
• Predicate pushdown. The filter goes down to the file reader, so non-matching rows are never materialised at all.

This is also where Parquet and Polars compound. Columnar storage makes reading three of forty-three columns cheap, and lazy execution is what tells the reader it only needs three.

One trap worth knowing, because it silently undoes all of it:

lazy_df.collect().head(100)   # runs the whole query, throws away almost everything
lazy_df.head(100).collect()   # pushes the limit into the plan, stops scanning early

If you learned Polars from older material you may reach for .fetch(n). That is deprecated in Polars 1.x; .head(n).collect() is the current form.

2. No hidden copies, because everything is Arrow

Polars stores data in Arrow buffers, which is the same layout DuckDB and PyArrow use. When you pass a frame between them, they exchange pointers rather than bytes.

That sounds like an implementation detail until you count the boundaries in a real pipeline. Read Parquet into pandas with the NumPy backend, transform, hand the result to DuckDB, and each crossing allocates fresh memory the size of your data, spends CPU translating layouts, and adds garbage collector pressure. On a gigabyte of decoded data that is roughly 2.3 seconds through the copying path against 0.4 through the Arrow one, and about half the peak memory.

Memory is the more reliable win, and it is the one people notice on real jobs. Pandas materialises every intermediate frame, so a chain of eight transforms over a large table holds several copies at once. That is the difference between a job that fits in your container and one that gets killed.

---

This is the first part. The full post — including the rest of the working details — is on my site: Polars vs Pandas: the three differences that matter