DEV Community

Cover image for Polars 2.0: why joins no longer keep their row order
Nikoloz Turazashvili (@axrisi)
Nikoloz Turazashvili (@axrisi) Subscriber

Posted on

Polars 2.0: why joins no longer keep their row order

#python #dataframe #opensource #data

Polars 2.0 changed a default that many data pipelines depend on without knowing it. Calling collect() on a lazy query now runs the streaming engine, and the streaming engine does not promise the row order of join, group_by and unpivot. If your code compares output row by row, or writes a file that a later step reads by position, the result can change on a machine that gives a different answer.

TL;DR

  • Polars 2.0 (released 6 October 2026) runs collect() on the streaming engine by default.
  • For join, group_by and unpivot, the release does not guarantee input row order unless you set maintain_order.
  • Out-of-core processing is on by default. It starts spilling to disk at about 80 % of RAM, with a 64 GB default disk budget.
  • In our own run on polars 2.0.0, an inner join of 2,000,000 rows returned rows out of input order by default. With maintain_order set on the join, the input order was kept.
  • The benchmark in the release post is the vendor's own run, and its footnote says the results are not comparable to official TPC results.

What changed under the hood

The release post by Ritchie Vink describes the switch in one sentence: LazyFrame.collect() now defaults to the streaming engine, which brings "massive memory and performance improvements on most queries", and the streaming engine "doesn't guarantee row-order by default" for some operations (release post).

A streaming engine splits the data into chunks, runs the chunks in parallel, and merges the results as they finish. That is the general mechanism behind unordered output: a chunk that finishes first is emitted first, whatever its position in the input. Polars does not spell this out in the release post, so read the explanation as the general pattern for streaming engines.

The release also turns on out-of-core processing by default. Operations that can spill to disk now do so when memory fills: sort, window functions and many expressions. Out-of-core join and group_by are on the roadmap, not in this release (release post).

What the upgrade guide says

The migration guide puts the change in the breaking-changes list, under "The Lazy API defaults to the streaming engine". Its row-order note reads, in full:

The exact row order shown above is not guaranteed — only that it is no longer the pre-2.0 order. Use instead, if you rely on the order: sort explicitly.

The migration guide's row-order note, with the sentence highlighted

That sentence is the whole contract. The guide does not promise any particular order after a join. It promises only that the order differs from the 1.x behaviour, and it recommends an explicit sort when the order matters (upgrade guide).

What we ran

We installed polars==2.0.0 from PyPI on Python 3.10 and ran an inner join of a 2,000,000-row table with a 1,000-key lookup table, once with the defaults and once with maintain_order set on the join:

ordered = left.lazy().join(
    right.lazy(), on="k", how="inner",
    maintain_order="left_right").collect()
Enter fullscreen mode Exit fullscreen mode
Run Input order kept
join with defaults False
join with maintain_order="left_right" True

This is one machine, one join shape and one run. It shows that the order can change. It does not show how often it changes, and it does not measure performance. The script is in our evidence folder, and it reproduces the result on any machine running polars 2.0.0.

Which errors you get earlier, and which you do not

Polars 2.0 is stricter about types, and collect_schema() resolves types without reading data. We checked two cases:

  • A missing column raised ColumnNotFoundError in collect_schema(), before any data was read.
  • A strict cast that fails on the data (pl.col("a").cast(pl.Int64, strict=True) on text values) passed collect_schema(), and raised InvalidOperationError only when collect() ran.

The release post says the same thing: some errors depend on the data and cannot be caught while the query plan is built (release post). So collect_schema() is a fast first check that catches only part of the errors.

How to read the benchmark

The release post says Polars leads DataFusion and DuckDB on TPC-H and TPC-DS, with DuckDB 1.5.6, DuckDB 2.0 alpha and DataFusion 54.0.0 as the rivals. The method is described in the post: five runs per query, the best run kept, a separate process per query, a 60-second timeout, and the file cache cleared once per engine and benchmark, so queries within one benchmark share a warm cache. Queries that DataFusion timed out on or ran out of memory on were excluded for every engine.

Three things to keep in mind:

  • The benchmark is derived from TPC-H and TPC-DS, and the post's own footnote says the results "are not comparable" to official TPC results.
  • The post says Polars with 192 threads has a constant overhead that hurts small queries, and that Polars limited to 32 cores is competitive or winning on every benchmark.
  • Moving from 16 to 192 vCPUs at SF100 makes Polars 3.8x faster on TPC-H, against 3.2x for DuckDB 1.5.6, by the post's own numbers.

The speed claims may hold for your workload. Check them on your own data.

What to do before you upgrade

  1. Find every join, group_by and unpivot that feeds a test, a file, or a later step that reads rows by position.
  2. Either add an explicit sort after the operation, or set maintain_order on it. The release post recommends the explicit sort. Our run used maintain_order="left_right" on the join and kept the input order.
  3. Run your existing tests on polars 2.0.0 with the default settings, and compare the output of each step with the output you saved on 1.x.
  4. Check memory limits. Out-of-core spilling starts at about 80 % of RAM, and the default disk budget is 64 GB.
  5. Read the whole breaking-changes list in the upgrade guide. It covers more than this one change. The same page lists, for example, pl.read_csv now dispatching to pl.scan_csv(...).collect(), and a new default for show_graph's plan_stage.

Verdict: NEEDS REVIEW

The change is useful and it is not free. The default makes most queries faster and more memory-safe. It also silently changes the row order of three common operations, and the guide only says the order is "no longer the pre-2.0 order". I would review every join before upgrading.

FAQ

Does Polars 2.0 change the results of my queries?
Not the values, as far as our run showed. It changes the row order of join, group_by and unpivot unless you set maintain_order or sort explicitly.

Which flag keeps the old order?
maintain_order is set on the operation itself. The release post names it for join, group_by and unpivot; we used maintain_order="left_right" on a join.

Is Polars 2.0 faster than DuckDB?
The vendor's benchmark says so on TPC-H and TPC-DS derived data. The post's footnote says the results are not comparable to official benchmarks, and the run was done by Polars. Measure your own queries.

What does collect_schema() catch?
Errors in the query's structure, such as a missing column, are raised before any data is read. Errors that depend on the data, such as a cast that fails on one value, are only raised by collect().

Sources

This article expands on an episode of **The Daily Diff, a five-minute daily video on what shipped and what broke in tech.
Watch the episode · Subscribe on YouTube · the written diff lands in your inbox every morning at thedailydiff.dev.

Video transcript and primary sources

Top comments (1)

Collapse
 
arhancanli profile image
Arhan Canli •

One run showing False can't separate two different claims: "the order differs from 1.x" and "the order differs from run to run". The guide's wording ("not the pre-2.0 order") only promises the first, but the second is what breaks tests. Cheap way to find out on your 2M x 1k join: collect it 20 times, hash the key column of each result, and count distinct hashes; then repeat with POLARS_MAX_THREADS=1 and with the full core count. If one thread gives a single stable order and many threads give several, a snapshot test will pass on a 2-core CI runner and fail on a 16-core laptop, which is a worse failure than a one-time change.

On the fix: sorting after every join costs a full sort in production just to satisfy tests. For the tests themselves polars.testing.assert_frame_equal(..., check_row_order=False) compares without depending on order, and maintain_order is then only needed where a later step reads rows by position (a written file, an iter_rows loop with a running state). It would also be worth timing the maintain_order="left_right" run next to the default, since the guide's reason for dropping the order is the cost of keeping it.