Polars 2.0 Will Silently Reorder Your Rows: What to Check Before You Upgrade

Last week I would have told you the safest kind of major release is one with no new features. Then Polars published the 2.0 release candidate with exactly that pitch: no big features, "we hope it to be a boring experience for you," in the words of founder Ritchie Vink's announcement post. I installed it that evening expecting a quiet afternoon of renamed methods.

Two hours later I had a list of five changes in my own scripts that would have passed every test and still corrupted results in production. No exceptions. No stack traces. Just different numbers, quietly.

If you run Polars in production, this is the rare upgrade where the dangerous changes are not the ones that break your build. It is the ones that do not.

The headline change: streaming is now the default

When you call collect() on a LazyFrame in 2.0, the query now runs on the streaming engine. Previously you had to opt in. The payoff is real: the announcement claims the streaming engine is "easily 5x faster" in aggregate, with large memory improvements, because it processes data in batches instead of loading everything at once.

The cost is subtler. The streaming engine does not guarantee row order for operations that do not semantically require it: joins, group_by, unpivot. The old in-memory engine happened to preserve your input order in these cases. The new one feels free to shuffle, because the guarantee was never part of the contract.

Here is the trap, and I hit it myself on the release candidate. I ran the exact example from the migration guide on a tiny frame:

import polars as pl

lf = pl.LazyFrame({"k": [0, 1, 2], "l": ["a", "b", "c"]})
other = pl.LazyFrame({"k": [2, 1, 0], "r": ["x", "y", "z"]})

lf.join(other, on="k", how="left").collect()

On three rows, the output came back in perfect left-frame order: 0, 1, 2. Looks safe, right? The order survived. That is precisely the problem. On small test data, the streaming engine often happens to preserve order. On the ten-million-row table you join in production, with multiple threads processing batches concurrently, it will not. The Polars docs say it straight: the change "may silently impact the results of your pipelines."

If your code does a join and then relies on the rows coming out in left-frame order, say, to align two lists positionally, that assumption now dies at scale, not in CI.

The fix is one argument. When order matters, say so explicitly:

lf.join(other, on="k", how="left", maintain_order="left").collect()

Or, if you genuinely just need a deterministic order for output, sort explicitly:

lf.unpivot(pl.selectors.numeric(), index="a").collect().sort(pl.all())

And if you are not ready for streaming at all, you can pin the old behavior globally while you migrate:

pl.Config.set_engine_affinity("in-memory")   # process-wide

or per query with lf.collect(engine="in-memory"), or via the POLARS_ENGINE_AFFINITY=in-memory environment variable. I would treat that as a bridge, not a destination. The 5x figure only exists on the streaming path.

The quiet row-count change: explode

This is the one I would never have caught from the changelog alone. In 1.x, exploding a list column that contained an empty list produced one null row for it. In 2.0, an empty list explodes into zero rows.

I verified it on the RC:

df = pl.DataFrame({"a": [[1, 2, 3], [], [4, 5, 6]]})
df.explode("a")
# 1.x: [1, 2, 3, null, 4, 5, 6]  -> 7 rows
# 2.0: [1, 2, 3, 4, 5, 6]        -> 6 rows

Your row counts just changed. Any reconciliation check that compares row counts before and after a transformation will start failing, which is annoying. Worse, any aggregation that counted those null placeholders, say, a per-day event count where an empty tag list used to contribute a row, now reports different totals, which is not annoying at all, it is wrong. If you need the old behavior, pass empty_as_null=True explicitly.

Note the asymmetry while you are at it: a null list still explodes into one null row. Only empty lists changed.

Horizontal concat no longer pads

In 1.x, pl.concat([df1, df2], how="horizontal") silently padded the shorter frame with null values when heights differed. That "convenience" is exactly how misaligned data sneaks into pipelines: two frames you assumed matched, glued together with a column of nulls papering over the mismatch.

In 2.0 it raises:

ShapeError: cannot concat dataframes with different heights in 'strict' mode

I confirmed this on the RC. The old behavior still exists but moved behind an honest name: how="horizontal_extend". If concat errors start appearing after your upgrade, that is Polars telling you two frames you thought were aligned are not. Do not reflexively switch to horizontal_extend. Go find out why the heights differ.

The type system got stricter, and one change alters values

Two changes here, one loud and one genuinely sneaky.

Loud: is_in refuses lossy comparisons. In 1.x, checking whether an integer column is_in a float list silently coerced both sides to float, so 1 in [1.99] could evaluate as true at the right precision. In 2.0 this raises an InvalidOperationError telling you to cast explicitly. Good riddance, and the error message even explains the history. The fix is a one-line .cast(pl.Int64) on whichever side is wrong.

Sneaky: integer + unsigned math changed its output type. Adding a signed integer column to a UInt64 column used to produce a Float64 result. It now produces Int128, which is exact instead of lossy, so this is an improvement, but it changes two things at once with no error: the output dtype, and potentially the computed values, since floats cannot represent every large integer exactly. Downstream code that expects Float64 or serializes to systems without 128-bit integers will notice. No exception will tell you where.

Renames that now fail fast

The deprecations everyone has been ignoring since 1.0 finally hard-fail, and Polars added two purpose-built exception types that tell you the replacement inline. I ran them on the RC:

• melt is gone. The error reads: `melt` was removed in version 2.0; use `LazyFrame.unpivot` instead, with `index` instead of `id_vars` and `on` instead of `value_vars`.
• with_row_count is gone, replaced by with_row_index, and the default column name changed from row_nr to index.
• The join_nulls argument on join is now nulls_equal.

These raise AttributeRemovedError or ArgumentRemovedError with the fix embedded in the message, which is a genuinely nice touch for anyone migrating with an AI coding agent in the loop. One caveat from the migration guide's footnotes: the coverage is not complete. DataFrame.group_by(...).count() and DataFrame.rolling(...).count() raise a plain AttributeError with no hint pointing you to len().

read_csv is now lazy under the hood

pl.read_csv is now dispatched through pl.scan_csv(...).collect(), which mostly means good things: it gains multi-file support and the lazy reader's parameters. But two behavioral details bit people in the RC threads, so check them:

• A schema_overrides list must now cover every column in the file. Partial overrides, where you specified just the one problematic column, now raise a SchemaError. Write out the full list.
• columns=[2, 1, 3] now returns columns in the order you requested. It used to return them sorted. I verified: the same call now returns ['c', 'b', 'd'] instead of ['b', 'c', 'd']. If you unpack the result positionally, check it.

Also gone from read_csv: n_threads, batch_size, sample_size, and rechunk. The same lazy-ification applies to read_ipc, which lost memory_map and rechunk.

Two more silent ones to grep for

pl.datetime stopped naming its output column "datetime". It now takes the name of its leftmost argument. pl.datetime("year", "month", "day", "hour") produces a column literally named year now. I confirmed this on the RC. If any downstream code does df["datetime"], it breaks, and in a with_columns call the renamed output can silently overwrite an existing year column. The fix is .alias("datetime"). The same change applies to pl.repeat.

Combining selectors with pl.col via &, |, ^ changed meaning. pl.selectors.integer() & pl.col("mask") used to select just the mask column. It now performs an element-wise bitwise operation across every selected column. With two integer columns, that means you silently get three columns of bitwise-ANDed values back instead of one filtered column. The docs flag this one explicitly as a silent-result change. Use pl.selectors.by_name("mask") instead.

The 15-minute pre-upgrade checklist

Run your suite against the RC (pip install polars==2.0.0rc1, the stable 2.0 lands in the following weeks) and work through this list in order:

• Grep for melt, with_row_count, join_nulls. These fail loudly with the fix in the error message. Mechanical fixes.
• Grep for read_csv / read_ipc calls with partial schema_overrides, memory_map, rechunk, n_threads, batch_size. Now loud errors, fix per the guide.
• Grep for pl.datetime( and pl.repeat( and check whether anything downstream expects the old output column name. Add .alias() where it does.
• Grep for selector expressions combined with pl.col using &, |, ^. Replace with pl.selectors.by_name(...).
• Now the hard part: every join, group_by, and unpivot on a LazyFrame. Ask one question per call site: does anything after this depend on row order? If yes, add maintain_order="left" on joins, maintain_order=True on group-by operations, or an explicit .sort().
• Every explode call site: do your row-count checks or aggregations assume one row per input, including empty lists? Pass empty_as_null=True if so.
• Every horizontal pl.concat: mismatches now raise. Resist switching to horizontal_extend until you understand why the heights differ.
• Any code adding signed and UInt64 columns together: the result is Int128 now. Check downstream dtype expectations and serialization targets.
• If you need a week to migrate safely: pl.Config.set_engine_affinity("in-memory") restores the old engine globally. Fix order-dependence first anyway, because the streaming engine is where the performance win lives.

One more habit worth adopting from this release, regardless of version: the Polars team now explicitly recommends collect_schema() for validating query structure without materializing data. It resolves types up front and catches schema-level mismatches before your pipeline has run for twenty minutes, and it is cheap enough to call in tests. That advice applies to 1.x too.

Why this release matters more than its changelog

The 2.0 announcement makes a point of saying the release is deliberately boring: no headline features, just removing old design decisions and changing defaults. What actually changed is the contract. In 1.x, Polars gave you well-behaved row order on operations where the SQL-style semantics did not require it, and silent null padding where frames mismatched. Both were convenient. Both were also the kind of implicit behavior that hides bugs, and the maintainers have decided that with the streaming engine's concurrency in the picture, those assumptions are no longer affordable.

I migrated two of my own scripts the evening I installed the RC. The loud changes took ten minutes. The row-order audit took the other two hours, because it is not a syntax problem, it is a "what did this code actually assume" problem, and only you know that. That is the real work of this upgrade, and no pip install will do it for you.

I write about data engineering, backend systems, and practical AI tooling every week. Subscribe, it's free.

Have you run your pipelines against the 2.0 RC yet? Did the streaming engine's row-order change bite you, or did you get lucky on small data like I did first? Tell me in the comments.