Polars hits the sweet spot between Pandas’ ease and Spark’s scale. If you’ve ever waited on a groupby or cursed a memory error, this cheat sheet is for you. I’ve pulled the patterns that save time in real pipelines, not just toy examples. Bookmark this before your next ETL run.
Setup and Basics
First, get Polars and a dataset. The lazy API is the default now, so you’ll rarely need to call .lazy() explicitly. Start with a CSV or Parquet file, or create a DataFrame from scratch.
- pip install polars pyarrow
- import polars as pl
- df = pl.read_csv('data.csv') # or pl.read_parquet()
- df = pl.DataFrame({'a': [1, 2], 'b': ['x', 'y']})
Selecting and Filtering
Polars uses expressions, not strings. This feels odd at first but pays off when you chain operations. The syntax is consistent: every column is an expression you can transform, filter, or aggregate.
- df.select(['a', 'b']) # columns by name
- df.select(pl.col('a').alias('renamed'))
- df.filter(pl.col('a') > 10)
- df.filter(pl.col('b').is_in(['x', 'z']))
- df.filter(pl.col('a').is_null())
Transforming Data
Polars expressions are composable. You can nest them, reuse them, and even store them in variables. This is where the library shines over Pandas.
- df.with_columns(pl.col('a').cast(pl.Float64))
- df.with_columns(pl.col('a').fill_null(0))
- df.with_columns((pl.col('a') * 2).alias('a_doubled'))
- df.with_columns(pl.col('b').str.to_uppercase())
- df.with_columns(pl.col('a').is_between(10, 20))
Grouping and Aggregations
Groupbys in Polars are lazy by default. This means you can stack multiple aggregations without materializing intermediate results. The syntax is clean, but watch out for the order of operations.
- df.group_by('b').agg(pl.col('a').sum())
- df.group_by('b').agg([pl.col('a').mean(), pl.col('a').max()])
- df.group_by('b').agg(pl.col('a').quantile(0.9))
- df.group_by_dynamic('timestamp', every='1d').agg(pl.col('a').sum())
Joins and Concatenation
Joins in Polars are explicit. You’ll specify the join type and the columns to join on. Concatenation is straightforward, but remember that Polars is strict about schema matching.
- df.join(other, on='key', how='inner')
- df.join(other, on='key', how='left')
- df.hstack([other]) # column-wise
- df.vstack([other]) # row-wise
- pl.concat([df1, df2], how='diagonal') # schema-safe union
Performance Tips
Polars is fast, but you can make it faster. The lazy API is your friend. Use it for any pipeline longer than a few operations. Also, avoid Python loops. Polars expressions are vectorized, so let the engine do the work.
- Use .lazy() and .collect() for pipelines with multiple steps.
- Prefer Parquet over CSV for I/O. It’s faster and smaller.
- Use .with_columns() instead of multiple .select() calls.
- Avoid .apply() unless absolutely necessary. Use expressions first.
- Set pl.Config.set_fmt_str_lengths(100) to see full strings in debug output.
Debugging and Inspection
Polars has great tools for debugging. The .explain() method shows the query plan, which is invaluable for optimizing lazy pipelines. For quick checks, use .head() or .sample().
- df.head(5) # first 5 rows
- df.sample(5) # random 5 rows
- df.describe() # summary stats
- df.schema # column names and types
- lazy_df.explain() # query plan for lazy DataFrames
Polars won’t replace Pandas for every task, but it’s the right tool for most data pipelines. The syntax takes a day to learn and a week to master. Once it clicks, you’ll write faster, cleaner code. Keep this cheat sheet handy until the patterns stick.
This post was originally published on my site. Read the full article and more →
Top comments (0)