Read and Write Files in Polars: CSV, Parquet, JSON

Reading and writing files is the bread and butter of any pipeline. Polars is fast at it, and for big files it can stream so you never load everything into memory. Here is the I/O you will actually use.

Read files

Each format has an eager reader that returns a DataFrame right away.

• df = pl.read_csv("data.csv")
• df = pl.read_parquet("data.parquet")
• df = pl.read_json("data.json")
• df = pl.read_ndjson("data.ndjson")
• df = pl.read_excel("data.xlsx")

Write files

The write_* methods mirror the readers. Prefer Parquet over CSV when you can: it is smaller, faster, and keeps types. This answers saving a Polars DataFrame to CSV.

• df.write_csv("out.csv")
• df.write_parquet("out.parquet")
• df.write_json("out.json")
• df.write_ndjson("out.ndjson")

Large files that do not fit in memory

For a large DataFrame, scan instead of read. scan_* returns a LazyFrame that only reads what your query needs, and collect(streaming=True) processes it in chunks.

• lf = pl.scan_parquet("huge.parquet")
• result = lf.filter(pl.col("amount") > 100).select(["id", "amount"]).collect(streaming=True)
• # scan many files at once with a glob:
• lf = pl.scan_csv("data/*.csv")

Before you write, you will usually filter rows or rename columns. The full command set is in the Python Polars cheat sheet.

---

This post was originally published on my site. Read the full article and more →