DEV Community

Cover image for Opening a Parquet File: pandas, Polars, DuckDB, Spark and a No-Code Option
Moksh Gupta
Moksh Gupta

Posted on Originally published at devtoollab.com

Opening a Parquet File: pandas, Polars, DuckDB, Spark and a No-Code Option

#programming #python #datascience #tutorial

Someone sends you orders_2025.parquet, you open it in Notepad++, and the screen fills with junk characters. Nothing is broken. Parquet is compressed, column-oriented binary, so a text editor was never going to show it. The quick answers: in Python, pip install pyarrow and then pd.read_parquet(path); in a terminal, duckdb -c "SELECT * FROM 'orders_2025.parquet' LIMIT 3"; with no code at all, a Parquet viewer that runs in the browser.

I timed every reader below against the same file in a longer guide on DevToolLab. This is the condensed tour.

Reading and opening a Parquet file

Why Readers Start at the End of the File

The Apache Parquet spec puts the 4-byte magic string PAR1 at both the start and the end of a file. The schema and the offsets of each row group live in a footer just ahead of the closing PAR1. A reader therefore jumps to the tail, parses the footer, and only then fetches the column chunks the caller asked for. Two practical effects follow: pulling two columns out of seven is cheap, and a file that stopped downloading halfway cannot be read at all, because the footer is exactly the part that never arrived.

The sample file in every example holds 1,200,000 US orders in 7 columns: order_id, order_date, customer_name, city, state, zip and amount_usd. DuckDB 1.5.6 wrote it with Snappy compression into 10 row groups of up to 122,880 rows. It takes 15.6 MB on disk; the identical data as CSV takes 67.8 MB. The timings come from an Apple M3 Pro with 18 GB of RAM, measured on October 11, 2026.

No Code: Decode It in the Browser

The Parquet Viewer parses the footer and decodes column chunks inside the tab, so nothing gets uploaded. On the sample file, Chrome showed the schema, all 10 row groups and the first 50 rows after 4,057 ms. Search runs across all 1.2 million rows, and you can export either the full file or only the search hits as CSV or JSON. It handles Snappy, Gzip, ZSTD and LZ4 pages. A file written with Brotli or LZO needs to be re-saved with another codec first.

The Parquet Viewer showing orders_2025.parquet with 1,200,000 rows, 7 columns, 10 row groups and Snappy compression, plus the first rows and each column's type

Choosing a Reader for Scripts

Library How to get it Call Time on the sample file
pandas 3.0.6 pip install pandas pyarrow pd.read_parquet(path) whole file 0.02 to 0.09 s, two columns 0.005 s
pyarrow 26.0.0 pip install pyarrow pq.read_table(path) Texas-only filter 0.006 to 0.009 s
Polars 2.0.0 pip install polars pl.scan_parquet(path) December query 0.004 s once warm
DuckDB 1.5.6 single binary SELECT * FROM 'path' per-state totals 0.01 to 0.02 s, startup included
PySpark 4.2.0 pip install pyspark and a JDK spark.read.parquet(path) 5.5 s, mostly the JVM booting

For the Python rows, only the read call was timed, and the range is the best and worst of five runs. The DuckDB and Spark rows include launching the process.

pandas

pandas ships no Parquet decoder. It passes the file to pyarrow, or to fastparquet when pyarrow is not installed, which is why pyarrow belongs in the same pip install.

import pandas as pd

df = pd.read_parquet("orders_2025.parquet")
print(df.shape)
print(df[df.city == "Boston"].head(2))
Enter fullscreen mode Exit fullscreen mode 
(1200000, 7)
   order_id  order_date   customer_name    city state    zip  amount_usd
1    100001  2025-05-01  Brandon Taylor  Boston    MA  02108      381.90
5    100005  2025-03-12    Rachel Lopez  Boston    MA  02108        5.67
Enter fullscreen mode Exit fullscreen mode

Look at the ZIP column. Parquet kept it as a string, so Boston is still 02108. Reading the CSV copy with pd.read_csv inferred integers and printed 2108. The order_date column comes back with dtype object, holding Python date objects; add dtype_backend="pyarrow" if you want Arrow's date32[day] instead.

Loading everything took 0.02 to 0.09 seconds and used 139 MB of memory. The CSV version of the same read took 0.43 to 0.48 seconds. Passing a column list skips decoding the rest, and this one finished in 0.005 seconds:

df = pd.read_parquet("orders_2025.parquet", columns=["state", "amount_usd"])
print(df.groupby("state")["amount_usd"].sum().nlargest(3).round(2))
Enter fullscreen mode Exit fullscreen mode 
state
TX    60458222.43
CA    30188386.67
OR    15314491.16
Name: amount_usd, dtype: float64
Enter fullscreen mode Exit fullscreen mode

pyarrow for Metadata and Filtered Reads

Going straight to pyarrow lets you inspect the file before reading a single row, and filter during the scan:

import pyarrow.parquet as pq

f = pq.ParquetFile("orders_2025.parquet")
print(f.metadata)

tx = pq.read_table("orders_2025.parquet", columns=["order_id", "amount_usd"], filters=[("state", "=", "TX")])
print(tx.num_rows)
Enter fullscreen mode Exit fullscreen mode 
<pyarrow._parquet.FileMetaData object at 0x1035c3ab0>
  created_by: DuckDB version v1.5.6 (build 069cc9f9b5)
  num_columns: 7
  num_rows: 1200000
  num_row_groups: 10
  format_version: 1.0
  serialized_size: 6804
239088
Enter fullscreen mode Exit fullscreen mode

All of that metadata came from a 6,804-byte footer, with no row data touched. The filters argument discards non-matching rows while scanning, so only the 239,088 Texas orders ever land in memory.

Polars, Lazily

pl.scan_parquet returns a query plan instead of a DataFrame, and no bytes are read until collect():

import polars as pl

top = (
    pl.scan_parquet("orders_2025.parquet")
    .filter(pl.col("order_date") >= pl.date(2025, 12, 1))
    .group_by("state")
    .agg(pl.len().alias("orders"), pl.col("amount_usd").sum().round(2).alias("revenue_usd"))
    .sort("revenue_usd", descending=True)
    .head(3)
    .collect()
)
print(top)
Enter fullscreen mode Exit fullscreen mode

The top three for December came back as TX (20,545 orders, $5,212,198.10), CA (10,193, $2,588,785.11) and CO (5,211, $1,319,827.67). The first call took 0.11 seconds and later ones 0.004 seconds. Because the plan only mentions three columns, Polars never decodes the other four.

DuckDB From the Terminal

DuckDB 1.5.6 came out on September 28, 2026, and its CLI is one binary that accepts a Parquet path wherever a table name would go:

duckdb -c "SELECT * FROM 'orders_2025.parquet' LIMIT 3"
Enter fullscreen mode Exit fullscreen mode 
┌──────────┬────────────┬────────────────┬──────────┬─────────┬─────────┬────────────┐
│ order_id │ order_date │ customer_name  │   city   │  state  │   zip   │ amount_usd │
│  int64   │    date    │    varchar     │ varchar  │ varchar │ varchar │   double   │
├──────────┼────────────┼────────────────┼──────────┼─────────┼─────────┼────────────┤
│   100000 │ 2025-06-11 │ Emily Smith    │ New York │ NY      │ 10001   │     368.78 │
│   100001 │ 2025-05-01 │ Brandon Taylor │ Boston   │ MA      │ 02108   │      381.9 │
│   100002 │ 2025-06-27 │ Michael Davis  │ Miami    │ FL      │ 33130   │     254.59 │
└──────────┴────────────┴────────────────┴──────────┴─────────┴─────────┴────────────┘
Enter fullscreen mode Exit fullscreen mode

The second header line carries the column types, so this single command tells you both what is in the file and how it is typed. A GROUP BY state across all 1.2 million rows returned the same Texas total pandas found, 239,088 orders and $60,458,222.43, in 0.01 to 0.02 seconds with startup counted. The DuckDB Parquet docs call this projection pushdown: columns the query never names are never read.

Getting It Into Excel

Parquet is not on Excel's list of documented data sources. Microsoft's Parquet connector page covers Power BI and Power Query Online, so Power BI Desktop can open the file as-is while Excel wants a CSV. Exporting just the Texas rows, which fit comfortably under Excel's 1,048,576-row ceiling, took DuckDB 0.06 seconds:

duckdb -c "COPY (SELECT * FROM 'orders_2025.parquet' WHERE state = 'TX') TO 'orders_tx.csv' (HEADER)"
Enter fullscreen mode Exit fullscreen mode

If you would rather skip the terminal, export the CSV from the Parquet Viewer and check it in the CSV Viewer before handing it to Excel.

Spark and Databricks

spark.read.parquet accepts a single file or a directory full of part files:

from pyspark.sql import SparkSession, functions as F

spark = SparkSession.builder.appName("read-parquet").getOrCreate()
df = spark.read.parquet("orders_2025.parquet")
(df.where(F.col("state") == "TX")
   .groupBy("city")
   .agg(F.count("*").alias("orders"), F.sum("amount_usd").cast("decimal(12,2)").alias("revenue_usd"))
   .orderBy(F.desc("revenue_usd"))
   .show())
Enter fullscreen mode Exit fullscreen mode 
+-----------+------+-----------+
|       city|orders|revenue_usd|
+-----------+------+-----------+
|     Austin| 59982|15141806.86|
|San Antonio| 59812|15126653.56|
|    Houston| 59667|15104170.73|
|     Dallas| 59627|15085591.28|
+-----------+------+-----------+
Enter fullscreen mode Exit fullscreen mode

Locally, PySpark 4.2.0 on Java 24 needed 5.5 seconds, and almost all of that was the JVM starting. Databricks notebooks already provide a spark session, so delete the builder line there. The Databricks Parquet guide, last updated August 26, 2026, reads from Unity Catalog volume paths like /Volumes/<catalog>/<schema>/<volume>/ and renders results with display(df). It describes Parquet as the most common format for data in Databricks, which makes sense once you remember Delta Lake tables are Parquet files with a transaction log on top.

VS Code and R

The Data Wrangler extension for VS Code supports .parquet: right-click the file in the Explorer and pick Open in Data Wrangler. In R, the arrow package's read_parquet() gives you a data frame, and col_select limits the columns much like pandas' columns=. The R path was not tested for the original post.

The Errors You Will Actually See

  • ImportError: Unable to find a usable engine; tried using: 'pyarrow', 'fastparquet'. is pandas 3.0.6 with no engine installed. pip install pyarrow fixes it.
  • pyarrow.lib.ArrowInvalid: ... Parquet magic bytes not found in footer. means the last 4 bytes are not PAR1. Either the file was never Parquet (a CSV renamed to .parquet triggered exactly this) or the copy is incomplete. tail -c 4 orders_2025.parquet should print PAR1.
  • A truncated file fails in every reader, each with its own wording: DuckDB says No magic bytes found at end of file, Polars says The file must end with PAR1, and Spark throws [FAILED_READ_FILE.CANNOT_READ_FILE_FOOTER]. Re-download instead of attempting a repair, since the missing piece is the footer itself.
  • [PATH_NOT_FOUND] Path does not exist: file:/... is Spark finding nothing at the path. The message shows the full path it resolved, so compare that against where the file really is, or switch to an absolute or volume path.

The full guide has the complete text of each message, including the byte values Spark found where PAR1 should have been.

Habits Worth Avoiding

Converting to CSV only to peek at a Parquet file costs you twice: the sample grew more than fourfold, and the column types disappeared, which is how 02108 turned into 2108. Reading a multi-gigabyte file whole into pandas is the other trap; name the columns, use scan_parquet, or let DuckDB query it in place. And before you drop a file full of customer records into any online viewer, confirm it decodes locally rather than uploading to a server.

Wrap-Up

Let the next step decide the tool. For a look with nothing installed, a browser viewer showed the schema and the first rows of the 1.2-million-row file in about 4 seconds. Inside a Python script, pd.read_parquet with a columns list is the short path, reading the full file in 0.02 to 0.09 seconds against 0.43 to 0.48 for the CSV. Files too large to load go to Polars' scan_parquet or DuckDB. Spark earns its startup cost only when the data already lives on a cluster, and for Excel, export a CSV first.

References

Top comments (0)