Parquet, Explained: How One File Format Quietly Won the Data World

Snowflake queries it. Spark writes it. Trino scans it, Flink lands it, BigQuery federates it, and DuckDB treats a directory of it as a database. Your pandas read_parquet pulls gigabytes per second through it. Cloud object storage is full of it. Recent versions of Excel can even open it.

It's not a database. It's not a query engine. It's not a product anyone sells, and no company owns it. It's a file format — Apache Parquet — and it's the closest thing modern data engineering has to a universal standard. The three big "table format" layers — Delta Lake, Apache Iceberg, and Hudi — all store their data as Parquet files underneath. The entire lakehouse movement is, physically, an enormous pile of Parquet plus opinions.

How does a file format nobody owns become the center of gravity for an entire industry? And why do practitioners who learn its inner workings keep discovering that their storage bills were three times bigger than they needed to be?

This article is the full story: how Parquet works down to structures you can point at in a hex dump, why it became ubiquitous, where it genuinely hurts, and one physical-design trick — sort order — that routinely makes Parquet tables an order of magnitude smaller. That last one sounds like a stunt. By the time we get there, it'll just be arithmetic.

Contents

• It's 2013, and analytics has a reading problem
• Store it sideways
• Inside a Parquet file
• The life of a query
• How the compression sausage gets made
• Why it won
• The honest cons
• When to use it (and when not to)
• The trick that sounds fake: sort by cardinality, low to high
• The fine print on sort order
• More lifehacks in the same spirit
• Closing thought

It's 2013, and analytics has a reading problem

It's 2013. Storage has never been cheaper — Hadoop clusters built from commodity disks can hold everything — but reading it hasn't caught up. An analytical query like "average revenue by country" needs three columns out of forty, and yet the formats of the day (CSV, JSON, Avro, SequenceFiles) all store data row by row. To touch the three columns you want, you pay to read all forty.

Meanwhile, the queries are getting bigger. Twitter's analytics team is scanning billions of rows across a cluster and watching half the cluster's throughput carry bytes nobody asked for: usernames and device strings and JSON blobs, streamed off disk only to be immediately discarded.

The row-versus-column idea wasn't new — data warehouses and C-store-style research systems had exploited it for years, and Google's 2010 Dremel paper had just demonstrated columnar storage working at web scale, including a scheme ("repetition and definition levels") for storing nested data — structs, arrays, maps — column by column without shredding it into mush. What was missing was an open, general-purpose file format that brought all of it to the Hadoop ecosystem.

So in 2013, engineers from Twitter and Cloudera built one. The lore says the name comes from parquetry — the wood-floor pattern where slats lie side by side — which is either a beautiful coincidence or exactly the right metaphor, because that's precisely how it stores your columns. Parquet joined the Apache Software Foundation, and the rest is a very quiet, very complete victory.

One framing note before we go deeper: Parquet is a file format, and nothing more. Not a database, not an engine, not a query planner, not a table. It's a meticulously specified way to lay bytes down on disk so that any engine — Spark, Trino, Snowflake, DuckDB, pandas — can read them back efficiently. That humility turns out to be a big part of the answer to "why did it win," and we'll return to it.

Store it sideways

Everything about Parquet starts from one decision: store data by column instead of by row.

Take this tiny events table — our running example for the article:

name	os	country	amount
Alice	iOS	DE	42
Bob	iOS	US	17
Cara	Android	DE	8
Dan	iOS	DE	23

A row-oriented format (CSV, JSON, the heap files of your favorite OLTP database) stores it exactly as printed: one row after another, all columns of a row adjacent. That's great when you want Dan's entire row — it's one contiguous read. It's terrible when you want the average of amount, because you have to wade past every name, os, and country byte to fish out the values you care about.

Analytical queries almost always touch some columns of lots of rows — SELECT country, AVG(amount) ... GROUP BY country — so flipping the layout pays twice:

• You read less. Want two columns out of forty? Read two column chunks. The other thirty-eight never leave disk.
• You compress better. Values of the same type and same domain now sit next to each other, and "things that look alike compress alike" is the oldest trick in the information-theory book. A column of country codes is a sea of repetition; the same codes scattered between names and timestamps are noise.

Column-orientation is the idea. The next two sections are the engineering.

Inside a Parquet file

Parquet files are built from four nested structures, and their names come up constantly in tuning guides, so they're worth knowing cold:

• Row group — a horizontal slice of the table (all columns, a batch of rows). This is the unit of parallelism and of I/O: one worker per row group, and — this matters for later — the entire row group is typically processed by one task. Most writers default to something around 128 MB or a million rows per group.
• Column chunk — within a row group, each column's values are stored contiguously. This is the unit of column pruning: skip a column, skip its whole chunk.
• Page — column chunks are sliced into pages of roughly 1 MB. This is the unit of encoding and compression, and the smallest thing a reader decodes.
• Footer — the metadata mother lode: the full schema, the offsets of every row group and column chunk, which encodings and compression each chunk uses, and per-chunk statistics (min, max, null count, distinct count).

And here's the layout of the file itself, byte by byte:

The file starts and ends with the magic bytes PAR1, so you can identify one from a mile away. Everything important is at the end: to open a Parquet file, a reader grabs the last 8 bytes (footer length + magic), then range-requests exactly the metadata it needs, then range-requests exactly the column chunks it needs. On object storage like S3, where a ranged GET is cheap and a full scan is not, this layout is native-level friendly.

Two more structural superpowers worth flagging:

• Splittability. Row-group boundaries are recorded in the footer, so a big file can be split across many workers with zero coordination — no "unzip the whole 40 GB gzipped CSV on one node" era nostalgia.
• Self-description. The schema, the types, the encodings, the statistics — it's all inside the file. Hand a Parquet file to a tool that has never seen your data stack, and it knows what it's looking at. (This is also what the table formats build on — more on that in a minute.)

The life of a query

Here's the payoff diagram — the life of an analytical query against a Parquet-based lake, using our events table:

Every one of those steps is the format doing work your query engine would otherwise have to do expensively, with data it would otherwise have to load. Engines pair this with vectorized execution — decoding columns straight into dense, SIMD-friendly batches — which is why "just point DuckDB at a directory of Parquet files" is a legitimate analytics strategy in 2026.

Notice how much of this diagram runs on those footer statistics. Remember that; it's about to become load-bearing, twice.

How the compression sausage gets made

Parquet's size reductions come from two layers working in series — specialized encodings first, general-purpose compression second:

• Dictionary encoding: each page's distinct values go into a small dictionary, and the column becomes a stream of tiny integer indices into it. Your 2-byte country strings become 1-byte numbers before any "compression" has even happened.
• Run-length encoding (RLE): consecutive repeats collapse into (value, count) pairs. This is the encoding that cares about sort order, and therefore the encoding doing the heavy lifting in the trick at the end of this article.
• Bit-packing: when values have no runs, they're packed at minimum bit width (3 distinct OSes ≈ 2 bits each).
• Delta encodings (for integers, dates, sorted keys): store differences between consecutive values instead of the values. Sorted IDs have small gaps; shuffled IDs have giant ones.
• Then zstd / snappy / gzip squeezes whatever is left, finding repeated byte patterns across everything above.

The important mental model: these are pattern-finders. Dictionary encoding exploits few distinct values. RLE exploits adjacent equal values. Delta exploits monotonic sequences. zstd exploits repeated byte patterns. A column only gets compressed to the extent its physical arrangement exhibits one of those patterns — and sort order is the one lever you control that changes which patterns exist, without changing the data's meaning at all.

Hold onto that sentence. It's the key to the last third of this article.

Why it won

You'll notice this article hasn't claimed Parquet is objectively the best at any single thing. It doesn't have ORC's pedigree of Hive-native optimization; a bespoke binary format could be smaller or faster for any specific workload. Its dominance comes from a more interesting place: it's excellent at everything analytical workloads need, open, and — critically — everywhere.

The properties that got it there:

1. It's open and neutral. Apache-licensed, vendor-owned by no one. In a market terrified of lock-in, "your bytes are yours" is a strategy.
2. It decouples storage from compute. Because it reads beautifully over cheap object storage with ranged GETs, your data can live on S3 while Snowflake, Trino, and DuckDB all take turns querying it. The lakehouse architecture — the dominant pattern of the last decade — is basically unthinkable without a format like this.
3. Network effects. Every engine reads it because every other engine reads it. New tools ship Parquet support on day one to be relevant. That flywheel, not technical superiority, is the moat — but it's a very real moat. (Just ask ORC, which is excellent and still lives mostly in Hive-land.)
4. It's the substrate of the table formats. Delta Lake, Apache Iceberg, and Hudi — the three big "SQL table" layers over lakes — all store their data as Parquet files, adding ACID transactions, schema evolution, and time travel on top. The "new standard" is literally built out of the old one.
5. It's cheap to keep. Three-to-ten-times smaller than the equivalent CSV/JSON is the typical range, before you've applied a single trick from this article. Storage is a recurring cost; good layout is a one-time effort.
6. It's fast to query for the reasons in the query-path diagram: column pruning, row-group pruning, predicate pushdown, splittable parallelism.

A quick aside for the Arrow-curious: Apache Arrow is Parquet's in-memory cousin — same columnar religion, designed for zero-copy in-process work rather than on-disk storage. The two are designed to round-trip each other cheaply, and the pairing is why "read Parquet into Arrow, compute vectorized, write Parquet back" is the default circulation of the modern data stack. (If you've never watched a pandas read_parquet hit multiple GB/s via Arrow, it's worth trying.)

The honest cons

A dominant format with no downsides would be suspicious. Here's where Parquet genuinely hurts:

• Point lookups and OLTP are not its job. Fetching one full row means seeking into every column chunk to reassemble it. Row formats exist precisely because transactional workloads want rows. Use Postgres for your orders table; use Parquet for your analytics over the orders table.
• Writes are expensive and files are immutable. A writer must buffer an entire row group, encode it, compress it, and write it with its footer. There's no "update row 4,077" — Parquet files are write-once. Updates and deletes are why Delta/Iceberg/Hudi exist: they rewrite files for you and keep a transaction log straight.
• The small-files problem. Thousands of tiny Parquet files (the natural byproduct of frequent streaming writes) wreck performance: metadata overhead per file, footer round-trips per file, task scheduling chaos, S3 request charges. This is such a consistent foot-gun that "compaction" is a core feature of every table format.
• Not human-readable. You can't cat it, can't eyeball it in a text editor, can't email it to a business analyst who opens it in Notepad. (The world is softening — recent Excel can import Parquet, and a quick duckdb -c "select * from file.parquet limit 5" gets you a peek in seconds, not minutes — but CSV's universal readability remains unmatched.)
• Overkill for small data. A 5,000-row lookup table shipped with your app? A config file? Just use CSV or JSON. Parquet's machinery pays off at scale and is pure ceremony below it.
• Randomness doesn't compress. GUID columns, uniformly random floats, already-compressed blobs (images, encrypted data): columnar layout and dictionaries have nothing to say about entropy. Don't expect magic, and don't bother compressing binary blobs a second time.
• Deeply nested, repeated data can be slow to read back. Repetition/definition levels are elegant for storage, but reconstructing a forest of nested structs row by row costs CPU. Very nested Parquet read into very row-oriented code is where the format's reputation for "slowness" actually comes from.
• Schema evolution is limited. Adding nullable columns at the end: fine. Renaming, reordering, or changing types: prepare for pain (or a table format's versioning machinery).

None of these are scandalous. They're the predictable trade-offs of a format optimized for scanning billions of rows, touching a few columns, from cheap shared storage. It's a sprinter complaining about its swimming.

When to use it (and when not to)

Format	Reach for it when
Parquet	Analytical data at any scale beyond "fits in a spreadsheet"; data lakes and lakehouses; interchange between engines; long-term storage of query-able data
CSV	Human-facing handoffs, tiny datasets, maximum tool compatibility
JSON (lines)	APIs, semi-structured streaming events, documents with wildly varying shape
Avro	Row-oriented streaming pipelines with schema-registry needs (Kafka ecosystems)
ORC	Deep Hive-land; otherwise Parquet's ecosystem gravity usually wins
A real database	OLTP, concurrent transactions, point updates, enforceable constraints

The trick that sounds fake: sort by cardinality, low to high

Here's a stunt. Take a wide events table — the kind with a few low-cardinality dimension columns (device_operating_system: 3 values; country: ~250 values; maybe app_version, event_name, locale...) and some high-cardinality ones (user_id: millions). Write it to Parquet in the order the events arrived. Note the file size — call it 30 GB.

Now rewrite the exact same rows, sorted by (device_operating_system, country, user_id). Same schema, same row count, same everything — SQL results are byte-for-byte identical. The file lands under 3 GB.

Nothing was deleted. Nothing was lossy-compressed. The data just lies down differently. Reductions of this shape — 10x and sometimes much more — are real and reproducible, and this section will get you to the point where you can predict which of your tables has this hiding inside them. It comes down to two facts you already learned:

1. Parquet compresses a column by finding patterns among adjacent values.
2. Sort order is the one lever that changes which patterns exist.

Step 1: Meet your columns as Parquet sees them

With dictionary encoding, each low-cardinality column becomes a stream of tiny integers — the dictionary indices. What does that stream look like?

Unsorted data (rows arrive in event order, i.e., chaos):

os column, as dictionary indices:      1 0 1 2 0 0 1 2 1 0 0 2 1 0 1 1 2 0 ...
country column, as dictionary indices: 83 17 42 0 91 17 3 88 42 17 91 0 55 ...

Equal values are almost never adjacent. RLE finds no runs, so each column falls back to bit-packing: roughly 2 bits per row for os, roughly 8 bits per row for country — forever, no matter what. On a billion-row table, that's ~250 MB for the OS column and ~1 GB for the country column, before zstd shrugs at the noise. And user_id, with millions of distinct values? Its dictionary overflows and the writer quietly falls back to storing raw 8-byte IDs — random, unordered, essentially incompressible.

Sorted by (os, country, user_id):

os column:      0 0 0 0 0 0 ... 0 | 1 1 1 1 1 ... 1 | 2 2 2 ... 2
country column: 0 0 0 ... 0 | 1 1 1 ... 1 | ... (sorted runs inside every os block)
user_id column: locally sorted, ascending, inside every (os, country) block

Now every encoding in the pipeline has something to eat:

• os collapses to 3 runs. Three (value, count) tuples. The column that cost 250 MB now costs bytes.
• country becomes at most ~750 long runs (250 values inside each of 3 OS blocks). The 1 GB column now costs kilobytes.
• user_id is locally sorted, so delta encoding turns it into a stream of small gaps instead of random 64-bit noise.

And then zstd, arriving last, finds every page monotonous instead of chaotic, and gets dramatically better matches on everything — including the payload columns you didn't sort by, because rows that share (os, country) tend to correlate on app version, event type, locale, and friends. That compounding across all columns at once is where order-of-magnitude reductions come from.

Step 2: Why lowest cardinality first, specifically?

This is the part people memorize without deriving, so here's the derivation. It's short.

When data is sorted by keys (k1, k2, ..., kn), a column can only collapse into long runs if the columns before it in the sort have a small combined number of distinct combinations. Formally: column j ends up with at most

runs(column j)  ≤  C(k1) x C(k2) x ... x C(kj)

distinct runs, where C() is cardinality, capped by the row count. The total storage for the sort-key columns is roughly the sum of prefix products, and the last term is C1 x C2 x ... x Cn — which is identical no matter how you permute the keys. So you minimize the sum by minimizing the earlier terms, which means putting the smallest cardinalities first. Ascending order is provably the best of the simple orderings.

Concretely, with our three columns (cardinalities 3, 250, ~300M) on a billion rows:

Sort order	os runs	country runs	user_id state
(os, country, user_id)	3	≤ 750	locally sorted → delta-friendly
(country, os, user_id)	≤ 750	250	locally sorted → delta-friendly
(os, user_id, country)	3	~1 billion (catastrophe)	locally sorted
Unsorted	~1 billion	~1 billion	random noise

Two things jump out:

1. The truly catastrophic ordering is the third one — interleaving a high-cardinality column between two low-cardinality ones. Once user_id comes before country, every (os, user_id) pair is nearly unique, so country degenerates back to one value per row and pays bits-per-row again. Keep your low-cardinality dimensions adjacent, at the front — that's the sharpest cliff in the whole landscape.
2. The difference between the first two rows is real but modest: swapping (os, country) for (country, os) mostly moves a few hundred runs between two columns that are both nearly free either way. The big wins are (a) sorting at all, and (b) keeping the low-cardinality cluster together at the front. Ascending order is the safe default that also wins the math, so use it — just don't expect a further 10x from re-swapping two adjacent low-card columns.

Honest calibration: the 30x-and-beyond stunts happen on wide, dimension-heavy tables — a dozen scattered low-cardinality string dimensions all collapsing at once. Typical gains from adding a thoughtful sort are more like 2x–10x on storage, which is still, conservatively, free money. And the trick does nothing for pure entropy: a table of GUIDs and random floats has no patterns to expose, no matter how you sort it.

Step 3: There's a second, hidden prize — data skipping

Remember the footer statistics? Sorted data doesn't just compress better; it prunes better. If os is the first sort key and the table has 100 row groups, each row group contains essentially one OS value, so its min = max = 'iOS' (or 'Android', or...). A WHERE os = 'iOS' filter can now skip two-thirds of the file without reading a byte of it. DuckDB's Parquet tips call this out explicitly: sort by your frequently-filtered columns, and row-group min/max stats become a homemade index. Sort keys are the gift that keeps giving — once per compression, again per query.

Step 4: Do it

The canonical pattern — partition by time, sort by ascending cardinality within each partition — in DuckDB:

COPY (
    SELECT *
    FROM 'raw_events/'
    ORDER BY device_operating_system, country, user_id
) TO 'events/'
(FORMAT parquet, PARTITION_BY (event_date), COMPRESSION zstd, ROW_GROUP_SIZE 1_000_000);

And in Spark, one crucial refinement — you don't need a global sort. Compression happens inside each file's row groups, so sorting within each partition is sufficient, and sortWithinPartitions gets you that without the cluster-wide shuffle of a full orderBy:

(df.sortWithinPartitions("device_operating_system", "country", "user_id")
   .write
   .partitionBy("event_date")
   .option("compression", "zstd")
   .save("s3://lake/events/"))

Neither operation changes the data. Every downstream SELECT returns identical results. You've simply packed the suitcase better — same clothes, half the suitcase.

The fine print on sort order

One honest caveat before you ascend-cardinality-sort every table you own: the compression-optimal sort is not always the query-optimal sort.

If your hot queries are WHERE event_date BETWEEN ... time-range scans, putting event_date early in the sort (or partitioning by it — say, a directory per day) gives dramatically better pruning than burying it last. The canonical resolution of this tension is the pattern from Step 4: partition by the range-filtered column, sort by ascending cardinality within partitions. Partitioning handles the time dimension; the sort handles compression and dimension filtering. If you can't partition, a pragmatic hybrid is (your_most_filtered_column, lowest-card column, ..., highest-card column) — sacrifice a little compression for a lot of skipping.

Rules of thumb, in descending priority:

1. Sort by something. Unsorted is the expensive state.
2. Partition (or lead the sort with) the column your queries range-filter on most — usually time.
3. Keep low-cardinality dimensions adjacent, at the front of the sort.
4. Within that, ascending cardinality.
5. Never interleave a high-cardinality column between low-cardinality ones.

More lifehacks in the same spirit

The sort trick has siblings. All of them are "arrange the bytes so the format's machinery has something to eat."

1. Switch from Snappy to Zstd. Snappy became the default when CPU was scarcer than disk; in 2026, Zstd at a moderate level typically lands 20–30% smaller at comparable read speeds. One word in your writer options. It stacks multiplicatively with the sort trick.

2. Kill your small files (and small row groups). Every Parquet file pays footer metadata, and every row group truncates runs — a thousand 5 MB files hold a thousand tiny, badly-compressed fragments of what should be long runs. Target files in the 128 MB–1 GB range with row groups big enough for runs to actually form (DuckDB's default row-group size is 122,880 rows; their guidance is at least as many row groups per file as you have threads reading it, which biases toward somewhat larger files). If streaming ingestion gives you a swarm of small files nightly, run compaction — it's the sort trick's best friend, because compaction + re-sort is exactly the "rewrite the table neatly" operation.

3. Use Parquet v2 data pages — but check your data shape first. The format's newer encodings — DELTA_BINARY_PACKED for integers and timestamps, BYTE_STREAM_SPLIT for floats — live in v2 data pages, and on the right data they're dramatic: DuckDB's own best-case numbers show up to 99% size reduction on a cleanly monotonic integer sequence. That's the best case, though, not the typical one. DuckDB's own issue tracker documents the opposite result on medium-entropy data — values that repeat but aren't monotonically increasing — where DELTA_BINARY_PACKED can make a file ~3x larger than v1, because it turns repeated values into effectively random-looking deltas that compress worse than the raw values would have. That's one of two real reasons DuckDB doesn't default to writing v2 yet — the other being that some reader engines still can't parse it.

The rule of thumb: v2 is close to a free win on sorted IDs, timestamps, and counters — test before flipping it on for anything else. In DuckDB it's COPY ... TO 'f.parquet' (FORMAT parquet, PARQUET_VERSION v2).

4. Store real types. Dates as DATE instead of '2026-08-27' strings, 3-value flags as dictionary-encoded booleans or tiny ints instead of 'YES'/'NO' strings, IDs as integers instead of zero-padded strings. Types are compression decisions: the string version of every value fights your dictionary, your bit-packing, and your min/max statistics all at once. (Strings-as-types also quietly breaks statistics pruning — min over '10' and '9' is meaningless.)

5. Turn on the skipping machinery for high-cardinality lookups. Min/max statistics can't help with WHERE user_id = 42 on a shuffled billion-row table (42 is plausibly in every row group's range). Two features fix exactly this: bloom filters (a compact "definitely not in this chunk" probabilistic summary — writers like Spark and Iceberg can emit them per column) and the page index (page-level min/max stats, enabling skips within chunks). Enable both for your favorite equality-filtered high-cardinality columns.

6. Diagnose before you tune. Don't guess where the bytes are — look. DuckDB will show you every column chunk's encodings, compression, and sizes straight from the file:

SELECT path_in_schema, encodings, compression,
       total_uncompressed_size, total_compressed_size
FROM parquet_metadata('events/partition_date=2026-08-01.parquet')
ORDER BY total_compressed_size DESC;

If the user_id column dwarfs everything, think delta encodings and sort position. If a low-card column is huge, it wasn't sorted when written. If encodings says PLAIN on a column that should be dictionary-encoded, its dictionary overflowed — another sign sorting (or better typing) is needed:

Thirty seconds of this beats an afternoon of folklore.

Closing thought

Parquet won not by being magical but by being well-mannered: open, self-describing, columnar, splittable, compressible, and readable by everyone. It stores your data sideways, tells readers what it knows, and gets out of the way.

And once you see that its entire compression story is "find patterns among adjacent values," a whole shelf of lifehacks stops being folklore and becomes arithmetic. Sort low-to-high cardinality. Keep the little dimensions together. Partition by what you filter, sort by what you group. Use Zstd and v2 pages and real types. Don't drown in small files. None of these change your data — they change how it lies down on disk, and Parquet is ferociously opinionated about lying down.

Somewhere in your data lake is a table that's three times bigger than it needs to be, wearing a trench coat. You now know exactly how to take the coat off.

---

Further reading

• The Apache Parquet format specification — readable, surprisingly short, and the encodings section will click now that you know why they exist.
• Google's Dremel paper (2010) — where columnar-at-scale and nested-data-via-levels were proven out.
• DuckDB's Parquet tips — the practical writer-side guidance quoted in the lifehacks section. Their Parquet encodings post and bloom filters post are excellent companions.
• Your favorite table format's docs (Iceberg, Delta, Hudi) — note how each one is, at bottom, a very clever pile of Parquet files plus a log.