Parquet File Format Internals: Row Groups, Encodings, Predicate Pushdown & Bloom Filters

Parquet File Format Internals: Row Groups, Encodings, Predicate Pushdown & Bloom Filters

The parquet file format is the on-disk shape that the entire modern data lake is built on — the format your Spark job scans, your Snowflake external table reads, your DuckDB query hits, and your Iceberg / Delta table stores its data files in — and yet most engineers treat it as an opaque .parquet blob that "is columnar and compresses well." That black-box mental model is exactly what fails you the moment an interviewer asks "walk me through what happens when a query reads one column out of a 10 GB Parquet file," or "why does adding a WHERE clause on a sorted column make the scan 50× cheaper," or "when does a bloom filter help and a min/max statistic doesn't." The speed of columnar analytics is not magic; it is a very specific set of layout decisions — where bytes physically sit, what metadata is written alongside them, and which of those bytes a reader is allowed to skip without ever decompressing them.

This guide is the internals walkthrough that turns that black box into a mechanism you can reason about. It works down the format layer by layer: the columnar-versus-row read model that explains why the whole thing exists, the exact physical anatomy — magic bytes, row groups, column chunks, data and dictionary pages, and the thrift-encoded footer that holds the FileMetaData — then the encoding stack (dictionary, run-length, bit-packing, delta, byte-stream-split) with general-purpose compression layered on top, and finally the read-time skipping machinery that is the real payoff: projection pushdown, min/max column statistics, the page index, dictionary filtering, row-group skipping, and bloom filters. Every section pairs a teaching block with real, runnable pyarrow.parquet code, a worked interview answer, a step-by-step trace, an output table, and a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the optimization practice library →, rehearse on the ETL practice library →, and sharpen the storage layer with the file-io practice library →.

---

On this page

• Why the Parquet layout makes columnar analytics fast
• Physical layout — row groups, column chunks, pages, and the footer
• Encodings and compression — dictionary, RLE, bit-packing, delta, then Snappy/Zstd
• Predicate and projection pushdown — statistics, page index, dictionary filtering
• Bloom filters, tuning, and interview signals
• Cheat sheet — Parquet internals recipes
• Frequently asked questions
• Practice on PipeCode

---

1. Why the Parquet layout makes columnar analytics fast

Columnar storage is a read-cost optimization — the layout exists so a scan can touch bytes for only the columns and rows it needs

The one-sentence invariant: the parquet file format is a columnar, self-describing, immutable file layout whose entire design goal is to let a reader answer an analytical query by reading the smallest possible set of bytes — only the columns the query projects, only the row groups and pages whose statistics can't be proven irrelevant, and only after a single metadata read tells it exactly where those bytes live. Every property people cite about Parquet — "it compresses well," "it's fast for analytics," "it supports predicate pushdown" — is a downstream consequence of one decision: store values of the same column next to each other on disk, and write rich metadata describing every chunk so the reader can skip most of them.

Row-oriented vs columnar — the read model that explains everything.

• Row-oriented storage (CSV, JSON Lines, Avro, a classic OLTP heap) stores each record contiguously: all fields of row 1, then all fields of row 2. Great when you fetch or write whole rows one at a time — the OLTP access pattern.
• Columnar storage (Parquet, ORC, Arrow) stores each column contiguously: every value of user_id, then every value of country, then every value of amount. Great when you scan a few columns over millions of rows — the OLAP access pattern.
• The analytical query shape. A warehouse query like SELECT country, sum(amount) FROM events WHERE amount > 100 GROUP BY country touches 2 columns out of maybe 40. Row storage forces you to read all 40 columns of every row to get at 2; columnar storage reads exactly 2 column streams and skips the other 38 entirely. That is projection pushdown, and it is free in a columnar layout.

Why same-column-together is a compression multiplier, not just a skip trick.

• Type homogeneity. A column chunk holds values of one physical type — all int64, all BYTE_ARRAY. Encoders can assume the type and specialize: bit-pack small integers, delta-encode a monotonic timestamp, dictionary-encode a low-cardinality string.
• Value locality. Adjacent values in a column are far more similar than adjacent fields in a row (a country column is a handful of repeating strings; a row mixes an id, a name, a price, a timestamp). Similar-neighbor data encodes and compresses dramatically better.
• Vectorized execution. Because a column arrives as a contiguous typed buffer, engines process it in tight SIMD loops — sum a million int64s without per-row branching or deserialization. Columnar storage feeds vectorized execution the layout it wants.

The self-describing, write-once contract.

• Self-describing. A Parquet file carries its own schema and all chunk metadata in a footer. No external schema file, no header row ambiguity — a reader opens the footer and knows every column's type, every chunk's byte offset, and every chunk's statistics.
• Immutable / write-once. Parquet files are not updated in place. Compaction, Delta, and Iceberg all treat a Parquet file as an immutable unit and manage change at the file level. This is what lets the format bake statistics into the footer at write time and trust them forever at read time.
• The read path in one line. Open file → read the footer → decide which row groups and columns to read → seek directly to those byte ranges → decode. The footer is read once; everything else is targeted I/O.

What interviewers listen for.

• Do you say "columnar means values of one column are stored contiguously" rather than "it's a fast format"? — required answer.
• Do you connect columnar layout to projection pushdown, better compression, and vectorized execution as three consequences of one decision? — senior signal.
• Do you know the reader reads the footer first (from the end of the file), not the file top-to-bottom? — senior signal.
• Do you distinguish encoding (structure-aware, per column) from compression (byte-level, on top)? — senior signal.
• Do you frame Parquet as immutable / write-once, which is why the statistics can be trusted? — senior signal.

Detailed explanation — the three consequences of "same column together"

Detailed explanation. Everything in Parquet flows from the columnar decision. It is worth internalizing the three consequences as a single chain, because interviewers reward the candidate who derives them rather than lists them.

• Consequence 1 — projection is I/O-cheap. Because each column lives in its own contiguous region (a column chunk per row group), reading 2 of 40 columns reads roughly 2/40 of the data bytes. In row storage that same query is a full-file scan.
• Consequence 2 — encoding and compression get better. Homogeneous, locally-similar values encode into far fewer bytes (dictionary for repeats, delta for sequences, RLE for runs), then a general compressor squeezes the encoded bytes further.
• Consequence 3 — skipping becomes possible. Because data is chunked into row groups and pages, and each chunk carries min/max/null statistics, the reader can prove a chunk is irrelevant to a predicate and never read it. This is predicate pushdown, and it only works because the layout is chunked and annotated.

Worked example — the read-cost difference between row and columnar

Detailed explanation. Make the "columnar is cheaper" claim concrete with a byte count. Consider an events table with 40 columns and 100 million rows, average 20 bytes per column value (800 bytes per row → 80 GB raw). A dashboard query reads 2 columns and filters on one of them.

• Row-oriented cost. To read 2 columns you must read every row's full 800 bytes, because the 2 fields are scattered across each 800-byte record. Read cost ≈ 80 GB (minus whatever compression, which is weaker on mixed-type rows).
• Columnar cost. The 2 columns are 2 contiguous streams of ≈ 2 GB each (2 columns × 100M × ~20 bytes, before encoding). Read cost ≈ 4 GB raw, and encoding/compression typically cut that several-fold.

Question. Estimate the bytes a reader must fetch for SELECT country, sum(amount) FROM events WHERE country = 'US' under row storage versus Parquet, and explain which Parquet features drive the difference.

Input.

Fact	Value
Rows	100,000,000
Columns	40
Avg bytes/value	20
Columns touched	2 (country, amount)
Predicate	country = 'US' (≈ 20% of rows)

Code.

# A back-of-envelope model of read cost, row vs columnar
rows          = 100_000_000
cols          = 40
bytes_per_val = 20
touched       = 2          # country, amount
selectivity   = 0.20       # fraction of rows where country = 'US'

raw_row_bytes = rows * cols * bytes_per_val            # whole records
row_read      = raw_row_bytes                          # must read every field of every row

col_bytes     = rows * touched * bytes_per_val         # only 2 columns
encoding_gain = 4                                       # dictionary+compression on low-card country, zstd on amount
rowgroup_skip = selectivity                             # min/max lets us skip US-free row groups

col_read      = (col_bytes / encoding_gain) * rowgroup_skip

print(f"row-oriented read : {row_read/1e9:6.1f} GB")
print(f"columnar read     : {col_read/1e9:6.3f} GB")
print(f"speedup           : {row_read/col_read:6.0f}x")

Step-by-step explanation.

1. Row storage has no choice: the 2 needed fields sit inside each 800-byte record, so reading them means reading all 80 GB. There is no projection to exploit because columns are interleaved.
2. Columnar projection cuts the 40 columns to 2 — a 20× reduction before anything else. This is the single biggest lever and it is purely structural.
3. Encoding + compression on those 2 columns (dictionary on the low-cardinality country, Zstd on amount) typically buys another ~4×; the exact factor depends on cardinality and codec.
4. Predicate pushdown on country lets the reader skip whole row groups whose min/max statistics prove they contain no 'US' rows. If data is clustered by country, most row groups are skipped; if it's random, few are — sorting matters (Section 4).
5. The three effects multiply: 20× (projection) × 4× (encoding) × up-to-5× (skipping) is why a columnar scan of a 80 GB table can touch a fraction of a GB.

Output.

Path	Bytes read (model)	Driver
Row-oriented	80.0 GB	must read every field of every row
Columnar (projection only)	4.0 GB	2 of 40 columns
Columnar (+ encoding)	1.0 GB	dictionary + Zstd
Columnar (+ skipping)	0.20 GB	row-group min/max on country

Rule of thumb. Read cost in Parquet is (columns_touched / total_columns) × (1 / encoding_gain) × (fraction_of_row_groups_not_skipped). Optimize a slow scan by attacking whichever factor is largest — usually projection first, then clustering/sorting for skipping.

Worked example — the footer-first read path

Detailed explanation. A Parquet reader never streams the file top to bottom. It reads the end first, because that is where the map lives. Understanding this read path is the single most clarifying thing about the format.

• Step 1 — read the last 8 bytes. The final 4 bytes are the magic string PAR1; the 4 bytes before that are a little-endian int32 giving the footer (FileMetaData) length in bytes.
• Step 2 — seek back and read the footer. Knowing the length, the reader seeks to filesize - 8 - footer_len and reads the thrift-encoded FileMetaData: schema, num_rows, and per-row-group / per-column-chunk metadata including byte offsets and statistics.
• Step 3 — plan. From that metadata the reader decides which columns to read (projection) and which row groups to skip (predicate). It now knows the exact byte range of every column chunk it wants.
• Step 4 — targeted reads. It issues seeks/range-reads for exactly those column chunks and decodes them. On object storage this is a handful of ranged GETs, not a full download.

Question. Describe the byte-level steps a reader performs to open a Parquet file on S3 and read one column from one row group, and identify how many round trips it takes.

Input.

Region	Contents
Bytes 0–3	magic PAR1 (header)
Middle	row groups → column chunks → pages
Footer	thrift FileMetaData
Last 8 bytes	int32 footer length + magic PAR1

Code.

import struct

def read_parquet_map(path):
    """Show the footer-first read path by hand (no parquet library)."""
    with open(path, "rb") as f:
        # 1. header magic
        assert f.read(4) == b"PAR1", "not a parquet file"

        # 2. last 8 bytes: footer length (int32 LE) + trailing magic
        f.seek(-8, 2)                       # 8 bytes before EOF
        footer_len = struct.unpack("<I", f.read(4))[0]
        assert f.read(4) == b"PAR1", "bad trailing magic"

        # 3. seek back to the start of the thrift FileMetaData
        import os
        size = os.path.getsize(path)
        f.seek(size - 8 - footer_len)
        footer_bytes = f.read(footer_len)   # thrift-encoded FileMetaData

    return footer_len, footer_bytes[:16]

flen, head = read_parquet_map("/tmp/events.parquet")
print(f"footer length      : {flen} bytes")
print(f"footer starts with : {head!r}")

Step-by-step explanation.

1. The reader confirms the leading PAR1 — a cheap sanity check that this is a Parquet file at all.
2. It reads the trailing 8 bytes to learn the footer length and confirm the trailing PAR1. Two magic strings bracket the file so a truncated file is detectable from either end.
3. It computes filesize - 8 - footer_len and reads exactly the footer. This is the only metadata read; everything the reader needs to plan the query is in these bytes.
4. The thrift FileMetaData decodes into the schema plus, for each row group, a list of column chunks — each with its codec, encodings, data_page_offset, dictionary_page_offset, sizes, and statistics.
5. With that map, reading "one column from one row group" is a single seek to that column chunk's offset and a read of its byte length — on S3 that's roughly two round trips total (one ranged GET for the footer, one for the column chunk).

Output.

Read	Bytes	Purpose
Last 8 bytes	8	footer length + magic
Footer	footer_len	full metadata / query plan
Target column chunk	chunk size	the actual data
Full file	never (for a projected query)	—

Rule of thumb. Parquet is a "read the map, then fetch the pieces" format. The footer is the map; a well-planned query on object storage is two-to-three ranged reads, not a download. This is why tiny files (thousands of them) are slow — you pay the footer round trip per file.

Common beginner mistakes

• Treating Parquet like a compressed CSV. It is not a row format with gzip on top; the columnar layout, per-chunk statistics, and page structure are the point. Reasoning about it as "CSV but smaller" leads you to miss projection and predicate pushdown entirely.
• Assuming one .parquet file is one indivisible blob. A file has internal structure — row groups, column chunks, pages — that readers exploit for parallelism and skipping. "Read the file" is really "read a chosen subset of chunks."
• Thinking columns are compressed as one giant stream. Each column chunk is split into pages, each page encoded and compressed independently, so the reader can skip and decode at page granularity.
• Believing more/smaller files is always faster. Thousands of tiny files means thousands of footer round trips; the small-files problem is a top cause of slow lake scans.

Interview question on the columnar read model

A senior interviewer often opens with: "Forget the API for a second. At the byte level, explain why reading two columns out of a forty-column Parquet table is cheap, why the same query on a CSV is not, and what specifically in the Parquet layout the reader uses to avoid reading the other thirty-eight columns and the irrelevant rows."

Solution Using the columnar layout, the footer map, and per-chunk statistics

import pyarrow as pa
import pyarrow.parquet as pq
import pyarrow.dataset as ds

# Build a 40-column-ish table to make the projection story concrete
import random
random.seed(0)
n = 200_000
cols = {f"c{i}": pa.array([random.random() for _ in range(n)]) for i in range(38)}
cols["country"] = pa.array([random.choice(["US", "GB", "IN", "DE"]) for _ in range(n)])
cols["amount"]  = pa.array([round(random.random() * 500, 2) for _ in range(n)])
tbl = pa.table(cols)

pq.write_table(tbl, "/tmp/wide.parquet", row_group_size=50_000, write_statistics=True)

# 1. Projection pushdown — read ONLY the 2 columns we need
two_cols = pq.read_table("/tmp/wide.parquet", columns=["country", "amount"])
print("projected columns:", two_cols.column_names)

# 2. Predicate pushdown — let the reader use row-group statistics to skip
res = ds.dataset("/tmp/wide.parquet").to_table(
    columns=["country", "amount"],
    filter=(ds.field("country") == "US") & (ds.field("amount") > 400),
)
print("rows after pushdown:", res.num_rows)

# 3. Prove the reader only needs the footer to plan: inspect the map
md = pq.ParquetFile("/tmp/wide.parquet").metadata
print("row groups:", md.num_row_groups, "columns:", md.num_columns)

Step-by-step trace.

Step	What the reader does	Bytes touched
Open	read last 8 bytes → footer length + PAR1	8
Plan	decode FileMetaData (schema + chunk offsets + stats)	footer only
Project	keep only country, amount column chunks	2 of 40 chunks/row group
Predicate	drop row groups whose country/amount min-max excludes the filter	skipped chunks: 0
Decode	read + decompress surviving column chunks	tiny fraction of file

On a CSV the same query is a full scan: there is no footer to plan with, no column boundaries to project on, and no statistics to skip with, so the reader must parse every byte of every row to evaluate the filter and pull the two fields.

Output:

Aspect	CSV	Parquet
Columns read	all 40	2
Metadata read	none (parse everything)	footer only
Row skipping	none	row-group min/max
Typical bytes	full file	small fraction

Why this works — concept by concept:

• Columnar layout — each column is a contiguous column chunk per row group, so projecting 2 of 40 columns is a structural byte reduction, not a filter applied after reading.
• Footer map — the thrift FileMetaData at the file end holds every chunk's byte offset and statistics, so the reader plans the whole query from one metadata read before touching a data byte.
• Per-chunk statistics — min/max/null_count per column chunk let the reader prove a row group can't satisfy the predicate and skip it without decompressing anything.
• Immutability — because the file is write-once, the statistics baked in at write time are trustworthy at read time forever; there is no update path to invalidate them.
• Cost — planning is O(footer size); reading is O(surviving column chunks), not O(file). Compared to CSV's O(file) parse, Parquet turns an analytical scan from "read everything" into "read the map, then read the pieces."

Optimization
Topic — optimization
Optimization problems on columnar scan cost

Practice →

Data Processing Topic — file-io Data-processing problems on file formats

Practice →

---

2. Physical layout — row groups, column chunks, pages, and the footer

A Parquet file is magic bytes, a sequence of row groups (each a grid of column chunks made of pages), then a thrift footer — know every level of the hierarchy

The mental model in one line: on disk, the parquet file format is PAR1 magic bytes, then a sequence of row groups where each row group holds one column chunk per column, each column chunk is a run of pages (an optional dictionary page followed by data pages), and finally a thrift-encoded FileMetaData footer plus a 4-byte footer length and a trailing PAR1 — and every read decision (skip this row group, read that column, decode this page) maps to a specific level of that hierarchy. Internalize the four levels — file, row group, column chunk, page — and every performance question becomes "which level does this happen at?"

The four levels of the hierarchy.

• File. Brackets everything with PAR1 at byte 0 and PAR1 as the last 4 bytes. The footer (FileMetaData) sits just before the trailing magic, preceded by its own 4-byte length. One file = one schema = one immutable unit.
• Row group. A horizontal slice of rows (e.g. rows 0–127 of a target-128 MB group). All columns' data for those rows live here as column chunks. Row groups are the unit of parallelism and the coarse unit of skipping.
• Column chunk. Within a row group, one column's data — contiguous on disk. Carries its own ColumnMetaData: type, codec, encodings, value count, sizes, page offsets, and statistics. Column chunks are the unit of projection.
• Page. Within a column chunk, the smallest encoded+compressed unit (default target ~1 MB). A DICTIONARY_PAGE (optional) precedes DATA_PAGEs. Pages carry definition/repetition levels plus encoded values, and are the finest unit of decoding and (with the page index) skipping.

The footer — FileMetaData is the whole map.

• Schema. The full column schema (names, physical + logical types, repetition) as a tree — Parquet's schema is Dremel-style to support nesting.
• Row group metadata. For each row group: num_rows, total_byte_size, and a list of ColumnChunk entries.
• Column chunk metadata. Per chunk: path_in_schema, type, encodings, codec, num_values, total_uncompressed_size, total_compressed_size, data_page_offset, dictionary_page_offset, and statistics (min, max, null_count, distinct_count).
• Thrift + length + magic. The footer is Apache Thrift compact-encoded, followed by a little-endian int32 length and the trailing PAR1. That length is how the reader finds the footer start from the file end.

Pages — the sub-column detail.

• Page header. Each page starts with a thrift PageHeader: page type, uncompressed and compressed sizes, value count, and (for data pages) the encoding of values and of the def/rep levels.
• Data page v1 vs v2. DATA_PAGE (v1) compresses the whole page including levels; DATA_PAGE_V2 stores repetition and definition levels uncompressed and separately, so a reader can evaluate nullability/nesting without decompressing values — better for pushdown.
• Dictionary page. If a column uses dictionary encoding, one DICTIONARY_PAGE at the start of the chunk holds the distinct values; data pages then store dictionary indices.
• Definition & repetition levels. Dremel encoding: definition levels record how deep a nullable/optional value is defined; repetition levels record where repeated (list) values start. For flat, required columns these are trivial, but they are always conceptually present.

Common interview probes on the layout.

• "What are the units of parallelism, projection, and skipping?" — row group, column chunk, and (row group / page) respectively.
• "Where does a reader start reading?" — the footer, found via the last 8 bytes.
• "What's in a column chunk's metadata?" — codec, encodings, offsets, sizes, statistics.
• "Why two magic strings?" — bracket the file so truncation is detectable and the footer is locatable from the end.

Worked example — inspecting the full hierarchy with pyarrow

Detailed explanation. The fastest way to make the hierarchy real is to write a file and walk its metadata: file → row groups → column chunks → statistics and offsets. pyarrow exposes every level of FileMetaData.

• File level. ParquetFile(path).metadata gives num_rows, num_row_groups, num_columns, created_by.
• Row group level. metadata.row_group(i) gives num_rows, total_byte_size.
• Column chunk level. row_group(i).column(j) gives path_in_schema, compression, encodings, data_page_offset, dictionary_page_offset, total_compressed_size, and statistics.

Question. Write a file with a known row-group size and print the four-level hierarchy: file, each row group, and each column chunk's codec, encodings, offsets, and min/max.

Input.

Parameter	Value
Rows	100,000
Columns	event_id, user_id, country, amount, ts
row_group_size	20,000 (→ 5 row groups)
Statistics	on

Code.

import pyarrow as pa
import pyarrow.parquet as pq
import random

random.seed(7)
n = 100_000
tbl = pa.table({
    "event_id": pa.array(range(n), type=pa.int64()),
    "user_id":  pa.array([random.randint(1, 5_000_000) for _ in range(n)], type=pa.int64()),
    "country":  pa.array([random.choice(["US", "GB", "IN", "DE", "BR"]) for _ in range(n)]),
    "amount":   pa.array([round(random.random() * 500, 2) for _ in range(n)], type=pa.float64()),
    "ts":       pa.array([1_700_000_000 + i for i in range(n)], type=pa.int64()),
})

pq.write_table(tbl, "/tmp/events.parquet",
               row_group_size=20_000,
               use_dictionary=["country"],
               write_statistics=True)

md = pq.ParquetFile("/tmp/events.parquet").metadata
print(f"FILE  rows={md.num_rows}  row_groups={md.num_row_groups}  cols={md.num_columns}")
print(f"      created_by={md.created_by}")

rg = md.row_group(0)                       # first row group
print(f"ROWGROUP 0  rows={rg.num_rows}  bytes={rg.total_byte_size}")
for j in range(md.num_columns):
    c = rg.column(j)
    s = c.statistics
    print(f"  CHUNK {c.path_in_schema:9s} codec={c.compression:7s} "
          f"enc={c.encodings} dict_off={c.dictionary_page_offset} "
          f"data_off={c.data_page_offset} "
          f"min={s.min} max={s.max} nulls={s.null_count}")

Step-by-step explanation.

1. md.num_row_groups is 5 because 100,000 rows at row_group_size=20,000 splits into 5 horizontal slices. Row-group count is the knob that controls parallelism granularity and skipping resolution.
2. rg.total_byte_size is the compressed on-disk size of that row group across all its column chunks — the number Spark/Trino use to decide split boundaries.
3. Each c is a column chunk. c.compression is the codec, c.encodings the encoding list (e.g. RLE_DICTIONARY for country, PLAIN for a non-dictionary int), and the two offsets say where the dictionary page and first data page start inside the file.
4. c.statistics carries min, max, and null_count. For country you'll see min='BR' max='US'; for ts a tight [1700000000, 1700019999] range because that row group holds a contiguous timestamp block — exactly the property that makes range predicates on ts skippable.
5. dictionary_page_offset is populated only for dictionary-encoded columns; for country it points at the DICTIONARY_PAGE that precedes the data pages, while a PLAIN-encoded column has dictionary_page_offset = None.

Output.

Level	Field	Example value
File	row_groups	5
Row group 0	num_rows	20,000
Chunk country	encodings	('PLAIN','RLE','RLE_DICTIONARY')
Chunk ts	min / max	1700000000 / 1700019999
Chunk user_id	codec	(as configured)

Rule of thumb. To debug any Parquet performance question, dump FileMetaData and read it top-down: how many row groups (parallelism/skipping), what codec + encoding per column (size), and how tight the per-chunk min/max ranges are (skippability). The metadata tells you why a scan is slow before you profile anything.

Worked example — reading pages and the dictionary page

Detailed explanation. Below the column chunk is the page. A reader decodes at page granularity, and the page index (Section 4) lets it skip at page granularity. Walk the page structure of one column chunk.

• Dictionary page first. For a dictionary-encoded column, the chunk begins with one DICTIONARY_PAGE holding the distinct values.
• Then data pages. Each DATA_PAGE (or DATA_PAGE_V2) holds def levels, rep levels, and encoded values (dictionary indices, or plain/delta-encoded values).
• Page size target. The writer flushes a data page when it hits data_page_size (default ~1 MiB). Smaller pages mean finer skipping but more per-page header overhead.

Question. Show how to enumerate the pages of a column chunk conceptually and explain what each page contributes, using the metadata and a row-count reconciliation.

Input.

Concept	Meaning
DICTIONARY_PAGE	distinct values for a dictionary-encoded column
DATA_PAGE (v1)	levels + values, whole page compressed
DATA_PAGE_V2	levels stored uncompressed + separately
data_page_size	writer flush threshold (~1 MiB default)

Code.

import pyarrow.parquet as pq

pf = pq.ParquetFile("/tmp/events.parquet")
md = pf.metadata

# Reconcile: sum of per-row-group values == file rows, per column
col_index = md.schema.names.index("country")
total = 0
for r in range(md.num_row_groups):
    chunk = md.row_group(r).column(col_index)
    total += chunk.num_values
    print(f"rowgroup {r}: country chunk num_values={chunk.num_values} "
          f"uncompressed={chunk.total_uncompressed_size} "
          f"compressed={chunk.total_compressed_size} "
          f"has_dict_page={chunk.dictionary_page_offset is not None}")

print("sum of chunk values:", total, "== file rows:", md.num_rows)

# The uncompressed/compressed ratio is the codec's effect on this column
chunk0 = md.row_group(0).column(col_index)
ratio = chunk0.total_uncompressed_size / max(chunk0.total_compressed_size, 1)
print(f"country row-group-0 compression ratio: {ratio:.1f}x")

Step-by-step explanation.

1. Each column chunk reports num_values; summing across row groups reconciles to the file's num_rows, proving the row groups partition the rows exactly with no overlap.
2. total_uncompressed_size vs total_compressed_size on a chunk shows the codec's byte savings for that column — low-cardinality country under dictionary + Snappy compresses hard, a random float amount far less.
3. dictionary_page_offset is not None confirms the chunk opens with a dictionary page. When present, the data pages store small integer indices into that dictionary rather than the raw strings.
4. A reader that only needs country seeks to this chunk's dictionary_page_offset (or data_page_offset if no dictionary), reads total_compressed_size bytes, and decodes — it never touches any other column's pages.
5. With the page index enabled (Section 4), the reader can go finer: read the OffsetIndex to find a specific page's byte range and read just that page, skipping the rest of the chunk.

Output.

Metric	Meaning
num_values per chunk	rows in that row group for that column
sum == num_rows	row groups partition rows exactly
uncompressed/compressed	per-column codec effect
dictionary_page_offset set	chunk leads with a dictionary page

Rule of thumb. The chunk-level num_values and size fields are your cheapest diagnostic: they reconcile row counts and expose per-column compression ratios without reading a single data byte. If one column dominates total_compressed_size, that's your encoding/compression target.

Common beginner mistakes

• Confusing row groups with pages. Row groups are the coarse horizontal slices (unit of parallelism/skipping); pages are the fine sub-column units inside a column chunk. Predicate pushdown skips at row-group level always, and at page level only with the page index.
• Assuming the schema lives at the file top. The schema is in the footer, not a header — that's why a reader seeks to the end first.
• Forgetting the dictionary page. A dictionary-encoded column stores indices in its data pages and the actual values once in a leading dictionary page; miss it and the data pages look like meaningless small integers.
• Setting a huge row-group size for "fewer files." Row groups are internal to a file; oversizing them wastes memory on write and coarsens skipping — it does not reduce file count.

Interview question on the physical layout

A senior interviewer might ask: "Draw the on-disk layout of a Parquet file from the first byte to the last. Name every level of the hierarchy, say what metadata lives where, and explain exactly how a reader goes from 'open this file on S3' to 'I have the amount column for row group 3 decoded' — including which bytes it reads and in what order."

Solution Using the four-level hierarchy and the footer-driven read plan

import pyarrow as pa
import pyarrow.parquet as pq
import struct, os

# Write a file we can dissect
n = 60_000
tbl = pa.table({
    "id":     pa.array(range(n), type=pa.int64()),
    "amount": pa.array([round(i * 0.5, 2) for i in range(n)], type=pa.float64()),
})
pq.write_table(tbl, "/tmp/layout.parquet", row_group_size=15_000, write_statistics=True)

path = "/tmp/layout.parquet"
size = os.path.getsize(path)

# 1. footer-first: read the map
with open(path, "rb") as f:
    assert f.read(4) == b"PAR1"                       # header magic
    f.seek(-8, 2)
    footer_len = struct.unpack("<I", f.read(4))[0]    # footer length
    assert f.read(4) == b"PAR1"                        # trailing magic

md = pq.ParquetFile(path).metadata
print(f"file size={size}  footer_len={footer_len}  row_groups={md.num_row_groups}")

# 2. plan: locate the amount column chunk in row group 3
amount_idx = md.schema.names.index("amount")
rg3 = md.row_group(3)
chunk = rg3.column(amount_idx)
start = chunk.dictionary_page_offset or chunk.data_page_offset
length = chunk.total_compressed_size
print(f"amount@rg3: byte range [{start}, {start + length}) = {length} bytes")

# 3. targeted read: only that column of only that row group
one = pq.read_table(path, columns=["amount"]).slice(3 * 15_000, 15_000)
print(f"decoded amount rows for rg3: {one.num_rows}, first={one['amount'][0].as_py()}")

Step-by-step trace.

Step	Action	Bytes / result
Header	read bytes 0–3	PAR1
Tail	read last 8 bytes	footer length + PAR1
Footer	decode FileMetaData	schema + 4 row groups
Locate	row_group(3).column('amount')	offset + total_compressed_size
Fetch	ranged read of that chunk	one column, one row group
Decode	decompress + decode pages	15,000 amount values

The reader goes end → footer → chunk offset → ranged read → decode. It never reads the id column chunks, never reads row groups 0–2, and never scans the file front-to-back.

Output:

Question	Answer
Where does reading start?	last 8 bytes → footer
What is projected?	only the amount column chunks
What is skipped?	row groups 0–2 (not requested)
Round trips on S3	~2 (footer + chunk)

Why this works — concept by concept:

• Magic-byte brackets — PAR1 at both ends makes the file self-delimiting and lets the reader find the footer from the end via the 4-byte length.
• FileMetaData footer — one thrift structure holds schema plus every column chunk's byte offset, size, codec, encodings, and statistics: the complete query-planning map.
• Row group + column chunk addressing — because each chunk records its own offset and length, "column X of row group Y" resolves to an exact byte range, enabling a single ranged read.
• Page granularity underneath — the fetched chunk decodes page by page (dictionary page then data pages), which is also the granularity the page index skips at.
• Cost — planning is O(footer); fetching is O(one column chunk) here, not O(file). The hierarchy turns "read a Parquet file" into "read the footer, then read the few chunks the query needs," which is the entire performance story of the format.

Data Processing
Topic — file-io
Data-processing problems on file internals

Practice →

ETL Topic — etl ETL problems on Parquet write layout

Practice →

---

3. Encodings and compression — dictionary, RLE, bit-packing, delta, then Snappy/Zstd

parquet encoding is structure-aware and happens per column; compression is a byte-level codec layered on top — they are two distinct stages and both matter

The mental model in one line: parquet encoding transforms a column's typed values into a compact representation that exploits the column's structure — a dictionary encoding for repeats, a run-length encoding for runs, bit-packing for small integers, delta encoding for sequences — and then a general-purpose compression codec (Snappy, Zstd, Gzip, LZ4) squeezes the already-encoded bytes further; encoding understands what the data means, compression only sees bytes, and a well-configured column uses both. Confusing the two stages is the most common Parquet knowledge gap; keeping them separate is a senior signal.

Encoding stage 1 — the encodings, and when each wins.

• PLAIN. Values written raw (little-endian ints, IEEE floats, length-prefixed byte arrays). The fallback when nothing smarter applies; large but trivially decodable.
• Dictionary encoding (RLE_DICTIONARY). Distinct values go once into a dictionary page; data pages store small integer indices into it. Enormous win for low-cardinality columns (country, status, category). The writer falls back to PLAIN if the dictionary would exceed dictionary_pagesize_limit (default ~1 MiB) — i.e. cardinality is too high.
• Run-length + bit-packing hybrid (RLE). Used for the dictionary indices and for definition/repetition levels. Long runs of the same value collapse to (value, run_length); short runs are bit-packed into the minimum bits per value. A column of mostly-repeating indices becomes tiny.
• Delta encoding (DELTA_BINARY_PACKED). Stores differences between consecutive integers, then bit-packs the (small) deltas. Ideal for sorted or monotonic integer columns — ids, timestamps. A dense ts sequence encodes to near-zero.
• Delta strings (DELTA_BYTE_ARRAY / DELTA_LENGTH_BYTE_ARRAY). Prefix/length delta for sorted string columns — shared prefixes (user_0000001, user_0000002) are stored once.
• Byte stream split (BYTE_STREAM_SPLIT). Splits each float into its constituent bytes and stores same-position bytes together, so a general compressor finds patterns floats otherwise hide. Good for scientific/float-heavy columns.

Encoding stage 2 — compression codecs, layered on the encoded bytes.

• Snappy (default). Fast compress/decompress, moderate ratio. The safe default for hot analytical data where CPU matters.
• Zstd. Better ratio than Snappy, tunable level (higher = smaller/slower). The modern default for cold/large data where storage and I/O dominate.
• Gzip. High ratio, slow. Legacy; Zstd generally dominates it.
• LZ4 Very fast, lower ratio. Latency-sensitive paths.
• Uncompressed. Rare; only when CPU is the hard bottleneck and data is already tiny after encoding.

The order matters — encode then compress.

• Encoding first turns semantically-redundant values into fewer, smaller bytes (indices, deltas, runs). It is lossless and type-aware.
• Compression second finds byte-level redundancy the encoder didn't capture. It is lossless and type-blind.
• Why both. Compression alone on PLAIN data leaves value structure on the table; encoding alone leaves byte-level redundancy. Stacking them is why a low-cardinality string column can shrink 20–50×.

Common interview probes on encoding and compression.

• "Difference between encoding and compression?" — required answer: encoding is structure-aware and per column; compression is byte-level on the encoded output.
• "When does dictionary encoding hurt?" — high-cardinality columns, where the dictionary blows the size limit and the writer falls back to PLAIN anyway.
• "Best encoding for a sorted timestamp id?" — DELTA_BINARY_PACKED.
• "Snappy vs Zstd?" — Snappy for CPU-bound hot reads, Zstd for storage/I/O-bound cold data; Zstd level trades ratio for CPU.

Worked example — dictionary encoding on a low-cardinality column

Detailed explanation. Dictionary encoding is the single highest-leverage parquet encoding for warehouse data because so many columns are low cardinality. Show it by writing the same column with and without dictionary encoding and comparing sizes.

• Dictionary path. country has 5 distinct values; the dictionary page stores 5 strings, data pages store 3-bit indices, RLE collapses runs — tiny.
• PLAIN path. Every country value stored as a length-prefixed string — ~3 bytes of data plus overhead per row, millions of times.

Question. Write a 100k-row country column both with and without dictionary encoding and measure the on-disk difference; explain the fallback rule.

Input.

Column	Cardinality	Encoding A	Encoding B
country	5	dictionary (RLE_DICTIONARY)	PLAIN

Code.

import pyarrow as pa
import pyarrow.parquet as pq
import os, random

random.seed(3)
n = 100_000
country = pa.array([random.choice(["US", "GB", "IN", "DE", "BR"]) for _ in range(n)])
tbl = pa.table({"country": country})

# A. dictionary encoding on (default)
pq.write_table(tbl, "/tmp/dict.parquet",
               use_dictionary=["country"], compression="snappy")

# B. dictionary off -> PLAIN
pq.write_table(tbl, "/tmp/plain.parquet",
               use_dictionary=False, column_encoding={"country": "PLAIN"},
               compression="snappy")

a = os.path.getsize("/tmp/dict.parquet")
b = os.path.getsize("/tmp/plain.parquet")
print(f"dictionary : {a:>7} bytes")
print(f"plain      : {b:>7} bytes")
print(f"dictionary is {b / a:.1f}x smaller")

enc = pq.ParquetFile("/tmp/dict.parquet").metadata.row_group(0).column(0).encodings
print("dictionary file encodings:", enc)

Step-by-step explanation.

1. With dictionary encoding, the 5 distinct country strings are stored once in a dictionary page; each of the 100,000 rows becomes a 3-bit index (5 values fit in 3 bits), and RLE collapses long same-country runs even further.
2. With PLAIN encoding, each row stores its country as a length-prefixed byte array — the string "US" repeated 20,000 times as actual bytes, leaving only byte-level redundancy for Snappy to find.
3. The dictionary file is materially smaller even after Snappy runs on both, because the encoder removed value-level redundancy that Snappy can only partially recover from the PLAIN bytes.
4. The encodings list on the dictionary file includes RLE_DICTIONARY (indices) plus RLE (levels) and PLAIN (the dictionary page values themselves are PLAIN-encoded).
5. The fallback rule: if the dictionary for a column would exceed dictionary_pagesize_limit (default ~1 MiB) — i.e. too many distinct values — the writer abandons dictionary encoding for that chunk and reverts to PLAIN. High cardinality defeats dictionary encoding automatically.

Output.

Encoding	Relative size	Why
Dictionary + Snappy	1× (smallest)	5 strings once + 3-bit RLE indices
PLAIN + Snappy	several× larger	full strings per row, only byte-level dedupe

Rule of thumb. Leave dictionary encoding on (it's the default) for low- and medium-cardinality columns; it is the biggest single size win in a warehouse table. Only disable it for genuinely high-cardinality columns where the writer would fall back to PLAIN anyway, or where you deliberately want delta/byte-stream-split instead.

Worked example — delta encoding a sorted id, byte-stream-split a float

Detailed explanation. Dictionary is not always the right tool. A monotonic integer id wants DELTA_BINARY_PACKED; a float column wants BYTE_STREAM_SPLIT. Show both by choosing explicit encodings.

• DELTA_BINARY_PACKED. For id = 0,1,2,3,..., consecutive deltas are all 1, which bit-packs to ~1 bit each — the column nearly vanishes.
• DELTA_BYTE_ARRAY. For sorted strings like user_0000001, shared prefixes are stored once and only the differing suffix per row.
• BYTE_STREAM_SPLIT. Regroups float bytes by position so the sign/exponent bytes (often similar across neighbors) sit together and compress.

Question. Write a sorted-id, sorted-name, and float table with explicit per-column encodings and confirm the encodings landed.

Input.

Column	Data	Encoding
id	0..49999 (monotonic)	DELTA_BINARY_PACKED
name	user_0000000.. (sorted)	DELTA_BYTE_ARRAY
price	monotone floats	BYTE_STREAM_SPLIT

Code.

import pyarrow as pa
import pyarrow.parquet as pq

n = 50_000
tbl = pa.table({
    "id":    pa.array(range(n), type=pa.int64()),
    "name":  pa.array([f"user_{i:07d}" for i in range(n)]),
    "price": pa.array([i * 0.01 for i in range(n)], type=pa.float64()),
})

pq.write_table(
    tbl, "/tmp/enc.parquet",
    use_dictionary=False,                              # turn off dict so our choices apply
    column_encoding={"id": "DELTA_BINARY_PACKED",
                     "name": "DELTA_BYTE_ARRAY"},
    use_byte_stream_split=["price"],                   # float-specific transform
    compression="zstd",
    row_group_size=10_000,
    sorting_columns=[pq.SortingColumn(0)],             # declare id ascending in metadata
)

rg = pq.ParquetFile("/tmp/enc.parquet").metadata.row_group(0)
for j, col in enumerate(["id", "name", "price"]):
    print(f"{col:6s} encodings={rg.column(j).encodings}")
print("declared sort:", pq.ParquetFile("/tmp/enc.parquet").metadata.row_group(0).sorting_columns)

Step-by-step explanation.

1. use_dictionary=False is required first — pyarrow prefers dictionary encoding by default, and it would override the explicit column_encoding choices otherwise.
2. id gets DELTA_BINARY_PACKED: consecutive deltas are constant 1, so the bit-packed delta stream is a fraction of a bit-packed absolute-value stream. Monotonic ids and timestamps are the textbook case.
3. name gets DELTA_BYTE_ARRAY: because names are sorted, each row shares a long prefix with the previous one; the encoder stores the shared prefix length plus the differing suffix.
4. price gets BYTE_STREAM_SPLIT: the float bytes are transposed so that byte position 0 of every value is contiguous, byte position 1 contiguous, and so on — a layout Zstd compresses better than interleaved float bytes.
5. sorting_columns=[SortingColumn(0)] records in the footer that the file is sorted ascending by column 0 (id). This does not change bytes, but it tells downstream readers the file is sorted, which enables sort-merge and range-skipping optimizations.

Output.

Column	Encodings observed	Payoff
id	('RLE','DELTA_BINARY_PACKED')	constant deltas → near-zero
name	('RLE','DELTA_BYTE_ARRAY')	shared prefixes stored once
price	('RLE','BYTE_STREAM_SPLIT')	transposed bytes compress better

Rule of thumb. Match the encoding to the column's structure: dictionary for low cardinality, DELTA_BINARY_PACKED for monotonic ints/timestamps, DELTA_BYTE_ARRAY for sorted strings, BYTE_STREAM_SPLIT for floats. Turn dictionary off when you want one of the others to apply, and declare sorting_columns when the data is sorted so readers can exploit it.

Worked example — choosing a compression codec by workload

Detailed explanation. After encoding, the compression codec is a CPU-versus-ratio dial. The right choice depends on whether the workload is CPU-bound (hot, frequently scanned) or I/O/storage-bound (cold, large, rarely read).

• Snappy. ~fast, moderate ratio. Default for interactive/hot tables.
• Zstd (level 1–3). Better ratio at modest CPU; the modern default for most warehouse data.
• Zstd (level 9+). Best ratio, high CPU; for cold archival where you write once and rarely read.

Question. Write the same amount column under Snappy, Zstd level 3, and Zstd level 12 and compare size; explain how to choose.

Input.

Codec	Level	Bias
Snappy	n/a	speed
Zstd	3	balanced
Zstd	12	ratio

Code.

import pyarrow as pa
import pyarrow.parquet as pq
import os, random

random.seed(5)
n = 500_000
tbl = pa.table({"amount": pa.array([round(random.random() * 1000, 2) for _ in range(n)],
                                    type=pa.float64())})

def write(path, codec, level=None):
    kwargs = {"compression": codec}
    if level is not None:
        kwargs["compression_level"] = level
    pq.write_table(tbl, path, use_dictionary=False, **kwargs)
    return os.path.getsize(path)

s  = write("/tmp/amt_snappy.parquet", "snappy")
z3 = write("/tmp/amt_zstd3.parquet",  "zstd", 3)
z12 = write("/tmp/amt_zstd12.parquet", "zstd", 12)

print(f"snappy    : {s:>8} bytes")
print(f"zstd-3    : {z3:>8} bytes  ({s / z3:.2f}x vs snappy)")
print(f"zstd-12   : {z12:>8} bytes  ({s / z12:.2f}x vs snappy)")

Step-by-step explanation.

1. Snappy is the speed-first baseline: it decompresses fast, which matters when the same hot table is scanned thousands of times a day and CPU, not storage, is the constraint.
2. Zstd level 3 typically beats Snappy on size at a small CPU cost — for most warehouse tables this is the better default, because storage and network I/O usually dominate query cost more than decompression CPU.
3. Zstd level 12 squeezes further but the marginal ratio gain shrinks while CPU cost climbs; it pays off only for cold, write-once, rarely-read archival data.
4. Note this column is random floats, so encoding can't help much and the codec does the heavy lifting; on a structured column (sorted ids, low-card strings) the encoding would already have shrunk it and the codec difference would be smaller.
5. The choice is a function of read frequency: hot → Snappy/Zstd-low; warm → Zstd-mid; cold/archival → Zstd-high. Measure on your columns, because the ratio depends entirely on the data's structure.

Output.

Codec	Relative size	Choose when
Snappy	largest	hot, CPU-bound, interactive
Zstd-3	smaller	general warehouse default
Zstd-12	smallest	cold archival, write-once

Rule of thumb. Default new tables to Zstd level 3; keep Snappy for latency-critical hot paths; reserve high Zstd levels for cold archival. Always measure ratio on your real columns — encoding often matters more than the codec, so fix encoding first.

Common beginner mistakes

• Conflating encoding and compression. "Parquet uses Snappy" describes only stage two; the dictionary/RLE/delta encoding underneath usually does more of the shrinking. Both stages are configurable and both matter.
• Forcing dictionary encoding on a high-cardinality column. It silently falls back to PLAIN past the size limit, so you neither get dictionary benefits nor the delta/byte-split encoding you could have chosen.
• Cranking Zstd to max everywhere. High levels waste write CPU for little ratio gain on hot data and slow ingestion; match the level to read frequency.
• Ignoring sorting_columns. Sorting data enables both far better delta/RLE encoding and far better predicate skipping — but only if you actually sort before writing and declare it.

Interview question on encoding and compression

A senior interviewer might ask: "A colleague says 'we already gzip our Parquet files, so encoding doesn't matter.' Correct them: explain the difference between encoding and compression, walk through which encoding you'd pick for a low-cardinality status column, a monotonic event_ts column, and a random amount column, and say where the compression codec fits into that."

Solution Using per-column encoding choices with a compression codec on top

import pyarrow as pa
import pyarrow.parquet as pq

n = 200_000
tbl = pa.table({
    "status":   pa.array([["new", "paid", "shipped", "cancelled"][i % 4] for i in range(n)]),
    "event_ts": pa.array([1_700_000_000 + i for i in range(n)], type=pa.int64()),   # monotonic
    "amount":   pa.array([(i * 7919 % 100000) / 100 for i in range(n)], type=pa.float64()),  # noisy
})

pq.write_table(
    tbl, "/tmp/mixed.parquet",
    # stage 1 — encoding, chosen per column by its structure
    use_dictionary=["status"],                          # low cardinality -> dictionary
    column_encoding={"event_ts": "DELTA_BINARY_PACKED"},# monotonic -> delta
    use_byte_stream_split=["amount"],                   # float -> byte-stream-split
    # stage 2 — compression, one codec over the encoded bytes
    compression={"status": "snappy", "event_ts": "zstd", "amount": "zstd"},
    compression_level={"event_ts": 3, "amount": 3},
    row_group_size=50_000,
    write_statistics=True,
)

rg = pq.ParquetFile("/tmp/mixed.parquet").metadata.row_group(0)
for j, name in enumerate(["status", "event_ts", "amount"]):
    c = rg.column(j)
    print(f"{name:9s} enc={c.encodings} codec={c.compression} "
          f"ratio={c.total_uncompressed_size / max(c.total_compressed_size,1):.1f}x")

Step-by-step trace.

Column	Structure	Encoding (stage 1)	Codec (stage 2)
status	4 distinct values	dictionary + RLE indices	Snappy
event_ts	monotonic int	DELTA_BINARY_PACKED	Zstd-3
amount	noisy float	BYTE_STREAM_SPLIT	Zstd-3

The event_ts column shows the biggest ratio: constant deltas encode to almost nothing, then Zstd cleans up the residue. status shrinks hard from dictionary encoding; amount shrinks least because random floats have little structure for either stage to exploit.

Output:

Column	Dominant win	Why
status	encoding	4-value dictionary + RLE
event_ts	encoding	constant deltas ≈ 1 bit each
amount	compression	little structure; codec does the work

Why this works — concept by concept:

• Encoding is structure-aware — dictionary exploits low cardinality, delta exploits monotonicity, byte-stream-split exploits float byte layout. Each is chosen from what the column means, not how its bytes look.
• Compression is byte-level — Snappy/Zstd run on the already-encoded output and recover redundancy the encoder left behind; they neither know nor care about column semantics.
• The stages compose — encode-then-compress beats either alone, which is why a status column can shrink far more than gzip-on-CSV ever would.
• Per-column configuration — different columns get different encodings and codecs in the same file, because the right choice is a property of each column's data distribution.
• Cost — encoding is near-free on write and lossless; compression trades CPU for ratio. Get encoding right first (biggest, cheapest win), then dial the codec/level to the read-frequency of the table. gzip-on-Parquet alone leaves the encoding win — the larger one — on the table.

Optimization
Topic — optimization
Optimization problems on encoding and compression

Practice →

ETL Topic — etl ETL problems on write-time tuning

Practice →

---

4. Predicate and projection pushdown — statistics, page index, dictionary filtering

Pushdown is how Parquet turns a WHERE clause into skipped bytes — column statistics, the page index, dictionary filtering, and row-group skipping all let the reader avoid decompressing data it can prove is irrelevant

The mental model in one line: predicate pushdown is the reader using metadata written into the file — per-row-group and per-page min/max column statistics, the page index (ColumnIndex + OffsetIndex), and dictionary pages — to prove that a row group or page cannot contain any row matching the query filter, and skip reading it entirely, while projection pushdown skips whole columns; together they mean a selective query reads a small fraction of the file even though Parquet has no B-tree index. Pushdown is the feature interviewers probe hardest because it's where layout, statistics, and query planning meet.

Projection pushdown — the free one.

• What it is. Reading only the column chunks for columns the query references (SELECT, WHERE, GROUP BY, JOIN keys). Columns not referenced are never fetched.
• Why it's free. It's pure addressing: the footer knows each chunk's offset, so the reader just doesn't issue reads for unwanted columns. No statistics needed.
• Impact. Reading 2 of 40 columns is a ~20× I/O reduction before any predicate is considered.

Predicate pushdown level 1 — row-group skipping via column statistics.

• The statistic. Each column chunk stores min, max, null_count in the footer. For a predicate amount > 400, if a row group's amount max is 380, no row in that group can match — skip the whole group.
• The condition for skipping. Effective only when the predicate column's values are clustered per row group, so min/max ranges are narrow and non-overlapping. Random data → overlapping ranges → nothing skippable.
• Sorting is the multiplier. Sort (or partition/cluster) by the common filter column so each row group covers a disjoint range; then a range predicate reads one or two row groups instead of all of them.

Predicate pushdown level 2 — the page index.

• What it adds. The ColumnIndex stores per-page min/max (and null info) and the OffsetIndex stores each page's byte offset and row range — both written near the footer. Enabled with write_page_index=True.
• Why it helps. Without it, page-level min/max lives in each page header, so finding a skippable page means reading page headers sequentially. With the page index, the reader consults one compact structure and jumps straight to the surviving pages.
• Granularity. Turns skipping from row-group resolution (tens of thousands of rows) to page resolution (thousands of rows) — much finer for selective point/range queries.

Predicate pushdown level 3 — dictionary filtering.

• What it is. For an equality predicate (country = 'ZZ') on a dictionary-encoded column, the reader reads only the small dictionary page and checks whether the value is present. If 'ZZ' isn't in the dictionary, no data page in that chunk can contain it — skip the whole chunk.
• Why it's powerful. It's a cheap membership check against a tiny dictionary that can eliminate a large column chunk without reading a single data page.

Common interview probes on pushdown.

• "How does a WHERE clause skip data with no index?" — min/max column statistics per row group (+ page index per page).
• "Why does sorting the data speed up a range filter so much?" — disjoint, non-overlapping min/max ranges → most row groups skipped.
• "What does the page index give you over row-group statistics?" — page-level skipping without scanning page headers.
• "When do bloom filters help where min/max doesn't?" — high-cardinality equality (Section 5).

Worked example — row-group skipping on a sorted column

Detailed explanation. The clearest demonstration of predicate pushdown is to sort by the filter column, then show that a range predicate touches one row group. Contrast with unsorted data where ranges overlap and nothing skips.

• Sorted. Sort by ts; each row group covers a disjoint ts range, so ts BETWEEN a AND b maps to a contiguous, small set of row groups.
• Unsorted. Random ts; every row group's min/max spans nearly the whole range, so no group can be proven irrelevant.

Question. Write a sorted and an unsorted version of a ts column and show, via row-group statistics, how many row groups a range predicate can skip in each.

Input.

Version	Order	Expected skipping
sorted	ts ascending	most row groups skipped
unsorted	random ts	none skipped

Code.

import pyarrow as pa
import pyarrow.parquet as pq
import random

n, rg = 100_000, 20_000
sorted_ts   = list(range(n))
random_ts   = sorted_ts[:]; random.seed(1); random.shuffle(random_ts)

for name, data in [("sorted", sorted_ts), ("unsorted", random_ts)]:
    tbl = pa.table({"ts": pa.array(data, type=pa.int64())})
    pq.write_table(tbl, f"/tmp/{name}.parquet", row_group_size=rg, write_statistics=True)

def skippable(path, lo, hi):
    """How many row groups can be skipped for  lo <= ts <= hi ."""
    md = pq.ParquetFile(path).metadata
    skipped = 0
    for r in range(md.num_row_groups):
        st = md.row_group(r).column(0).statistics
        if st.max < lo or st.min > hi:        # range disjoint from predicate
            skipped += 1
    return skipped, md.num_row_groups

for name in ("sorted", "unsorted"):
    sk, total = skippable(f"/tmp/{name}.parquet", 40_000, 41_000)
    print(f"{name:9s}: predicate ts in [40000,41000] -> {sk}/{total} row groups skippable")

Step-by-step explanation.

1. Both files hold the same 100,000 values split into 5 row groups of 20,000; the only difference is order within the file.
2. In the sorted file, row group 0 covers ts 0–19,999, group 1 covers 20,000–39,999, group 2 covers 40,000–59,999, and so on — disjoint ranges.
3. The predicate ts BETWEEN 40000 AND 41000 falls entirely inside row group 2's range, so groups 0, 1, 3, 4 have min/max disjoint from the predicate and are skippable — 4 of 5 skipped.
4. In the unsorted file, every row group's min/max spans roughly 0–99,999 because random shuffling scatters all values everywhere; no group's range is disjoint from the predicate, so 0 of 5 are skippable.
5. The reader then only decompresses the surviving row group(s). Sorting converted a full-file scan into a one-row-group read — a 5× reduction here, and far larger at real row-group counts.

Output.

Version	Row groups skippable	Bytes read
sorted	4 / 5	~1 row group
unsorted	0 / 5	all row groups

Rule of thumb. Predicate pushdown is only as good as your data layout. Sort or cluster each table by its most common filter/join column before writing, so row-group min/max ranges become disjoint. Statistics are written automatically; the skipping they enable is entirely a function of how you ordered the rows.

Worked example — enabling and using the page index

Detailed explanation. Row-group skipping is coarse. The page index (ColumnIndex + OffsetIndex) enables page-level skipping and is a single write flag. Show enabling it and how the reader benefits.

• Write. write_page_index=True writes ColumnIndex (per-page min/max/null) and OffsetIndex (per-page offset + row range) near the footer.
• Read. The reader consults the page index to jump to surviving pages within a row group, instead of reading every page header in sequence.
• When it matters. Selective predicates where the matching rows cluster into a few pages of an otherwise large row group.

Question. Write a file with the page index enabled and confirm the reader can filter to a small result while (conceptually) skipping non-matching pages.

Input.

Setting	Value
write_page_index	True
data_page_size	64 KiB (small → many pages → finer skipping)
Predicate	id BETWEEN 55000 AND 55100

Code.

import pyarrow as pa
import pyarrow.parquet as pq

n = 200_000
tbl = pa.table({
    "id":  pa.array(range(n), type=pa.int64()),          # sorted
    "val": pa.array([i % 997 for i in range(n)], type=pa.int64()),
})

pq.write_table(
    tbl, "/tmp/pageidx.parquet",
    row_group_size=100_000,        # only 2 big row groups
    data_page_size=64 * 1024,      # small pages -> many per row group
    write_page_index=True,         # <-- ColumnIndex + OffsetIndex
    write_statistics=True,
)

md = pq.ParquetFile("/tmp/pageidx.parquet").metadata
print("row groups:", md.num_row_groups)

# A narrow predicate: with the page index, the reader skips non-matching pages
res = pq.read_table("/tmp/pageidx.parquet",
                    columns=["id", "val"],
                    filters=[("id", ">=", 55_000), ("id", "<=", 55_100)])
print("rows returned:", res.num_rows, "first id:", res["id"][0].as_py())

Step-by-step explanation.

1. With row_group_size=100_000, the file has only 2 row groups — so row-group skipping alone would still force reading a whole 100k-row group to answer a 100-row query.
2. data_page_size=64 KiB makes each row group contain many small pages; the sorted id gives each page a narrow, disjoint id range.
3. write_page_index=True records each page's min/max (ColumnIndex) and byte offset + first-row index (OffsetIndex) in a compact structure near the footer.
4. For id BETWEEN 55000 AND 55100, the reader consults the page index, finds the one or few pages whose id range overlaps the predicate, seeks directly to them via the OffsetIndex, and skips every other page in the row group — no sequential page-header scan.
5. The result is ~101 rows read from a 200,000-row file, at page granularity rather than row-group granularity — the page index is what makes a narrow predicate cheap even inside a large row group.

Output.

Feature	Without page index	With page index
Skip granularity	row group (100k rows)	page (~thousands of rows)
Finding skippable pages	read page headers in order	one compact index lookup
Bytes for narrow predicate	~1 full row group	a few pages

Rule of thumb. Turn on write_page_index=True for tables with selective range/point queries; it costs a little footer space and enables page-level skipping. Pair it with a small enough data_page_size and a sort on the predicate column so pages have tight, disjoint ranges.

Worked example — dictionary filtering for equality predicates

Detailed explanation. For an equality predicate on a dictionary-encoded column, the reader can check the dictionary page before touching any data page. If the value isn't in the dictionary, the whole chunk is skipped.

• The check. Read the small DICTIONARY_PAGE; is the literal present? If not, no data page can contain it.
• The payoff. A membership test against a handful of dictionary entries eliminates a full column chunk's data pages.
• The pairing. Works alongside min/max: min/max handles ranges, dictionary filtering handles equality on categorical columns.

Question. Show why an equality predicate for a value absent from a chunk's dictionary lets the reader skip the chunk, using a low-cardinality column.

Input.

Column	Distinct values	Predicate	Expected
region	US, EU, APAC	region = 'LATAM'	skip all chunks

Code.

import pyarrow as pa
import pyarrow.parquet as pq

n = 120_000
tbl = pa.table({
    "region": pa.array([["US", "EU", "APAC"][i % 3] for i in range(n)]),
    "v":      pa.array(range(n), type=pa.int64()),
})
pq.write_table(tbl, "/tmp/region.parquet",
               use_dictionary=["region"],
               row_group_size=30_000, write_statistics=True)

# Equality on a value that is NOT in the data at all
absent = pq.read_table("/tmp/region.parquet", filters=[("region", "==", "LATAM")])
present = pq.read_table("/tmp/region.parquet", filters=[("region", "==", "EU")])

print("rows for region='LATAM' (absent):", absent.num_rows)   # 0, chunk skipped
print("rows for region='EU'   (present):", present.num_rows)

# Confirm min/max also bracket the equality value for skipping
md = pq.ParquetFile("/tmp/region.parquet").metadata
st = md.row_group(0).column(0).statistics
print("region rg0 min/max:", st.min, st.max)

Step-by-step explanation.

1. region is dictionary-encoded with three distinct values, so each chunk carries a tiny dictionary page listing exactly the three values US, EU, APAC.
2. For region = 'LATAM', the reader checks each chunk's dictionary, finds 'LATAM' absent, and concludes no data page in that chunk can match — it skips the chunk's data pages entirely and returns zero rows without decompressing them.
3. Min/max statistics reinforce this: the chunk's min='APAC', max='US' string range also lets a reader reason about ordering, though for equality the dictionary membership test is the direct mechanism.
4. For region = 'EU', 'EU' is in the dictionary, so the reader cannot skip on membership; it decodes the data pages and returns matching rows (filtered further at page/row level).
5. Dictionary filtering is the equality-predicate complement to min/max range skipping: min/max prunes ranges, dictionary membership prunes categorical equality, and together they cover the common warehouse filter shapes on low-cardinality columns.

Output.

Predicate	Mechanism	Result
region = 'LATAM'	dictionary membership → absent	0 rows, chunk skipped
region = 'EU'	value present → must read	matching rows returned

Rule of thumb. Keep low-cardinality categorical columns dictionary-encoded (the default): it shrinks them and enables dictionary-filter skipping for equality predicates. For high-cardinality equality (ids, UUIDs) where the dictionary would be huge or absent, reach for bloom filters instead — that's Section 5.

Common beginner mistakes

• Expecting pushdown to help on unsorted data. Statistics exist regardless, but skipping only happens when min/max ranges are disjoint — which requires sorting or clustering by the filter column.
• Forgetting to enable the page index. Without write_page_index=True, page-level skipping falls back to sequential page-header scans; a narrow predicate then still reads a whole row group.
• Assuming Parquet has a B-tree index. It doesn't — pushdown is min/max + page index + dictionary + bloom filters, all statistical/probabilistic, not a sorted index structure.
• Selecting * and expecting speed. Projection pushdown is your biggest lever; SELECT * reads every column chunk and throws the advantage away.

Interview question on predicate pushdown

A senior interviewer might ask: "A query SELECT * FROM events WHERE event_ts BETWEEN '2026-01-01' AND '2026-01-02' scans a 500 GB Parquet table and reads almost all of it, despite the tight date range. The data is written in random order. Explain why pushdown isn't helping, and redesign the write path so the same query reads a small fraction — cover statistics, sorting, the page index, and projection."

Solution Using sorted writes, per-row-group statistics, the page index, and projection

import pyarrow as pa
import pyarrow.parquet as pq
import pyarrow.compute as pc
import random

# Simulate 'events' written in RANDOM order (the slow status quo)
n = 200_000
random.seed(2)
ts = [1_735_689_600 + random.randint(0, 30 * 86_400) for _ in range(n)]   # Jan 2026, shuffled
tbl = pa.table({
    "event_ts": pa.array(ts, type=pa.int64()),
    "user_id":  pa.array([random.randint(1, 9_999_999) for _ in range(n)], type=pa.int64()),
    "amount":   pa.array([round(random.random() * 200, 2) for _ in range(n)], type=pa.float64()),
})

# BEFORE — random order: min/max of every row group spans the whole month
pq.write_table(tbl, "/tmp/events_random.parquet", row_group_size=20_000, write_statistics=True)

# AFTER — sort by event_ts, small pages, page index on
sort_idx = pc.sort_indices(tbl, sort_keys=[("event_ts", "ascending")])
sorted_tbl = tbl.take(sort_idx)
pq.write_table(
    sorted_tbl, "/tmp/events_sorted.parquet",
    row_group_size=20_000,
    data_page_size=128 * 1024,
    write_statistics=True,
    write_page_index=True,
    sorting_columns=[pq.SortingColumn(0)],   # declare event_ts sorted
)

def skippable(path, lo, hi):
    md = pq.ParquetFile(path).metadata
    return sum(1 for r in range(md.num_row_groups)
               if md.row_group(r).column(0).statistics.max < lo
               or md.row_group(r).column(0).statistics.min > hi), md.num_row_groups

lo, hi = 1_735_689_600, 1_735_776_000   # a one-day window
print("random  :", skippable("/tmp/events_random.parquet", lo, hi))
print("sorted  :", skippable("/tmp/events_sorted.parquet", lo, hi))

Step-by-step trace.

Change	Before (random)	After (sorted + page index)
Row-group event_ts ranges	all span the whole month	disjoint per-day-ish blocks
Row groups skippable for 1-day filter	~0	most of them
Skip granularity	none	row group + page
Columns read	all (SELECT *)	project only what's needed
Declared sort	none	sorting_columns on event_ts

Sorting by event_ts makes each row group cover a contiguous time block, so the one-day predicate's min/max prunes nearly every row group; the page index prunes further inside the surviving group; and projecting only the needed columns cuts the per-surviving-group bytes. The same query now reads a small fraction of the 500 GB.

Output:

Metric	Random write	Sorted write
Row groups read (1-day filter)	~all	a few
Skip resolution	none	row group + page
Fix cost	—	sort before write + 2 flags

Why this works — concept by concept:

• Column statistics — min/max/null per row group is written automatically, but only useful when ranges are disjoint; random order makes every range span the whole domain, so nothing is provably skippable.
• Sorting the write — ordering rows by the filter column makes per-row-group min/max ranges narrow and non-overlapping, which is the precondition that turns statistics into skipped I/O.
• Page index — write_page_index=True adds ColumnIndex/OffsetIndex so skipping drops from row-group to page granularity inside the surviving group.
• Projection — reading only referenced columns (not SELECT *) multiplies the win by cutting per-surviving-group bytes.
• Cost — a one-time sort on write plus two flags converts an O(file) scan into O(matching row groups + pages). Statistics are free; the layout that makes them pay off is a deliberate write-path decision.

Indexing
Topic — indexing
Indexing problems on min/max and skipping

Practice →

Optimization Topic — optimization Optimization problems on predicate pushdown

Practice →

---

5. Bloom filters, tuning, and interview signals

Bloom filters skip row groups on high-cardinality equality where min/max can't help; row-group and page sizing tune the whole thing — and knowing when each applies is the senior interview signal

The mental model in one line: a bloom filter is a compact probabilistic set, stored per column chunk, that answers "is value V possibly in this chunk?" with no false negatives — so for a high-cardinality equality predicate (user_id = 12345, a UUID lookup) where min/max ranges are useless and the value isn't in any dictionary, the reader checks the bloom filter and skips every chunk that definitely doesn't contain V; combine that with deliberate row-group and page sizing, and you control the entire read-cost profile of the parquet file format. Bloom filters are the answer to the interview question "how do you skip on equality when the column is high-cardinality?"

Bloom filters — the high-cardinality equality tool.

• What it is. A bitset per column chunk built by hashing each value into k bit positions. To test membership, hash the query value and check those bits: any bit unset → definitely absent (skip); all set → possibly present (read).
• No false negatives, some false positives. It never wrongly skips a chunk that contains the value; it may occasionally read a chunk that doesn't (a false positive, tuned by fpp).
• Where it wins. Equality on high-cardinality columns — user_id, order_id, UUIDs, emails — where min/max ranges overlap uselessly and there's no small dictionary to check.
• Where it doesn't. Range predicates (use min/max), low-cardinality equality (use dictionary filtering). Bloom filters are equality-only and cost space, so add them selectively.

Tuning bloom filters — ndv and fpp.

• ndv (number of distinct values). Set roughly to the distinct count per chunk (a good default is the row count). Too low → over-full filter → high false-positive rate; too high → wasted space.
• fpp (false-positive probability). Typical 0.1, 0.05, 0.01. Lower fpp → bigger bitset but fewer wasted reads. Space grows ~log(1/fpp).
• Selectivity. Only add bloom filters to columns you actually filter by equality; a bloom filter on a column no query filters on is pure overhead.

Row-group sizing — the parallelism-vs-skipping dial.

• Bigger row groups. Better compression and fewer footer entries, but coarser skipping and higher per-task memory. Target ~128 MB is the common default.
• Smaller row groups. Finer skipping and lower memory, but more metadata overhead and weaker compression. Useful for highly selective queries on well-sorted data.
• The constraint. One row group should comfortably fit in a reader task's memory; too-large groups cause spills and OOMs.

Page sizing — the fine-skipping dial.

• Bigger pages (default ~1 MiB): less per-page header overhead, coarser page-index skipping.
• Smaller pages: finer page-index skipping for selective predicates, more header overhead.
• Pairs with the page index. Small pages only pay off for skipping when write_page_index=True and the data is sorted so pages have tight ranges.

Common interview probes on bloom filters and tuning.

• "How do you skip on user_id = X for a billion-row table?" — bloom filter per row group; min/max can't help on high-cardinality.
• "False positives vs false negatives?" — bloom filters have false positives, never false negatives, so they're safe for skipping.
• "How big should a row group be?" — target ~128 MB, must fit reader memory; smaller for selective sorted queries.
• "Bloom filter vs dictionary filter?" — dictionary for low-cardinality equality, bloom for high-cardinality equality.

Worked example — writing and using a bloom filter for a point lookup

Detailed explanation. Show a bloom filter earning its space: a high-cardinality user_id column where an equality lookup skips row groups that provably don't contain the id.

• Write. Passing bloom_filter_options a per-column spec for user_id (with ndv=n and fpp=0.01) builds a per-chunk bloom filter for user_id.
• Read (present). A user_id that exists → its row groups test "possibly present" → read and return the row.
• Read (absent). A user_id that exists in no chunk → every chunk tests "definitely absent" → all skipped → zero rows, minimal I/O.

Question. Write a file with a bloom filter on user_id, then run a present and an absent equality lookup and explain the skipping.

Input.

Setting	Value
Column	user_id (high cardinality)
ndv	row count
fpp	0.01
Predicates	present id; absent id -999

Code.

import pyarrow as pa
import pyarrow.parquet as pq
import random

random.seed(1)
n = 300_000
uids = [random.randint(1, 50_000_000) for _ in range(n)]
tbl = pa.table({
    "user_id": pa.array(uids, type=pa.int64()),
    "amount":  pa.array([round(random.random() * 100, 2) for _ in range(n)], type=pa.float64()),
})

pq.write_table(
    tbl, "/tmp/bloom.parquet",
    row_group_size=50_000,
    write_statistics=True,
    bloom_filter_options={"user_id": {"ndv": n, "fpp": 0.01}},
)

present = uids[123_456 % n]
hit  = pq.read_table("/tmp/bloom.parquet", filters=[("user_id", "==", present)])
miss = pq.read_table("/tmp/bloom.parquet", filters=[("user_id", "==", -999)])

print("present id:", present, "-> rows:", hit.num_rows)     # >= 1
print("absent  id: -999   -> rows:", miss.num_rows)          # 0, all chunks skipped

# min/max is useless here: every row group spans almost the whole id domain
md = pq.ParquetFile("/tmp/bloom.parquet").metadata
for r in range(md.num_row_groups):
    st = md.row_group(r).column(0).statistics
    print(f"rg{r} user_id min={st.min} max={st.max}")

Step-by-step explanation.

1. user_id is high-cardinality and randomly distributed, so every row group's min/max spans almost the full [1, 50M] domain — min/max skipping is worthless here, which is exactly the gap bloom filters fill.
2. The bloom_filter_options spec for user_id builds a bitset per column chunk by hashing every user_id in that chunk into k bit positions. ndv=n sizes it for the distinct count; fpp=0.01 targets a 1% false-positive rate.
3. For the present id, the reader hashes it and finds all its bits set in the row group(s) that actually contain it → "possibly present" → it reads those chunks and returns the matching row.
4. For the absent id -999, hashing it finds at least one unset bit in every chunk → "definitely absent" everywhere → the reader skips all data pages and returns zero rows, having read only footer + bloom filters.
5. The guarantee is one-directional: a bloom filter never reports "absent" for a value that is present (no false negatives), so skipping is always correct; the only cost of a false positive is an occasional unnecessary chunk read, bounded by fpp.

Output.

Predicate	min/max useful?	Bloom filter result	Rows
user_id = <present>	no	possibly present → read	≥ 1
user_id = -999	no	definitely absent → skip all	0

Rule of thumb. Add bloom filters to high-cardinality columns you filter by equality (user_id, order_id, UUIDs) and nowhere else. Size ndv near the distinct count and pick fpp around 0.01–0.05. They're the only pushdown mechanism that helps equality on high-cardinality columns — min/max and dictionaries both fail there.

Worked example — tuning row-group and page size

Detailed explanation. Row-group and page sizes trade compression and metadata overhead against skipping resolution and memory. Show the effect by writing the same data at different sizes and reasoning about the trade-offs.

• Large row groups. Fewer groups, better compression, coarser skipping, more memory per task.
• Small row groups. More groups, finer skipping, weaker compression, more footer metadata.
• Page size. Analogous trade-off one level down, and only pays for skipping with the page index on.

Question. Write the same table at row-group sizes 128k and 8k and compare row-group count and skipping resolution for a selective predicate.

Input.

Variant	row_group_size	Bias
coarse	128,000	compression/throughput
fine	8,000	skipping resolution

Code.

import pyarrow as pa
import pyarrow.parquet as pq
import os

n = 256_000
tbl = pa.table({
    "id":  pa.array(range(n), type=pa.int64()),          # sorted
    "val": pa.array([i * 3 % 1000 for i in range(n)], type=pa.int64()),
})

def write(path, rgs):
    pq.write_table(tbl, path, row_group_size=rgs,
                   write_statistics=True, write_page_index=True)
    md = pq.ParquetFile(path).metadata
    return md.num_row_groups, os.path.getsize(path)

for label, rgs in [("coarse", 128_000), ("fine", 8_000)]:
    groups, size = write(f"/tmp/{label}.parquet", rgs)
    # For predicate id in [100000,100100], how many row groups survive?
    md = pq.ParquetFile(f"/tmp/{label}.parquet").metadata
    survive = sum(1 for r in range(md.num_row_groups)
                  if not (md.row_group(r).column(0).statistics.max < 100_000
                          or md.row_group(r).column(0).statistics.min > 100_100))
    print(f"{label:7s} row_group_size={rgs:>7}  groups={groups:>3}  "
          f"file={size:>8}B  groups_read_for_100-row_filter={survive}")

Step-by-step explanation.

1. The coarse variant packs 256,000 rows into 2 row groups; the fine variant into 32. More groups means more footer metadata and slightly larger files, but finer skipping.
2. For the selective predicate id BETWEEN 100000 AND 100100 (101 rows) on sorted data, the coarse variant must read a whole 128,000-row group to reach those 101 rows — the surviving group is huge relative to the answer.
3. The fine variant confines the matching ids to one 8,000-row group, so the reader touches ~8,000 rows instead of ~128,000 — a much tighter read for the same query.
4. The trade-off runs the other way for compression and scan throughput: smaller groups compress a little worse and add per-group overhead, and very small groups hurt large full-scan queries and object-storage read efficiency.
5. With write_page_index=True, even the coarse variant can skip at page level inside its big group — so in practice you tune row-group size for memory/throughput and lean on the page index for fine skipping, rather than shrinking row groups drastically.

Output.

Variant	Row groups	Rows read for 101-row filter	Trade-off
coarse (128k)	2	~128,000 (or page-index pages)	best compression/throughput
fine (8k)	32	~8,000	finest row-group skipping

Rule of thumb. Default row groups to the ~128 MB target and make sure one fits comfortably in reader memory; don't shrink them drastically for skipping — enable the page index and sort instead. Reach for smaller row groups only when queries are extremely selective and the data is well sorted.

Worked example — the whole picture: a tuned write for a point-lookup + range workload

Detailed explanation. Real tables serve mixed workloads: range scans on a time column and point lookups on an id. A single tuned write can serve both — sort by the range column, bloom-filter the id column, enable the page index.

• Sort by event_ts → range predicates skip row groups via min/max.
• Bloom filter on user_id → equality point lookups skip row groups probabilistically.
• Page index + dictionary on categoricals → finer skipping and small categorical columns.

Question. Write one file tuned for both event_ts BETWEEN ... and user_id = ..., and confirm both predicates can skip.

Input.

Workload	Column	Mechanism
range	event_ts	sort + min/max + page index
point	user_id	bloom filter
categorical eq	country	dictionary filtering

Code.

import pyarrow as pa
import pyarrow.parquet as pq
import pyarrow.compute as pc
import random

random.seed(9)
n = 300_000
tbl = pa.table({
    "event_ts": pa.array([1_735_689_600 + random.randint(0, 30 * 86_400) for _ in range(n)], type=pa.int64()),
    "user_id":  pa.array([random.randint(1, 50_000_000) for _ in range(n)], type=pa.int64()),
    "country":  pa.array([random.choice(["US", "GB", "IN", "DE", "BR"]) for _ in range(n)]),
    "amount":   pa.array([round(random.random() * 300, 2) for _ in range(n)], type=pa.float64()),
})

# Sort by the range column so its row-group min/max become disjoint
sorted_tbl = tbl.take(pc.sort_indices(tbl, sort_keys=[("event_ts", "ascending")]))

pq.write_table(
    sorted_tbl, "/tmp/tuned.parquet",
    row_group_size=25_000,
    data_page_size=256 * 1024,
    use_dictionary=["country"],                              # categorical -> dict filtering
    compression={"event_ts": "zstd", "user_id": "zstd",
                 "country": "snappy", "amount": "zstd"},
    compression_level={"event_ts": 3, "user_id": 3, "amount": 3},
    write_statistics=True,
    write_page_index=True,                                   # page-level skipping
    bloom_filter_options={"user_id": {"ndv": n, "fpp": 0.01}},  # high-card equality
    sorting_columns=[pq.SortingColumn(0)],
)

md = pq.ParquetFile("/tmp/tuned.parquet").metadata
print("row groups:", md.num_row_groups)
# range predicate skipping on event_ts
lo, hi = 1_735_689_600, 1_735_776_000
skip = sum(1 for r in range(md.num_row_groups)
           if md.row_group(r).column(0).statistics.max < lo
           or md.row_group(r).column(0).statistics.min > hi)
print(f"event_ts 1-day range: {skip}/{md.num_row_groups} row groups skippable via min/max")
# point lookup on user_id via bloom filter
print("absent user_id rows:", pq.read_table("/tmp/tuned.parquet", filters=[("user_id", "==", -1)]).num_rows)

Step-by-step explanation.

1. Sorting by event_ts before writing makes each row group cover a contiguous time block, so a one-day range predicate's min/max prunes most row groups — the range half of the workload is handled by statistics.
2. write_page_index=True adds page-level skipping inside the surviving time block, so even a narrow range within the matching row group reads only the relevant pages.
3. bloom_filter_options on user_id handles the point-lookup half: because user_id is high-cardinality and unsorted (the file is sorted by event_ts, not user_id), min/max is useless, but the bloom filter still lets an equality lookup skip chunks that definitely lack the id.
4. use_dictionary=["country"] keeps the categorical column tiny and enables dictionary filtering for country = X equality predicates — a third pushdown mechanism in the same file.
5. One write thus serves three predicate shapes: range on event_ts (min/max + page index), high-cardinality equality on user_id (bloom filter), and categorical equality on country (dictionary) — the complete pushdown toolkit applied deliberately per column.

Output.

Predicate shape	Column	Skipping mechanism
range	event_ts	sort + min/max + page index
high-card equality	user_id	bloom filter
categorical equality	country	dictionary filtering

Rule of thumb. Design the write path around the read path: sort by the dominant range column, bloom-filter the high-cardinality equality columns, keep categoricals dictionary-encoded, and enable the page index. Each pushdown mechanism covers a different predicate shape; a well-tuned table uses the right one per column.

Common beginner mistakes

• Adding bloom filters to every column. They cost space and only help high-cardinality equality; on low-cardinality or range-filtered columns they're pure overhead.
• Expecting bloom filters to help ranges. They answer equality membership only; ranges are a min/max job.
• Making row groups tiny for skipping. You lose compression and add overhead; enable the page index and sort instead of shrinking row groups drastically.
• Sizing ndv wrong. Too small over-fills the filter and spikes false positives; set it near the per-chunk distinct count.

Interview question on bloom filters and tuning

A senior interviewer might ask: "You have a 2-billion-row Parquet table on S3. One workload filters by event_date ranges; another does point lookups by user_id. Range queries are fast, but user_id = X lookups scan the whole table. Explain why, and give the exact write-side changes — encoding, statistics, bloom filters, row-group and page sizing — that make both workloads fast without a separate index system."

Solution Using a bloom filter on the id, sorting on the range column, and tuned sizing

import pyarrow as pa
import pyarrow.parquet as pq
import pyarrow.compute as pc
import random

random.seed(4)
n = 400_000
tbl = pa.table({
    "event_date": pa.array([20_260_101 + (i % 30) for i in range(n)], type=pa.int32()),
    "user_id":    pa.array([random.randint(1, 80_000_000) for _ in range(n)], type=pa.int64()),
    "amount":     pa.array([round(random.random() * 500, 2) for _ in range(n)], type=pa.float64()),
})

# Sort by the range column; bloom-filter the point-lookup column
sorted_tbl = tbl.take(pc.sort_indices(tbl, sort_keys=[("event_date", "ascending")]))

pq.write_table(
    sorted_tbl, "/tmp/twoworkloads.parquet",
    row_group_size=25_000,           # fits reader memory; fine enough for date skipping
    data_page_size=256 * 1024,
    write_statistics=True,           # min/max for event_date range skipping
    write_page_index=True,           # page-level skipping inside a date block
    bloom_filter_options={"user_id": {"ndv": n, "fpp": 0.01}},  # high-card equality
    compression="zstd", compression_level=3,
    sorting_columns=[pq.SortingColumn(0)],
)

md = pq.ParquetFile("/tmp/twoworkloads.parquet").metadata
# range: event_date = 20260115 -> how many row groups skippable?
day = 20_260_115
range_skip = sum(1 for r in range(md.num_row_groups)
                 if md.row_group(r).column(0).statistics.max < day
                 or md.row_group(r).column(0).statistics.min > day)
print(f"date range skipping : {range_skip}/{md.num_row_groups} row groups")
# point: absent user_id -> bloom filter skips everything
print("absent user lookup  :", pq.read_table("/tmp/twoworkloads.parquet",
                                              filters=[("user_id", "==", -7)]).num_rows, "rows")

Step-by-step trace.

Concern	Before	After
event_date range	fast (already sorted-ish)	fast: min/max + page index
user_id = X lookup	full-table scan	bloom filter skips absent chunks
Why lookup was slow	high-cardinality, unsorted → min/max useless, no dictionary	bloom filter added
Row-group size	too large / unset	25k: fits memory, decent skipping
Codec	default	Zstd-3 for storage/I/O

The date range was already skippable via min/max because the file is sorted by event_date. The user_id lookup was slow because high-cardinality equality can't use min/max (ranges overlap) and there's no dictionary — a bloom filter is the only mechanism that helps, and adding one makes absent-id lookups skip nearly every chunk.

Output:

Workload	Mechanism	Result
event_date range	sort + min/max + page index	few row groups read
user_id equality	bloom filter (fpp 0.01)	absent → skip all; present → read few
index system needed	none	pushdown does it in-file

Why this works — concept by concept:

• Bloom filter — a per-chunk probabilistic set that answers equality membership with no false negatives, the only pushdown that helps high-cardinality equality where min/max ranges overlap and no dictionary exists.
• Sort on the range column — makes event_date row-group min/max disjoint so range predicates prune row groups; a file can only be physically sorted by one key, so pick the dominant range column and bloom-filter the other.
• Page index — ColumnIndex/OffsetIndex give page-level skipping inside the surviving date block, tightening the range read further.
• Right-sized row groups — 25k rows fits reader memory and gives decent date skipping without wrecking compression; the page index handles finer resolution.
• Cost — bloom filters add a bounded per-chunk bitset (sized by ndv/fpp); sorting is a one-time write cost. Together they serve both a range and a point-lookup workload from one file with no external index, turning two full scans into two targeted reads.

Indexing
Topic — indexing
Indexing problems on bloom filters and lookups

Practice →

Optimization
Topic — optimization
Optimization problems on Parquet tuning

Practice →

---

Cheat sheet — Parquet internals recipes

• The four-level hierarchy. File (PAR1 magic at both ends + thrift FileMetaData footer) → row group (horizontal row slice, unit of parallelism + coarse skipping, target ~128 MB) → column chunk (one column per row group, unit of projection, carries ColumnMetaData + statistics) → page (dictionary page + data pages, ~1 MiB, unit of decode + page-index skipping). Every performance question maps to one of these four levels; name the level and the answer follows.
• Footer-first read path. Read last 8 bytes = int32 footer length + PAR1; seek to filesize - 8 - footer_len; decode thrift FileMetaData (schema + per-chunk offsets, sizes, codec, encodings, statistics); plan projection + predicate; issue ranged reads for surviving column chunks. Planning is O(footer); a projected query on S3 is ~2–3 ranged reads, not a download.
• Encoding vs compression — two stages. Stage 1 encoding is structure-aware per column: RLE_DICTIONARY (low cardinality), DELTA_BINARY_PACKED (monotonic ints/timestamps), DELTA_BYTE_ARRAY (sorted strings), BYTE_STREAM_SPLIT (floats), PLAIN (fallback). Stage 2 compression is byte-level on the encoded output: Snappy (fast/hot), Zstd (ratio/warm-cold, tunable level), Gzip (legacy), LZ4 (latency). Fix encoding first — it's the bigger, cheaper win — then dial the codec to read frequency.
• pyarrow write recipe. pq.write_table(tbl, path, row_group_size=..., data_page_size=..., use_dictionary=[...], column_encoding=..., use_byte_stream_split=[...], compression=..., compression_level=..., write_statistics=True, write_page_index=True, bloom_filter_options=..., sorting_columns=[pq.SortingColumn(idx)]) — pass column_encoding, compression, compression_level, and bloom_filter_options as per-column dicts. compression_level only applies to codecs that support it (Zstd/Gzip, not Snappy) — pass it as a per-column dict to avoid the "Snappy doesn't support level" error.
• Projection pushdown. Read only referenced columns: pq.read_table(path, columns=[...]) or the dataset API's columns=. It's free (pure addressing via footer offsets) and usually the biggest single lever — never SELECT * on a wide table when you need a few columns.
• Predicate pushdown levels. (1) Row-group skipping via per-chunk min/max/null_count — only works when the predicate column is sorted/clustered so ranges are disjoint. (2) Page-level skipping via the page index (ColumnIndex per-page min/max + OffsetIndex per-page offset/row range) — enable with write_page_index=True. (3) Dictionary filtering — equality on a dictionary-encoded column skips a chunk when the literal isn't in its dictionary page. (4) Bloom filters — high-cardinality equality.
• Sorting is the pushdown multiplier. Statistics are written automatically but only pay off when min/max ranges are disjoint. Sort (or cluster/partition) each table by its dominant range/join column before writing, and declare it via sorting_columns. Random write order makes every row group's min/max span the whole domain → zero skipping.
• Bloom filter rule. Add per-chunk bloom filters only to high-cardinality columns you filter by equality (user_id, order_id, UUID, email). Size ndv near the per-chunk distinct count (row count is a safe default), pick fpp in 0.01–0.05. No false negatives (always safe to skip), some false positives (bounded by fpp). Useless for ranges and wasteful on low-cardinality columns.
• Row-group sizing. Target ~128 MB; the hard constraint is that one row group must fit comfortably in a reader task's memory. Bigger → better compression, fewer footer entries, coarser skipping, more memory. Smaller → finer skipping, weaker compression, more overhead. Prefer the page index over drastically shrinking row groups.
• Page sizing. Default ~1 MiB data_page_size. Smaller pages give finer page-index skipping for selective queries at the cost of more page-header overhead; only worthwhile with write_page_index=True and sorted data so pages have tight ranges.
• Data page v1 vs v2. DATA_PAGE (v1) compresses the whole page including def/rep levels; DATA_PAGE_V2 stores levels uncompressed and separately, so a reader can evaluate nullability/nesting without decompressing values. Set via data_page_version="2.0" when the reader supports it.
• Small-files anti-pattern. Thousands of tiny Parquet files means thousands of footer round trips and poor per-file compression. Compact to files with a healthy number of ~128 MB row groups; the footer read is per-file overhead you pay regardless of how little data you want.
• Diagnosing a slow scan. Dump FileMetaData: how many row groups (parallelism/skipping resolution), codec + encoding per column (size), how tight per-chunk min/max ranges are (skippability), whether the page index and bloom filters exist. The metadata explains the slowness before you profile — usually it's SELECT *, unsorted data, or missing bloom filters on a point-lookup column.

Frequently asked questions

What is the Parquet file format in one sentence?

The parquet file format is an open-source, columnar, self-describing, immutable on-disk format for analytical data: it stores the values of each column contiguously (rather than each row contiguously), splits the data into row groups and per-column column chunks made of encoded-and-compressed pages, and writes a thrift-encoded FileMetaData footer holding the schema plus per-chunk byte offsets and statistics. That layout is what enables the three things people associate with Parquet — strong compression (homogeneous values encode well), projection pushdown (read only the columns you need), and predicate pushdown (skip row groups and pages whose statistics prove they can't match the filter). It's the default storage format for Spark, Trino, DuckDB, Snowflake external tables, and the data files inside Delta Lake and Apache Iceberg, which is why "explain Parquet internals" is a staple of senior data-engineering interviews.

Why is Parquet faster than CSV for analytics?

Three structural reasons, all consequences of columnar storage. First, projection: an analytical query touches a few columns out of many, and because each column is a contiguous column chunk, Parquet reads only those chunks while CSV must parse every field of every row. Second, encoding and compression: same-column values are homogeneous and locally similar, so dictionary encoding, run-length encoding, delta encoding, and a codec like Zstd shrink them far more than gzip-on-CSV ever could. Third, skipping: Parquet writes min/max column statistics per row group and per page, plus optional bloom filters, so a selective WHERE clause can skip most of the file without decompressing it — CSV has no metadata to skip with and must scan everything. Add the footer-first read path (plan the whole query from one metadata read) and a selective Parquet query reads a small fraction of the bytes a CSV scan would.

What are row groups, column chunks, and pages?

They are the three internal levels below the file. A row group is a horizontal slice of rows (target ~128 MB) and is the unit of read parallelism and coarse predicate skipping. Within a row group, a column chunk holds all of one column's data contiguously and is the unit of projection — it carries its own metadata (type, codec, encodings, byte offsets, and min/max/null statistics). Within a column chunk, data is split into pages (default ~1 MiB): an optional DICTIONARY_PAGE holding distinct values, followed by DATA_PAGEs holding definition/repetition levels and encoded values. Pages are the finest unit of decoding, and with the page index enabled they're also the finest unit of skipping. A useful shorthand: parallelism happens at the row group, projection at the column chunk, and decoding/fine-skipping at the page.

How does predicate pushdown work in Parquet?

predicate pushdown is the reader using file metadata to avoid reading data that can't match the filter, and it operates at several levels. The base level is row-group skipping: each column chunk stores min, max, and null_count, so for amount > 400 a row group whose amount max is 380 is provably empty of matches and skipped. The finer level is the page index (ColumnIndex + OffsetIndex, enabled with write_page_index=True), which stores per-page min/max and offsets so the reader skips at page granularity without scanning page headers. For equality on a low-cardinality column, dictionary filtering checks the dictionary page and skips the chunk if the value is absent. For equality on a high-cardinality column, a bloom filter answers membership. Crucially, min/max skipping only helps when the predicate column is sorted or clustered so ranges are disjoint — random write order defeats it entirely.

When should I use a bloom filter in Parquet?

Use a bloom filter for equality predicates on high-cardinality columns — user_id = 12345, order_id = ..., UUID or email lookups — where min/max statistics are useless (every row group's range spans the whole domain) and there's no small dictionary to check. A bloom filter is a compact per-column-chunk bitset that answers "is this value possibly in the chunk?" with no false negatives and a tunable false-positive rate (fpp), so the reader safely skips every chunk that definitely doesn't contain the value. Add them selectively: only on columns you actually filter by equality, sized with ndv near the per-chunk distinct count and fpp around 0.01–0.05. Don't add them to low-cardinality columns (dictionary filtering already handles those) or to columns you only range-filter (that's a min/max job) — a bloom filter on the wrong column is pure space overhead with no read benefit.

How do I choose row-group and page sizes?

Start from the constraint, then tune. Row-group size targets ~128 MB, and the hard rule is that one row group must fit comfortably in a reader task's memory — too large causes spills and OOMs, too small floods the footer with metadata and weakens compression. Bigger row groups compress better and parallelize coarser; smaller ones skip finer. For most tables the ~128 MB default is right, and you get fine skipping from the page index rather than by shrinking row groups. Page size defaults to ~1 MiB (data_page_size); smaller pages give finer page-index skipping for very selective queries at the cost of more page-header overhead, and only pay off when write_page_index=True and the data is sorted so pages have tight, disjoint ranges. The meta-rule: size row groups for memory and throughput, enable the page index for fine skipping, and sort by the dominant filter column so both statistics and page ranges become useful.

Practice on PipeCode

• Drill the optimization practice library → for the columnar scan-cost, projection, encoding, and predicate-pushdown problems senior interviewers use to probe whether you actually understand why Parquet is fast.
• Rehearse on the ETL practice library → for the write-path tuning, sorting, compaction, and small-files problems that decide whether your pipeline's Parquet output is skippable or a full-scan trap.
• Sharpen the storage layer with the file-io practice library → for the file-format internals, row-group/page structure, and metadata-reading exercises that map directly onto the four-level hierarchy.
• Layer in the indexing practice library → for the min/max statistics, page index, and bloom-filter membership problems that separate a candidate who says "Parquet has an index" from one who can name each skipping mechanism and the predicate shape it covers.
• Stack the prerequisites against PipeCode's broader 450+ data-engineering catalogue to anchor the Parquet read-cost model against real graded inputs.

Lock in Parquet internals muscle memory

Docs describe the format. PipeCode drills explain the decision — when projection is your biggest lever, when sorting turns min/max statistics into skipped bytes, when a bloom filter beats a min/max range, when the page index earns its footer space. Pipecode.ai is Leetcode for Data Engineering — internals-first practice tuned for the storage-layer trade-offs senior data engineers actually face.

Practice optimization problems →
Practice file-io problems →