Every article about ORC says the same thing.
"Columnar storage, built-in indexes, predicate pushdown - so it's fast."
That's not an explanation. That's a slogan.
So I wrote a small ORC parser from scratch, generated 5 million rows, and opened the file with my hands.
The question I actually wanted answered:
What happens inside an ORC file when a query asks for 2 columns out of 100?
What a Columnar File Actually Does
Strip away the marketing, and it comes down to three things.
- Store each column's values together, so you can read one column without touching the others.
- Encode those values in ways that only work when neighbours are similar.
- Keep a small map of what's inside, so a query can skip most of the file without opening it.
Everything else - compression codecs, bloom filters, ACID support - is built around that core job.
The Setup
An orders table. Deliberately boring.
order_id int64 sequential
customer_id int64 250k distinct
country string 10 distinct
product string 2,000 distinct
quantity int32 1-19
price double 40k distinct
order_date date 730 days
status string 5 distinct
5,000,000 rows, written with PyArrow:
orc.write_table(
tbl, "orders.orc",
compression="zlib",
stripe_size=8 * 1024 * 1024, # small on purpose
row_index_stride=10_000,
dictionary_key_size_threshold=0.8,
)
Production ORC uses 64-256 MB stripes. My whole file is 53 MB, so a default stripe would swallow the entire dataset and there'd be nothing to skip. Shrinking it is how you see multi-stripe behaviour on a laptop.
Same data also went out as a 310 MB CSV, for scale.
ORC Is Read Backwards
The last byte of the file is a single number: the length of the PostScript.
Here are the final 26 bytes of my file:
08 c0 03 10 01 18 80 80 04 22 02 00 0c 28 97 05 30 06 82 f4 03 03 4f 52 43 19
That's a Protocol Buffers message. Decoded:
| field | value | meaning |
|---|---|---|
| 1 | 448 | footer length |
| 2 | 1 | codec = ZLIB |
| 3 | 65536 | compression block size |
| 5 | 663 | metadata length |
So a query engine does three reads: last byte, PostScript, then footer + metadata. A few hundred bytes on a 53 MB file.
After that it knows the schema, the row count, every stripe's byte offset, and min/max stats for every column.
It hasn't touched a single row of data yet.
Note the ordering: the codec lives in the PostScript, and the PostScript is never compressed. It can't be - you'd need the codec to decompress the thing that tells you the codec.
Stripes
A stripe is a self-contained slab of rows. Its own index, its own data, its own footer.
# offset index data footer rows
0 3 8,412 8,544,070 159 687,104
1 8,552,644 8,487 8,556,939 158 688,128
...
7 59,917,580 2,796 2,315,836 152 186,176
8,412 bytes of index for 8.5 MB of data. 0.1% overhead for the ability to skip.
That ratio is the entire trade.
Because each stripe carries its own footer, you can hand a Spark executor one byte range and it can decode that stripe with zero knowledge of the rest of the file. That's the basis of parallel scans.
Where a Column Physically Lives
Inside a stripe, a column isn't "a column." It's a set of streams.
col name stream bytes
3 country DICTIONARY_DATA 23
3 country LENGTH 5
3 country DATA 310,404
6 price DATA 2,374,052
All streams for one column sit contiguously. "Read country from stripe 3" is one range read, not 687,104 scattered seeks.
There's also a PRESENT stream for nulls - and ORC omits it entirely when a column has none. My columns are all non-null, so null tracking cost zero bytes. A row format pays for nullability on every row whether you use it or not.
Encodings Do More Than Compression Does
People credit ORC's size to compression. Mostly wrong.
Bytes per column, whole file:
| column | bytes | bytes/row |
|---|---|---|
| order_id | 11,541 | 0.002 |
| price | 17,022,257 | 3.404 |
| customer_id | 13,293,357 | 2.659 |
Five million 64-bit integers in 11 KB.
That's ORC's RLE v2 picking its DELTA sub-encoding: for a monotonic sequence it stores a base, a delta width, and a run length. zlib never saw 40 MB of integers, because the encoder never produced them.
Same effect on sorted dates:
order_date, 5M values:
random order : 7,390,184 bytes
sorted : 4,898 bytes <- 1,500x smaller
The PyArrow gotcha
dictionary_key_size_threshold defaults to 0.0, which disables dictionary encoding entirely.
| DIRECT (default) | DICTIONARY | |
|---|---|---|
| status | 796,196 B/stripe | 271,047 B/stripe |
| whole file | 62.24 MB | 53.32 MB |
14% of the file, from one keyword argument.
Compression Sits On Top
Only after encoding does the codec run - in independent 64 KB chunks, each with a 3-byte header. Bit 0 says "stored raw", because if compressing a chunk made it bigger, ORC just doesn't.
Chunking is what makes seeking possible. One big compressed blob would mean inflating 8 MB to read the last value.
| codec | file MB | vs CSV | write s | full scan s |
|---|---|---|---|---|
| uncompressed | 86.14 | 3.6x | 2.15 | 1.73 |
| snappy | 69.49 | 4.5x | 2.35 | 0.51 |
| zlib | 53.32 | 5.8x | 4.45 | 1.02 |
| zstd | 51.72 | 6.0x | 2.34 | 0.53 |
zstd wins on all three axes. zlib is only still the default because ORC is old.
Also worth noting: uncompressed ORC is still 3.6x smaller than CSV. That gap is pure layout and encoding.
I Actually Measured the Skipping
ORC keeps min/max statistics at three zoom levels: file (448 B), stripe (663 B), and row group of 10,000 rows (8.4 KB per stripe).
Query:
SELECT price FROM orders
WHERE order_date BETWEEN '2024-06-01' AND '2024-06-30'
One month out of 24. About 4% of rows. Same query, same data, two files - one in insertion order, one sorted by order_date.
Unsorted:
stripe 0 2023-01-01 .. 2024-12-30 READ
stripe 1 2023-01-01 .. 2024-12-30 READ
...
stripes to read: 8/8
bytes read: 24,430,256
Every stripe spans the whole date range, so every min/max says "maybe". Zero pruning.
Sorted by order_date:
stripe 0 2023-01-01 .. 2023-04-10 SKIP
stripe 4 2024-02-02 .. 2024-05-11 SKIP
stripe 5 2024-05-11 .. 2024-08-19 READ
stripe 6 2024-08-19 .. 2024-11-26 SKIP
...
stripes to read: 1/8
bytes read: 2,335,435 <- 13.7% of the unsorted read
And inside that one surviving stripe, the row-group index eliminated another 68%: 22 of 69 row groups matched. The engine ends up decoding roughly 220,000 of 5,000,000 rows.
The uncomfortable conclusion:
ORC's statistics are a summary, not an index. Their usefulness is entirely determined by how you wrote the data.
Sorting by your dominant filter column isn't a tuning detail. It's the difference between skipping 87% of the file and skipping nothing.
A predicate stats can't help with
country = 'IN'
stripe ranges: ('AU','US'), ('AU','US'), ('AU','US'), ...
stripes that could contain 'IN': 8/8
Scattered values mean every range straddles the target. That's the gap bloom filters fill - they cost 1.4% of file size on product, and are a waste on country.
Does Column Projection Actually Pay Off?
I wrapped the file handle in a counter so I could see every byte the reader actually pulled off disk.
8 columns, 53 MB:
| query | bytes read | % of file |
|---|---|---|
SELECT * |
53,279,130 | 99.93% |
SELECT country, price |
19,265,356 | 36.13% |
SELECT order_id |
29,356 | 0.06% |
100 columns, 1M rows, 153 MB - the original question:
| query | bytes read | % of file | time |
|---|---|---|---|
SELECT * |
153,188,271 | 99.92% | 3.03s |
SELECT order_id, price |
3,424,697 | 2.23% | 0.04s |
2 columns out of 100 = 2.23% of the bytes. Not "less I/O" in a hand-wavy sense. 45x less, measured.
Reading 2 columns from the 310 MB CSV still requires pulling all 310 MB off disk and parsing every field to find the commas.
How Big Should a Stripe Be?
Same sorted data, same June predicate:
| stripe target | stripes | rows scanned | % rows |
|---|---|---|---|
| 2 MB | 29 | 350,208 | 7.0% |
| 8 MB | 8 | 680,960 | 13.6% |
| 32 MB | 2 | 2,294,592 | 45.9% |
| 64 MB | 1 | 5,000,000 | 100.0% |
Halve the stripe, halve the rows you're forced to scan - until pruning bottoms out at the real selectivity. Total file size moved 0.3% across that whole range, because more stripes cost more metadata, not more data.
So why not 2 MB stripes everywhere? Because a stripe is also the unit of parallelism, and on object storage each one is a separate GET. The row-group index already gives you 10,000-row granularity inside a big stripe - that's the layer meant to do fine pruning. Keep stripes big.
What I Couldn't Test
- A real cluster. Everything here is one machine, local NVMe. On S3, request count matters more than byte count, and my numbers say nothing about that.
- lz4 in this build. It produced 86.02 MB - essentially identical to uncompressed. Either the PyArrow build isn't wiring it up or something else is wrong. I'm reporting it rather than quietly dropping the row, but don't trust that number.
- Bloom filter effectiveness. I measured what they cost (1.4% of file size), not what they save, because PyArrow's reader doesn't expose filter pushdown for ORC. You'd need Hive or Trino for that.
Where This Doesn't Help
Columnar layout is a bet, and it loses on some workloads.
- Not for OLTP. Inserting one row means rewriting a stripe. Fetching one full record means touching every column's streams separately.
- Small files kill it. All this metadata machinery amortizes over large files. A thousand 2 MB ORC files is the classic way to make a fast format slow.
One Thing a Different Columnar Format Does Differently
ORC isn't the only format built on these ideas - Parquet uses the same stripe/row-group/statistics playbook. The clearest place they diverge is encoding scope.
Same data, same codec, same row grouping:
order_id (sequential) ORC 7.8 KB vs Parquet 7.2 MB (912x)
price (40k distinct) ORC 16.3 MB vs Parquet 10.2 MB (0.62x)
ORC's dictionary encoding applies only to string columns - a DOUBLE always goes out DIRECT, full width, and only the compressor gets a shot at it. Parquet dictionary-encodes any physical type, so a double with modest cardinality compresses to narrow indices where ORC can't.
That's a real, measured trade-off, not a verdict. It's also its own rabbit hole - I measured it properly in a follow-up post.
So, Why Is It Fast?
Not one reason. A chain, where each link multiplies the next.
columnar layout -> read 2 columns, not 100 (45x fewer bytes)
x encodings -> delta/dictionary before codec (up to 3,467x on one column)
x compression -> chunked, seekable (~2x on top)
x statistics -> skip stripes that can't match (8x when sorted)
x row-group index -> skip 10k-row blocks inside those (3x more)
---------------------------------------------------------------
= read 2.2% of the file, decode 4% of the rows you touched
Every layer is doing the same thing from a different angle: making it possible to not read something.
ORC isn't fast because it's clever with the bytes it reads. It's fast because it spends 0.1% of the file on a map detailed enough to avoid reading almost all of them.
And the part that's actually in your control: sorting by your filter column moved my query from 24 MB to 2.3 MB. No format setting in this entire post came close to that.
Top comments (0)