Choose a Columnar Format From the Read Path Backward

The columnar-format ecosystem is moving quickly. Feature tables encourage teams to ask which format has the newest encoding. A system-design decision starts elsewhere: which bytes must each workload read, how often does data change, and what happens when a writer fails?

Declare the workload

dataset:
  logical_size_gib: 500
  rows: 4000000000
  daily_append_gib: 12
queries:
  projection: 6_of_80_columns
  selectivity_p50: 0.02
  selectivity_p99: 0.60
  concurrency: 24
storage:
  object_store_rtt_ms: 35
updates:
  corrections_per_day: 200000

Without projection, selectivity, distribution, concurrency, object-store latency, update pattern, and engine versions, a compression or scan result does not transfer.

Model total query cost

query_time ≈ metadata_requests × request_latency
           + bytes_read / storage_throughput
           + bytes_decompressed / decode_throughput
           + filter_and_materialization_cpu

Smaller files may reduce wasted reads but multiply object-store requests and metadata. Large row groups can compress well while forcing broad scans. Sophisticated indexes may accelerate selective queries and increase write amplification, memory, or compatibility risk.

Benchmark with the actual engines that read and write the data. Pin format library, compression codec, row-group/page settings, object-store region, cache state, and instance type. Run cold and warm cases. Report p50 and p99 latency, bytes fetched, requests, CPU time, peak memory, encoded size, and write cost.

Test change and failure

A read benchmark is incomplete when the dataset receives corrections. Test append, update/delete representation, compaction, schema evolution, partial writer failure, orphan cleanup, corrupted metadata, reader compatibility during upgrade, and rollback.

Separate file format from table format. A table layer may add snapshots, transactions, catalogs, and change tracking around underlying files. Evaluate the combined read/write/recovery path your platform will actually operate.

The public MonkeyCode repository describes AI task and development-environment workflows that can generate operational and evaluation data. Columnar storage may eventually serve such analytics, but this article makes no claim about MonkeyCode's storage architecture or performance.

Disclosure: I contribute to the MonkeyCode project. The product connection is contextual; the workload model is independent.

Choose the format that wins your declared read, write, interoperability, and recovery workload—not the format with the longest feature list.