TL;DR — Columnar engines like Arrow, DuckDB, Polars, and GPU dataframes are built for scan-heavy, amortized access — ideal for building training sets, terrible for single-row inference lookups. The real value of Arrow in AI systems isn't query speed, it's zero-copy interchange between the offline columnar world and the online row-oriented world. Teams that try to make one engine do both pay for it in p99 latency.
Every few months another benchmark lands showing DuckDB or Polars or a GPU dataframe engine crushing some TPC-H-style workload. The numbers are real. The conclusion people draw from them — that the columnar engine is now the obvious backbone for AI data infrastructure, end to end — is where things go wrong.
Columnar engines are extraordinary at one thing: scanning a lot of rows to compute a few aggregates. They are structurally bad at the thing most AI inference paths actually need, which is fetching one row, fast, under load. Those are not the same problem, and the industry keeps building systems that quietly assume they are.
What "columnar" is actually optimizing for
Column stores exist because analytical queries touch a small number of columns across a huge number of rows. Storing data column-wise means you only read the bytes you need, you compress similar values together, and you can vectorize operations across contiguous memory with almost no branching. This is why Arrow's columnar memory format, DuckDB's execution engine, and GPU dataframe libraries all converge on the same core bet: amortize a fixed overhead over millions of values.
That bet is exactly right for building a training set. You're joining a fact table of events against a dimension table of user attributes, computing rolling aggregates, filtering, deduplicating — all scan-and-reduce operations over large batches. A columnar engine will do this an order of magnitude faster than a row-oriented database, and GPU-accelerated columnar execution pushes that further by parallelizing the same vectorized operations across thousands of cores.
None of this has anything to do with what happens at inference time.
Where the model breaks
An inference request — a chat turn, an agent step, a ranking call — needs a handful of specific rows: this user's current feature vector, this document's embedding, this session's running state. That's a point lookup, not a scan. In a column store, reconstructing a single logical row means touching N separate column chunks, each potentially compressed, each potentially on a different page or file. You pay that reconstruction cost whether you need one row or a million. Over a million rows, it disappears into the vectorized throughput. Over one row, at p99, under concurrent load, it's pure latency with no amortization to hide behind.
This is why nobody serves a production recommendation feature store out of a Parquet file scanned fresh every request. It's why vector databases don't store vectors in a generic columnar table and brute-force scan them — they build HNSW or IVF indexes specifically because the access pattern is "find me the nearest neighbors to this one query vector right now," which is the opposite of what columnar layouts are good at. The vector database world already learned this lesson. A lot of the broader AI data stack hasn't internalized it yet.
You see the same mistake show up in agent systems. An agent step that needs to read conversation state, tool outputs, and a slice of long-term memory is doing dozens of small, latency-sensitive point reads per turn. Bolt a columnar analytics engine underneath that and you've built a system that is blazingly fast at answering "how many agent runs happened last week" and painfully slow at answering "what did this specific agent remember five turns ago."
Arrow's real contribution isn't the query engine
This is where Arrow actually earns its place, and it's not the part most benchmarks celebrate. Arrow's contribution to AI infrastructure is a shared, zero-copy, in-memory columnar layout that multiple engines can read and write without serializing between them. Pandas, Polars, DuckDB, Spark, a PyTorch dataloader, a GPU dataframe — all of them can hand off the same block of memory without a conversion tax at every boundary.
That matters enormously for the offline side of an AI pipeline, where data moves through five or six tools on its way from raw logs to a training batch. Every serialization boundary you remove is real wall-clock time and real infrastructure cost returned to you, independent of how fast any single engine's query planner is. The headline-grabbing part of the Arrow ecosystem is "look how fast this SQL query runs." The quietly load-bearing part is "look how many copy operations this eliminated between tools nobody thought to combine."
Conflating those two contributions is the mistake. Teams read "Arrow is fast" and "DuckDB is fast" and conclude the columnar stack should extend all the way to the serving boundary. It shouldn't, and trying to force it there is how you end up with an inference path that's fast in a load test with batch size 10,000 and falls apart in production where real traffic looks like batch size one, repeated constantly, with a latency budget measured in milliseconds.
Designing for two speeds, not one
The architecture that actually works treats this as two distinct problems joined at a well-defined seam. Offline, use the columnar, GPU-accelerated stack for exactly what it's good at: building and refreshing training datasets, computing batch feature transforms, backfilling, running analytical evaluation over logged inference traffic. This is where Arrow-based engines and GPU dataframes deliver real, repeatable wins, because the workload genuinely is scan-and-reduce over large volumes.
Online, use a row-oriented store, a key-value layer, or a purpose-built index — whatever gives you single-digit-millisecond point lookups at your concurrency target. The feature store pattern of an offline columnar warehouse paired with a low-latency online store isn't legacy cruft to be modernized away by a faster columnar engine; it's the correct shape of the problem, and GPU acceleration on the columnar side doesn't change which side of the seam a given workload belongs on.
The seam itself is where Arrow should live — as the interchange format that lets your offline columnar pipeline hand off a batch of freshly computed features to whatever is populating the online store, without a serialization detour through JSON or a bespoke binary format. That's a real, meaningful win. It's just a different win than "
Top comments (0)