PyArrow, Explained: How Zero-Copy Actually Works, and Why It's Everywhere

Every time data moves between two tools in a modern Python data stack — a DuckDB query result becoming a pandas DataFrame, a Polars frame getting handed to a PySpark job, a warehouse client returning query results to a notebook — something has to decide how that data is laid out in memory on both ends, and whether crossing that boundary means copying it. Most of the time, historically, it did: pickle it, walk every value, rebuild it on the other side. Apache Arrow exists to make that copy unnecessary, and PyArrow is the Python implementation of it. This article is about what Arrow actually is, the specific mechanism that makes "zero-copy" a literal, checkable claim rather than a marketing phrase, and where it shows up across a real data engineering stack — including a real benchmark, not an assumed one.

What Arrow actually is

If you've read the companion piece on Parquet, you already have the right instinct for this: Parquet is a file format, specifying how columnar data should be laid out on disk. Apache Arrow is a memory format — a precise specification for how columnar data should be laid out in RAM, inside a running process. Different problem, same underlying idea: lay data out by column, not by row, because it's faster to scan and compress that way.

The detail that makes Arrow a genuinely different thing from Parquet isn't cross-language support — Parquet has that too, just as much as Arrow does. It's what each format is actually optimized for:

• Parquet trades compute-readiness for compact storage. It's compressed and encoded (dictionary encoding, run-length encoding, general-purpose compression) to take up as little space as possible on disk. You can't compute on it directly — it has to be decoded first, every time, by whatever's reading it.
• Arrow trades storage compactness for direct computability. It's already the decoded, ready-to-compute-on layout — uncompressed, fixed-width buffers a CPU can operate on directly, with no decode step standing between "here's the memory" and "here's usable data."

That's specifically what makes sharing Arrow between processes free, in a way sharing Parquet isn't: there's no decode step to pay on the way in.

The actual memory layout — buffers, not objects

Here's the detail that explains almost everything else in this article. A Python list of strings, or a NumPy array with dtype=object (which is what pandas falls back to for strings), doesn't actually store the strings contiguously. It stores an array of pointers, each one referencing a separate Python string object sitting wherever the memory allocator happened to put it:

An Arrow array of the same three strings is built from a small, fixed number of contiguous memory buffers instead:

• A validity bitmap — one bit per value, marking nulls. (Arrow's null handling is a bitmap sitting alongside the data, not a sentinel value mixed into it — this is also why Arrow's nullable-integer support is cleaner than NumPy's, which has no way to represent a null integer without upcasting the whole column to float.)
• An offsets buffer — for variable-length types like strings, an array of integers marking where each value starts and ends inside the values buffer.
• A values buffer — the actual bytes, laid out back-to-back, with no per-value object overhead and no pointer to chase to reach the next one.

This matters for two independent reasons, and it's worth keeping them separate:

1. It's faster to compute over, even within one process. Contiguous, uniformly-typed memory is exactly what lets a CPU vectorize (process several values per instruction) and stay cache-friendly (reading value n+1 doesn't mean jumping somewhere else in RAM the way following a pointer does). This is the same columnar-layout argument the Parquet article makes for reading off disk, just one level closer to the CPU.
2. It's what makes the memory shareable between processes and languages at all. A Python object array is full of language-specific state (Python's own object headers, reference counts) that only Python's runtime understands. A block of Arrow buffers is just bytes with a schema — nothing in it is specific to any language's runtime, which is the precondition for the next section.

Why it's called zero-copy — the actual mechanism

"Zero-copy" gets thrown around loosely enough that it's worth being precise about what specifically makes it true. The mechanism is called the Arrow C Data Interface, and it's smaller than you'd expect: a handful of plain C struct definitions (ArrowArray, ArrowSchema, and a streaming variant ArrowArrayStream) that any project can copy directly into its own source tree. There's no library to link against and no shared build dependency — two completely independently-compiled programs, in two different languages, can exchange an Arrow array at runtime just by agreeing on the layout of these structs and passing a pointer to one. Non-C/C++ languages participate through their own FFI layer — Python via ctypes/cffi, similarly for Rust, Go, and Julia.

Compare that to what pickle, JSON, or protobuf actually do: walk every value in the source structure, transform it into a different byte layout entirely, send that stream across, and have the receiving side parse it back into a new, freshly-allocated structure. That's real, unavoidable work that scales with the size of the data. Arrow's C Data Interface skips all of it — when two Arrow-aware libraries hand data to each other within the same process, they're not converting anything; they're agreeing to both look at the same buffers.

One nuance worth having precisely right: this specific mechanism is for sharing memory within a single process — the pointer only means something as long as both sides can address the same memory. The moment data genuinely needs to leave the process (over a network, to disk, to a separate machine), Arrow uses a different piece of the spec, the Arrow IPC format — but even that is barely a serialization step by the standards of pickle or JSON: IPC is close to writing the same in-memory buffers out sequentially with a thin framing header, not reshaping the data into a different structure. Feather files are just Arrow IPC written to disk. Arrow Flight, covered below, is Arrow IPC streamed over gRPC.

Proving it, not just asserting it

Rather than take the "zero-copy is fast" claim on faith, here's a real, reproducible comparison: handing a table from DuckDB to Polars via Arrow, versus a genuine serialization roundtrip (materializing rows, json.dumps, json.loads, then rebuilding a DataFrame) — timed at increasing row counts.

import time, json
import duckdb
import polars as pl

def bench_zero_copy(n_rows):
    con = duckdb.connect()
    con.execute(f"""
        CREATE TABLE t AS
        SELECT i AS id, (i % 100) AS bucket, CAST(i AS DOUBLE) * 1.5 AS amount
        FROM range({n_rows}) t(i)
    """)
    start = time.perf_counter()
    arrow_table = con.execute("SELECT * FROM t").arrow()   # DuckDB -> Arrow
    df = pl.from_arrow(arrow_table)                          # Arrow -> Polars
    return time.perf_counter() - start

def bench_serialize_roundtrip(n_rows):
    con = duckdb.connect()
    con.execute(f"""
        CREATE TABLE t AS
        SELECT i AS id, (i % 100) AS bucket, CAST(i AS DOUBLE) * 1.5 AS amount
        FROM range({n_rows}) t(i)
    """)
    start = time.perf_counter()
    rows = con.execute("SELECT * FROM t").fetchall()
    parsed = json.loads(json.dumps(rows))                   # the actual reformatting work
    df = pl.DataFrame(parsed, schema=["id", "bucket", "amount"], orient="row")
    return time.perf_counter() - start

Run on this machine — after one warm-up call to absorb import/connection-setup costs, and taking the median of 5 runs at each row count, since a single cold run swings around enough to be misleading on its own:

Rows	Zero-copy median (s)	Serialize roundtrip median (s)	Ratio
10,000	0.0009	0.0138	15x
100,000	0.0038	0.1428	38x
1,000,000	0.0133	2.0589	154x
5,000,000	0.1698	11.7037	69x

Two honest notes on this table, since fabricated-looking precision is worse than useful imprecision: these are wall-clock numbers from one machine under median-of-5 conditions, so absolute values — and even the exact ratio at each row count, which bounces around here rather than climbing smoothly — will vary with hardware, load, and how DuckDB's query planner happens to behave at each size. Don't treat 15x/38x/154x/69x as a formula. What's consistent and worth trusting is the shape: the zero-copy path stays under two-tenths of a second even at 5 million rows, while the serialization path grows to nearly 12 seconds over the same range, because it's doing real per-value work — building a byte stream, then rebuilding objects from it — that the zero-copy path simply never does.

Where this shows up in Spark and Databricks

This is the part that's easy to undersell as "Arrow makes Spark faster" without saying how. A plain Python UDF in PySpark serializes data row-by-row across the JVM↔Python process boundary — real, per-row overhead, the same category of cost as the JSON roundtrip above. Pandas UDFs (vectorized UDFs) fix this by batching rows and moving them across that boundary as Arrow data instead, which is where the commonly-cited "up to 100x faster than row-at-a-time UDFs" figure comes from — it's the same mechanism this article just demonstrated, applied specifically to the JVM/Python boundary.

Even Pandas UDFs still pay one small cost worth knowing about: converting an Arrow batch into a pandas Series isn't always free, particularly around null handling, so there's a real Arrow→pandas conversion step in between. Databricks introduced a further evolution in 2026: native Arrow UDFs, which operate directly on pyarrow.Array / RecordBatch objects and skip the pandas conversion entirely — one less hop between "data arrives as Arrow" and "your function runs on it."

The rest of the ecosystem

• pandas 2.0+. You can back an entire DataFrame with Arrow dtypes instead of NumPy (dtype_backend="pyarrow") — better nullable-type support, and less silent type coercion than NumPy's object-dtype fallback for strings.
• Parquet I/O. PyArrow is one of pandas' two Parquet engines (engine="pyarrow"), and generally the more spec-complete of the two — the direct link back to the companion article on this profile.
• DuckDB. Genuinely tight integration: duckdb.sql(...).arrow() hands back a PyArrow table, and DuckDB can query a PyArrow table in memory as if it were a SQL table, no loading step in either direction.
• Polars — worth being precise here: Polars doesn't depend on the pyarrow package internally; it has its own Rust-native Arrow implementation. But it speaks the same Arrow format, which is why pl.from_arrow() on a PyArrow table is cheap rather than a real conversion.
• Arrow Flight / Flight SQL. A gRPC-based transport built specifically to move Arrow data between systems fast — increasingly the transport layer for BI tools and newer database drivers, instead of row-oriented ODBC/JDBC.
• ADBC (Arrow Database Connectivity). An emerging standard positioning itself as ODBC/JDBC's Arrow-native successor. Snowflake's and BigQuery's Python connectors can already hand back query results as Arrow tables directly, rather than a row-by-row cursor.

When to reach for the pyarrow API directly

Most of the time, Arrow is working underneath a tool you're already using and you never touch the pyarrow package by name. A few situations where reaching for it directly is the right call:

• pyarrow.compute — a library of vectorized functions (filtering, string operations, aggregations) that operate straight on Arrow arrays, useful when you want columnar-speed operations without pulling in the rest of pandas or Polars.
• Writing Parquet with fine control — pyarrow.parquet exposes the row-group size, compression codec, and encoding options the Parquet article covers, when a higher-level df.to_parquet() call doesn't expose the knob you need.
• Arrow Flight, if you're building (not just consuming) a system that needs to move large columnar results between processes or machines fast.

Where the "zero-copy" story has real edges

In the spirit of not overselling this: not everything touching Arrow is actually zero-copy, and it's worth knowing where that breaks down before you assume it everywhere.

• Casting between incompatible types still copies. Converting an Arrow int32 array to int64 has to build a new buffer at the new width — there's no way around that being real work.
• Combining chunked arrays can copy. Arrow tables are often stored as multiple chunks (e.g., one per batch read from a file); an operation that needs a single contiguous array sometimes has to concatenate chunks first, which is a real copy, not a pointer handoff.
• Crossing into pandas' legacy NumPy object dtype for strings still copies, because that representation is fundamentally different from Arrow's buffer layout — this is exactly why pandas 2.0's Arrow-backed dtype option exists, to avoid needing to make that crossing at all.

None of these are bugs — they're places where the data genuinely has to change shape, and Arrow doesn't pretend otherwise. The claim isn't "nothing ever copies." It's "moving the same logical data between two systems that already agree on its layout doesn't have to," which is a much stronger and much more specific claim — and, per the benchmark above, a true one.

Takeaways

• Arrow is a memory format, not a library trick — the same byte layout works across languages, which is the precondition for zero-copy to mean anything.
• The mechanism is the Arrow C Data Interface: a tiny, stable set of C structs that let independently-built libraries hand off a pointer instead of re-serializing data.
• "Zero-copy" is a real, measurable claim, not a marketing word — the benchmark above shows the gap growing, not shrinking, as data gets larger.
• PyArrow shows up almost everywhere in a modern Python data stack — pandas, Polars, DuckDB, Spark/Databricks, warehouse connectors — usually invisibly, which is exactly the point.
• It's not infinitely free: type casts, chunk concatenation, and legacy pandas string handling are real, known places where a copy still happens.

If this is useful background, the Parquet article on this profile covers the disk-format half of this exact story — the two are designed to be read as a pair.