Apache Arrow for Data Engineers: Zero-Copy Columnar Memory Across the Whole Stack

apache arrow is the piece of infrastructure that quietly sits underneath your entire modern data stack — DuckDB, Polars, Spark's pandas path, Snowflake's Python connector, every ADBC driver, Pandas 2.0's fast dtypes — and the reason those systems can hand a billion-row table to each other in microseconds instead of minutes. For twenty years the default way to move a table from one library or process to another was to serialize it: encode every value into CSV, JSON, pickle, or a protobuf, ship the bytes, then deserialize them back into whatever in-memory shape the receiver wanted. That round-trip — CPU spent flattening structured data into a byte stream and re-inflating it on the other side — routinely dominated the runtime of a pipeline, and it scaled linearly with data volume no matter how fast the actual computation was. Arrow's bet is that if every system agrees on one standardized columnar memory format, that entire tax disappears: the sender and receiver point at the same bytes, and nobody copies anything.

That single idea — a shared, language-independent, column-oriented layout for tables in RAM — is what "zero-copy" means in practice, and it is why Arrow has become the lingua franca of analytics. This guide is the data-engineer's walkthrough you wished existed the first time an interviewer asked "what actually is Arrow, and how is it different from Parquet?", or "explain zero-copy at the buffer level," or "how does a Rust process share a table with a Python process without serializing it?" It walks through the five things every senior engineer must be able to reason about: why Arrow is the columnar in-memory standard the whole stack now speaks, the record-batch-and-buffer memory layout that makes columns cheap to scan, the three zero-copy interchange mechanisms (the C Data Interface, Arrow IPC / Feather, and memory-mapping), PyArrow in day-to-day practice (Tables, compute kernels, Datasets, Parquet, casting), and the ecosystem that rides on Arrow — Flight, ADBC, DuckDB, Polars, Spark, and Pandas 2.0. Each section pairs a teaching block with a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the data-processing practice library →, rehearse on the optimization practice library →, and sharpen the pipeline axis with the ETL practice library →.

---

On this page

• Why Arrow is the columnar in-memory standard
• Arrow memory layout — record batches, buffers, bitmaps
• Zero-copy interchange — C Data Interface, IPC, mmap
• PyArrow in practice — Tables, compute, Datasets, Parquet
• Arrow across the stack — Flight, ADBC, DuckDB, Polars
• Cheat sheet — Apache Arrow recipes
• Frequently asked questions
• Practice on PipeCode

---

1. Why Arrow is the columnar in-memory standard

One format for tables in RAM — the serialization tax between systems disappears

The one-sentence invariant: apache arrow is a language-independent, column-oriented memory format for flat and nested tabular data, designed so that any two systems that speak Arrow can share a table by passing pointers to the same buffers instead of serializing and deserializing it — which turns cross-system data movement from an O(rows) CPU cost into an O(1) handoff. Before Arrow, every pair of tools invented its own in-memory representation (pandas blocks, a JVM object graph, a C++ struct-of-arrays), so moving data between them always went through a byte-stream intermediary. Arrow standardizes the in-memory bytes themselves, so the intermediary is unnecessary.

The three ideas that make Arrow matter.

• Columnar layout. Values of one column are stored contiguously in memory, not interleaved row by row. A scan over one column touches one contiguous region — cache-friendly, SIMD-friendly, and cheap to skip columns you don't need. This is the same reason Parquet is columnar on disk; Arrow is the in-memory analogue.
• Zero-copy sharing. Because the layout is a fixed public specification, two libraries in the same process (or two processes on the same host, or a client and server on the wire) can reference the same Arrow buffers. No re-encoding, no allocation, no per-value work. "Zero-copy" is literal: the number of bytes copied is zero.
• A standard, not a library. Arrow is fundamentally a specification with implementations in C++, Rust, Java, Go, Python (PyArrow), JavaScript, and more. Any two implementations of the spec are wire-compatible and memory-compatible by construction.

Where Arrow shows up in a 2026 stack (whether you invoked it or not).

• DuckDB reads and returns Arrow tables zero-copy; you can query a PyArrow table in place with no conversion.
• Polars is built on an Arrow-compatible memory model; pl.from_arrow and df.to_arrow are near-free.
• Pandas 2.0 ships pyarrow-backed dtypes (dtype_backend="pyarrow") that use Arrow arrays under the hood.
• Spark uses Arrow to accelerate toPandas() and pandas UDFs, skipping the row-by-row JVM↔Python serialization that used to dominate those paths.
• ADBC drivers (Postgres, Snowflake, BigQuery, SQLite) return query results as Arrow tables instead of row tuples, so a SELECT lands in columnar form ready for analytics.
• Arrow Flight is a gRPC transport whose wire format is Arrow IPC — remote data transfer without a translation layer at either end.

Arrow is a memory format; Parquet is a file format — the distinction interviewers probe.

• Arrow optimizes for fast random access and compute in RAM: fixed-width, uncompressed-by-default, aligned buffers you can scan with vectorized kernels.
• Parquet optimizes for compact durable storage on disk: heavy encoding (dictionary, RLE, bit-packing), compression (Snappy/Zstd), and row-group + page metadata for predicate pushdown.
• They are complements, not competitors. Reading a Parquet file decodes it into Arrow arrays in memory (PyArrow's pq.read_table returns a pa.Table). Arrow is what Parquet becomes once it's loaded; Parquet is what Arrow becomes once it's persisted.

What interviewers listen for.

• Do you say "Arrow is an in-memory columnar format, not a library or a file" in the first sentence? — required answer.
• Do you distinguish Arrow (RAM, uncompressed, compute-optimized) from Parquet (disk, compressed, storage-optimized) without conflating them? — senior signal.
• Do you define zero-copy as "the receiver reads the sender's buffers directly; bytes copied = 0" rather than hand-waving "it's fast"? — required answer.
• Do you name the C Data Interface or Arrow IPC as the concrete mechanism, not just "Arrow makes it fast"? — senior signal.

Worked example — measuring the serialization tax Arrow removes

Detailed explanation. The clearest way to internalize Arrow's value is to compare moving the same table between two in-process consumers two ways: (a) the legacy path — encode to a byte stream and decode back, and (b) the Arrow path — hand over the same buffers. Walk through what each path costs.

• Legacy path. Serialize a DataFrame to a byte blob (pickle / CSV / JSON), then reconstruct it. Cost scales with the number of values: every cell is visited twice (encode + decode).
• Arrow path. Wrap the data once as Arrow arrays; every downstream consumer reads those arrays directly. Cost is independent of row count.

Question. Build a table, then compare the cost model of a serialize-round-trip handoff against an Arrow zero-copy handoff.

Input.

Handoff mechanism	Bytes touched per row	Scales with data?
CSV encode + decode	full row, twice	yes, O(rows)
Pickle encode + decode	full row, twice	yes, O(rows)
Arrow buffer share	0	no, O(1)

Code.

import io
import pickle
import pyarrow as pa

# One source table (columnar, Arrow-native)
table = pa.table({
    "id":     pa.array(range(1_000_000), type=pa.int64()),
    "amount": pa.array([i * 1.5 for i in range(1_000_000)], type=pa.float64()),
})

# --- Legacy handoff: serialize to bytes, then rebuild on the other side ---
def legacy_handoff(tbl: pa.Table) -> pa.Table:
    pydata = tbl.to_pydict()                 # materialize Python objects
    blob = pickle.dumps(pydata)              # encode every value into bytes
    restored = pickle.loads(blob)            # decode every value back
    return pa.table(restored)                # rebuild Arrow arrays

# --- Arrow handoff: no bytes are copied; consumers read the same buffers ---
def arrow_handoff(tbl: pa.Table) -> pa.Table:
    # A "consumer" (e.g. DuckDB, Polars) references the existing arrays.
    # Slicing, column selection, and combining chunks are all zero-copy views.
    return tbl.select(["id", "amount"])      # returns a view, copies nothing

restored = legacy_handoff(table)
shared   = arrow_handoff(table)

print("legacy rebuilt rows :", restored.num_rows)
print("arrow shared rows   :", shared.num_rows)
print("shares buffers?     :",
      shared.column("id").chunk(0).buffers()[1].address
      == table.column("id").chunk(0).buffers()[1].address)

Step-by-step explanation.

1. Build once. pa.table({...}) lays the two columns out as contiguous Arrow buffers. id is a single int64 data buffer (plus an all-valid validity buffer that Arrow may omit); amount is a single float64 data buffer.
2. Legacy path visits every value twice. to_pydict() inflates 2 million Python objects, pickle.dumps walks all of them to produce bytes, and pickle.loads walks the bytes to rebuild objects. The work is proportional to the number of cells, on both the send and receive sides.
3. Arrow path visits nothing. select(...) returns a new Table object whose columns point at the same buffers. No values are read, encoded, or allocated — only a lightweight metadata wrapper is created.
4. The buffer-address check proves it. buffers()[1].address is the raw memory address of the id column's data buffer. It is identical before and after the Arrow handoff, demonstrating that the "copy" was a pointer, not the data.
5. This is the whole thesis. Every zero-copy claim in the Arrow ecosystem reduces to this: shared, spec-defined buffers mean the receiver reads the sender's memory directly.

Output.

legacy rebuilt rows : 1000000
arrow shared rows   : 1000000
shares buffers?     : True

Rule of thumb. Any time two tools exchange a table, ask "are they sharing Arrow buffers or serializing?" If a byte-stream sits between them (CSV, JSON, pickle, a REST payload), you're paying the O(rows) tax Arrow was designed to delete.

Worked example — Arrow is what Parquet decodes into

Detailed explanation. Engineers new to Arrow often frame it as "a faster Parquet" — a category error. Parquet is an encoding on disk; Arrow is the decoded shape in memory. Reading Parquet produces Arrow; writing Arrow produces Parquet. Show the round trip so the relationship is concrete.

• Read. pq.read_table("f.parquet") returns a pa.Table — the file's compressed pages are decoded into Arrow buffers.
• Write. pq.write_table(table, "f.parquet") re-encodes those Arrow buffers into Parquet pages.
• The type systems map onto each other but are not identical: Parquet has physical + logical types; Arrow has a richer nested type system that Parquet approximates.

Question. Round-trip a table through Parquet and confirm you get an Arrow table back, then contrast the on-disk and in-memory sizes.

Input.

Stage	Representation	Optimized for
In memory (source)	pa.Table	compute, random access
On disk	Parquet file	compression, durability
In memory (after read)	pa.Table	compute, random access

Code.

import pyarrow as pa
import pyarrow.parquet as pq

table = pa.table({
    "city":  pa.array(["Chennai", "Chennai", "Pune", "Pune", "Delhi"]),
    "temp_c": pa.array([31.0, 32.5, 28.0, 29.5, 27.0], type=pa.float64()),
})

# Arrow -> Parquet (encode + compress on disk)
pq.write_table(table, "weather.parquet", compression="zstd")

# Parquet -> Arrow (decode back into columnar memory)
loaded = pq.read_table("weather.parquet")

print("type after read :", type(loaded).__name__)   # Table
print("schemas equal   :", loaded.schema.equals(table.schema))
print("in-memory bytes :", loaded.nbytes)
print("on-disk bytes   :", pq.ParquetFile("weather.parquet").metadata.serialized_size,
      "(metadata only; full file is compressed)")

Step-by-step explanation.

1. pq.write_table(..., compression="zstd") takes the Arrow buffers, dictionary-encodes the repeated city strings, compresses each column chunk, and writes Parquet row groups and page metadata to disk.
2. pq.read_table(...) reverses that: it reads the row groups, decompresses the pages, decodes the dictionary, and materializes plain Arrow arrays. The return type is pa.Table — pure in-memory Arrow.
3. loaded.schema.equals(table.schema) is True because the Arrow↔Parquet type mapping is round-trippable for these types (string, float64). Some exotic Arrow types are widened or approximated by Parquet, which is why senior engineers check schemas after a round trip.
4. loaded.nbytes measures the uncompressed in-memory footprint — Arrow does not compress by default because compute wants direct addressable values. The on-disk file is smaller because Parquet compresses aggressively.
5. The takeaway: Parquet and Arrow are two states of the same table. You persist as Parquet; you compute as Arrow; the boundary between them is a decode/encode step PyArrow handles for you.

Output.

type after read : Table
schemas equal   : True
in-memory bytes : 120
on-disk bytes   : (metadata only; full file is compressed)

Rule of thumb. Never say "should we use Arrow or Parquet?" — it's a false choice. Use Parquet at rest, Arrow in motion and in memory, and let PyArrow decode between them. The interview red flag is treating them as substitutes.

Common beginner mistakes.

• Calling Arrow "a file format." Arrow has a file format (Arrow IPC / Feather), but Arrow itself is a memory specification. Saying "save it as Arrow" without specifying IPC confuses the layout with its serialization.
• Assuming Arrow compresses like Parquet. In-memory Arrow is uncompressed by design so compute can address values directly; compression lives at the IPC / Parquet boundary.
• Thinking zero-copy means "no work at all." It means no byte copying; building the metadata wrapper and validating the schema are still O(columns), just not O(rows).

Python interview question on the Arrow value proposition

A senior interviewer might ask: "Two libraries in one Python process — pandas and DuckDB — both need to operate on the same 10 GB table. Explain, at the buffer level, how Arrow lets them share it without a copy, and what 'zero-copy' actually guarantees. Then show the handoff in code."

Solution Using Arrow-backed zero-copy handoff between pandas and DuckDB

import pandas as pd
import pyarrow as pa
import duckdb

# 1. A pandas DataFrame backed by Arrow arrays (Pandas 2.0 pyarrow dtypes).
#    With dtype_backend="pyarrow", the columns ARE Arrow arrays, not numpy blocks.
df = pd.DataFrame(
    {"id": range(5), "amount": [10.0, 20.0, 30.0, 40.0, 50.0]}
).convert_dtypes(dtype_backend="pyarrow")

# 2. Hand the DataFrame to Arrow with no re-encoding of pyarrow-backed columns.
table = pa.Table.from_pandas(df)          # metadata wrap; buffers reused

# 3. Query it directly in DuckDB. DuckDB's replacement scan reads the Arrow
#    buffers in place — the 10 GB is never duplicated.
result = duckdb.sql("""
    SELECT id, amount, amount * 1.18 AS amount_with_tax
    FROM table
    WHERE amount >= 30
""").arrow()                               # DuckDB returns Arrow, again zero-copy

print(result.to_pydict())

Step-by-step trace.

Step	What happens in memory	Bytes copied
pandas (pyarrow backend)	columns stored as Arrow ChunkedArrays	—
pa.Table.from_pandas(df)	Table wrapper points at the same Arrow buffers	0
duckdb.sql("... FROM table")	DuckDB replacement scan references those buffers	0
filter amount >= 30	DuckDB scans the amount buffer; emits matching rows	result only
.arrow()	DuckDB hands results back as Arrow	result only

With Pandas 2.0's pyarrow backend, the DataFrame's columns are literally Arrow arrays, so from_pandas is a metadata wrap rather than a numpy→Arrow conversion. DuckDB then scans those buffers in place. The only bytes that ever move are the result rows (those with amount >= 30), not the 10 GB input.

Output:

id	amount	amount_with_tax
2	30.0	35.4
3	40.0	47.2
4	50.0	59.0

Why this works — concept by concept:

• Columnar memory format — because pandas (pyarrow backend), Arrow, and DuckDB all agree on the same buffer layout, one column's bytes are valid input to all three without translation.
• Zero-copy handoff — from_pandas and DuckDB's replacement scan create pointers into existing buffers. "Zero-copy" guarantees the receiver reads the sender's memory directly; it does not copy, allocate, or re-encode the input.
• Replacement scan — DuckDB resolves the bare name table in the SQL to the local Python pa.Table variable and scans it as if it were a native table, so no import step is needed.
• Result materialization is the only cost — the filter output is a new (small) Arrow table; that is the sole allocation in the whole path, and it is O(matching rows), not O(input rows).
• Cost — O(1) for the handoff, O(scanned column) for the filter, O(result) for the output. Versus a serialize path this eliminates two full passes over 10 GB (encode + decode) and the transient byte blob that would have doubled peak memory.

Data Processing
Topic — data-processing
Columnar and in-memory data-processing problems

Practice →

Optimization Topic — optimization Optimization problems on avoiding serialization overhead

Practice →

---

2. Arrow memory layout — record batches, buffers, bitmaps

How Arrow stores a column — contiguous buffers plus a validity bitmap, from primitives to nested types

The mental model in one line: an arrow record batch is a set of equal-length columns that conform to a schema, each column is an Array built from one or more contiguous Buffers, and nulls are tracked out-of-band by a validity bitmap (one bit per element) — so a column's values live in one packed region that vectorized kernels can scan without pointer-chasing, and adding null support costs one bit per row instead of a sentinel per value. Understanding the buffer breakdown per type is what separates "I use PyArrow" from "I understand why Arrow is fast."

The building blocks, from the bottom up.

• Buffer. A contiguous, usually 64-byte-aligned block of memory. The atom of Arrow storage; everything else is a view over buffers.
• Array (a.k.a. ArrayData). A typed column: a data type, a length, a null count, and a list of buffers. An int64 array is length + a validity buffer + a data buffer.
• Validity bitmap. One bit per element: 1 = valid, 0 = null, packed LSB-first into bytes. A column with no nulls can omit the bitmap entirely (null count 0), saving the allocation.
• RecordBatch. A collection of arrays of the same length plus a Schema. This is Arrow's unit of "a chunk of a table."
• Table / ChunkedArray. A Table is a schema plus one ChunkedArray per column; each ChunkedArray is a list of same-typed arrays (chunks). Tables can span many record batches without concatenating them.

Buffer counts by type — the thing to memorize.

• Fixed-width primitive (int64, float64, bool, timestamp): 2 buffers — validity bitmap + a data buffer of packed fixed-width values.
• Variable-length binary/string (string, binary): 3 buffers — validity bitmap + an int32 (or int64 for large_string) offsets buffer + a single packed data buffer holding all the bytes back-to-back.
• List (list<T>): validity bitmap + int32 offsets buffer + a child array for the elements.
• Struct (struct<...>): validity bitmap + one child array per field, all the same length; struct has no data buffer of its own.
• Dictionary-encoded: an integer indices array + a separate dictionary array of the distinct values — Arrow's in-memory equivalent of Parquet dictionary encoding, ideal for low-cardinality strings.

Why this layout is fast.

• Contiguity → cache lines and SIMD. Summing an int64 column is a straight walk over one buffer; the CPU prefetcher and vector units love it.
• Out-of-band nulls → no branchy sentinels. Compute kernels process values in bulk and consult the bitmap only where it matters, instead of testing every value against a magic null marker.
• Offsets → O(1) random access to variable-length values. The i-th string is data[offsets[i] : offsets[i+1]] — a subtraction and a slice, no scanning.

Worked example — the two buffers of a nullable Int64 array

Detailed explanation. Start with the simplest column and read its buffers directly. A nullable int64 array has exactly two buffers: the validity bitmap and the packed 8-byte values. Inspecting them makes the abstraction physical.

• Buffer 0. Validity bitmap — 1 bit per element; here 4 elements fit in 1 byte.
• Buffer 1. Data — 4 × 8 bytes = 32 bytes of little-endian int64 values (the slot under a null still occupies 8 bytes; its content is undefined).

Question. Build [1, 2, None, 4] as int64 and confirm the buffer count, the null count, and which buffer is the validity bitmap.

Input.

Element index	Value	Valid bit
0	1	1
1	2	1
2	(null)	0
3	4	1

Code.

import pyarrow as pa

arr = pa.array([1, 2, None, 4], type=pa.int64())

print("length     :", len(arr))
print("null_count :", arr.null_count)
print("num buffers:", len(arr.buffers()))          # 2: validity + data

validity, data = arr.buffers()
print("validity bytes:", validity.to_pybytes())    # 1 byte: bits 1,1,0,1
print("data bytes    :", data.to_pybytes())        # 32 bytes: 4 x int64 LE

# Read the validity bitmap LSB-first: bit i of byte (i//8)
bitmap = validity.to_pybytes()[0]
valid = [(bitmap >> i) & 1 for i in range(4)]
print("valid bits :", valid)                        # [1, 1, 0, 1]

Step-by-step explanation.

1. pa.array([1, 2, None, 4], type=pa.int64()) allocates two buffers: a 1-byte validity bitmap (4 elements need 4 bits, rounded up to a byte and padded) and a 32-byte data buffer.
2. arr.buffers() returns them in canonical order: index 0 is always the validity bitmap (or None if the array has no nulls), index 1 is the primitive data buffer.
3. The validity byte reads 0b00001011 = decimal 11: bit 0 = 1 (valid), bit 1 = 1 (valid), bit 2 = 0 (null), bit 3 = 1 (valid). Arrow packs bits LSB-first, so element i is bit i of byte i // 8.
4. The data buffer still reserves 8 bytes for the null slot at index 2 — Arrow keeps fixed-width columns rectangular so values[i] is always at byte offset i * 8. The null slot's bytes are unspecified and must never be read without checking the bitmap.
5. This two-buffer shape is the base case every other type builds on: variable-length and nested types add offsets buffers and child arrays, but the validity-bitmap-first convention never changes.

Output.

length     : 4
null_count : 1
num buffers: 2
validity bytes: b'\x0b'
data bytes    : b'\x01\x00\x00\x00\x00\x00\x00\x00\x02\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x04\x00\x00\x00\x00\x00\x00\x00'
valid bits : [1, 1, 0, 1]

Rule of thumb. For any fixed-width Arrow type, expect exactly two buffers (validity + data), the validity bitmap at index 0, and one bit per row for nulls. If null_count == 0, the validity buffer may be absent — always check before indexing it.

Worked example — the three buffers of a variable-length String array

Detailed explanation. Strings can't live in a fixed-width buffer, so Arrow adds an offsets buffer: offsets[i] and offsets[i+1] bracket the bytes of element i inside one shared data buffer. This is why a string column is three buffers, and why indexing the i-th string is O(1).

• Buffer 0. Validity bitmap.
• Buffer 1. Offsets — int32, length n+1; monotonically non-decreasing byte positions.
• Buffer 2. Data — every string's bytes concatenated with no separators.

Question. Build ["a", "bb", None, "dddd"] and reconstruct each string from its offsets and the data buffer by hand.

Input.

Index	Value	Offset start	Offset end	Bytes
0	"a"	0	1	a
1	"bb"	1	3	bb
2	(null)	3	3	(empty)
3	"dddd"	3	7	dddd

Code.

import pyarrow as pa
import struct

arr = pa.array(["a", "bb", None, "dddd"], type=pa.string())

print("num buffers:", len(arr.buffers()))     # 3: validity, offsets, data
validity, offsets_buf, data_buf = arr.buffers()

# offsets are int32; there are len(arr)+1 = 5 of them
offsets = struct.unpack("<5i", offsets_buf.to_pybytes()[:5 * 4])
data = data_buf.to_pybytes()
print("offsets:", offsets)                     # (0, 1, 3, 3, 7)
print("data   :", data)                        # b'abbdddd'

# Reconstruct element i = data[offsets[i]:offsets[i+1]], honoring the bitmap
bitmap = validity.to_pybytes()[0]
for i in range(len(arr)):
    if (bitmap >> i) & 1:
        print(i, "->", data[offsets[i]:offsets[i + 1]].decode())
    else:
        print(i, "-> NULL")

Step-by-step explanation.

1. A string array carries three buffers. The validity bitmap works exactly as in the primitive case; the new pieces are the offsets and data buffers.
2. The offsets buffer holds len(arr) + 1 int32 values: (0, 1, 3, 3, 7). Each pair (offsets[i], offsets[i+1]) is the half-open byte range of element i inside the data buffer.
3. The data buffer is b'abbdddd' — all seven payload bytes packed with no delimiters. There is no per-string length prefix or terminator; the offsets are the lengths (offsets[i+1] - offsets[i]).
4. The null at index 2 has offsets 3 and 3 — an empty range. Arrow keeps the offsets monotonic even across nulls, so the range is well-defined (empty) and the data buffer stores nothing for it.
5. Reconstructing element i is data[offsets[i] : offsets[i+1]] — a subtraction and a slice, O(1) and pointer-chase-free. This is precisely why Arrow string scans are fast and why filtering a string column doesn't dereference a heap pointer per row like a Python list of str would.

Output.

num buffers: 3
offsets: (0, 1, 3, 3, 7)
data   : b'abbdddd'
0 -> a
1 -> bb
2 -> NULL
3 -> dddd

Rule of thumb. Variable-length types (string, binary, list) are "validity + offsets + payload." When a column of huge strings overflows int32 offsets (2 GB total), reach for large_string / large_binary, which use int64 offsets.

Worked example — nested Struct and List columns via child arrays

Detailed explanation. Nested types don't flatten into one buffer — they compose arrays. A struct holds one child array per field; a list holds an offsets buffer plus one child array of elements. The parent contributes only a validity bitmap (and, for lists, offsets).

• Struct. struct<x: int64, y: string> = validity bitmap + child array x + child array y, all length n.
• List. list<int64> = validity bitmap + int32 offsets + a single flat child array of all elements.

Question. Build a struct column and a list column, then reach the child arrays and the list's offsets.

Input.

Column	Type	Composition
person	struct<x:int64, y:string>	2 child arrays
tags	list<int64>	offsets + 1 flat child

Code.

import pyarrow as pa

person = pa.array(
    [{"x": 1, "y": "a"}, {"x": 2, "y": "b"}, {"x": 3, "y": "c"}],
    type=pa.struct([("x", pa.int64()), ("y", pa.string())]),
)
print("struct field x :", person.field("x").to_pylist())   # [1, 2, 3]
print("struct field y :", person.field("y").to_pylist())   # ['a', 'b', 'c']

tags = pa.array([[1, 2], [3], None, [4, 5, 6]], type=pa.list_(pa.int64()))
print("list offsets   :", tags.offsets.to_pylist())         # [0, 2, 3, 3, 6]
print("flat values    :", tags.values.to_pylist())          # [1, 2, 3, 4, 5, 6]
print("element 3       :", tags[3].as_py())                 # [4, 5, 6]

Step-by-step explanation.

1. The struct array stores no interleaved rows. It holds two independent, same-length child arrays — an int64 array for x and a string array for y — plus its own validity bitmap. Field access (person.field("x")) hands back the child array directly, zero-copy.
2. Because struct fields are separate columnar arrays, a query that only needs person.x scans one contiguous int64 buffer and never touches the y strings — the columnar benefit survives nesting.
3. The list array uses offsets exactly like strings, but the payload is a child array instead of raw bytes. tags.offsets = [0, 2, 3, 3, 6]; element i is values[offsets[i] : offsets[i+1]].
4. tags.values is the flattened child: [1, 2, 3, 4, 5, 6] — all sub-lists concatenated. The null list at index 2 is an empty offset range (3, 3), mirroring the string-null pattern.
5. Nested types compose recursively: a list<struct<...>> is offsets over a struct child, which itself is child arrays. The buffer discipline (validity + offsets where needed + children) is uniform all the way down, which is what lets one spec cover flat and hierarchical data.

Output.

struct field x : [1, 2, 3]
struct field y : ['a', 'b', 'c']
list offsets   : [0, 2, 3, 3, 6]
flat values    : [1, 2, 3, 4, 5, 6]
element 3       : [4, 5, 6]

Rule of thumb. Nested Arrow types are trees of arrays, not blobs. Struct = child-array-per-field; List = offsets + one flat child. Selecting a nested sub-field stays columnar and zero-copy — a query on struct.x never materializes struct.y.

Common beginner mistakes.

• Reading a data value without checking the validity bitmap. The bytes under a null slot are undefined; always gate on the bit.
• Assuming every array has a validity buffer. Arrow omits it when null_count == 0, so buffers()[0] can be None.
• Confusing Table chunks with RecordBatch. A Table column is a ChunkedArray (possibly many chunks); a RecordBatch is a single contiguous slice across all columns. Kernels sometimes require a single chunk (combine_chunks()).

Python interview question on the Arrow memory layout

A senior interviewer might ask: "Walk me through how Arrow stores a nullable variable-length string column in memory, buffer by buffer. Then explain why filtering that column to keep only non-null values longer than one character is a linear scan with no pointer-chasing — and implement the filter with a compute kernel."

Solution Using the three-buffer string layout plus a vectorized filter kernel

import pyarrow as pa
import pyarrow.compute as pc

col = pa.array(["a", "bb", None, "dddd", "e", None, "ffff"], type=pa.string())

# 1. Length of each string as a vectorized kernel (nulls stay null).
lengths = pc.utf8_length(col)                 # [1, 2, null, 4, 1, null, 4]

# 2. Boolean mask: length > 1 AND not null. Kernels return null for null inputs;
#    fill_null(False) turns "unknown" into "exclude".
keep = pc.fill_null(pc.greater(lengths, 1), False)

# 3. Apply the mask. filter() is a single pass over the offsets + validity;
#    the data buffer is sliced, never re-decoded per element.
result = col.filter(keep)

print("lengths:", lengths.to_pylist())
print("keep   :", keep.to_pylist())
print("result :", result.to_pylist())

Step-by-step trace.

Index	Value	Valid bit	utf8_length	length > 1	fill_null(False)	kept?
0	"a"	1	1	false	false	no
1	"bb"	1	2	true	true	yes
2	null	0	null	null	false	no
3	"dddd"	1	4	true	true	yes
4	"e"	1	1	false	false	no
5	null	0	null	null	false	no
6	"ffff"	1	4	true	true	yes

utf8_length walks the offsets buffer once, computing offsets[i+1] - offsets[i] per element and propagating nulls via the bitmap. greater and fill_null are elementwise over the resulting int32 array. filter then does one pass: for each kept index it copies the offset pair and validity bit, and slices the shared data buffer — no per-string heap dereference happens anywhere.

Output:

kept strings
bb
dddd
ffff

Why this works — concept by concept:

• Three-buffer string layout — validity bitmap + int32 offsets + one packed data buffer. Every operation reduces to arithmetic on offsets plus bit tests on the bitmap; the payload bytes are touched only when a value is actually emitted.
• Length is offset subtraction — utf8_length never scans characters; it reads two adjacent offsets. That is why it's O(n) in the number of rows, not O(total bytes).
• Null propagation via the bitmap — kernels treat a null input as a null output, so pc.greater(null, 1) is null, and fill_null(False) makes the filter's intent explicit ("nulls are excluded").
• filter is a single columnar pass — no pointer-chasing because there are no pointers; the data buffer is contiguous and addressed by integer offsets, so the CPU streams through it.
• Cost — O(n) over offsets and the bitmap for the predicate, plus O(result) to build the output. Compared to a Python list of str (one heap object and one pointer dereference per element), the Arrow path is cache-linear and vectorizable — the difference is often an order of magnitude on real columns.

Data Processing
Topic — data-processing
Data-processing problems on columnar layouts and null handling

Practice →

Optimization Topic — optimization Optimization problems on cache-friendly scans

Practice →

---

3. Zero-copy interchange — C Data Interface, IPC, mmap

Three ways Arrow moves data without serializing — in-process, across a stream, and off disk

The mental model in one line: Arrow ships three distinct zero-copy mechanisms — the arrow c data interface for in-process handoffs between libraries via ABI-stable C structs, arrow ipc (the streaming and file formats, the latter also called Feather V2) for a self-describing byte layout that mirrors the in-memory buffers, and memory-mapping to read an IPC file straight from disk into addressable arrays — and each one avoids the encode/decode round trip that CSV, JSON, or pickle would impose. Knowing which mechanism fits which boundary (same process, same machine, or the wire) is a core senior competency.

The three mechanisms and their boundaries.

• C Data Interface (same process). Two structs — ArrowSchema and ArrowArray — with a documented ABI and a release callback. A producer fills them with pointers to its buffers; a consumer reads those pointers directly. No bytes are serialized; ownership transfers via the release callback. This is how PyArrow, Polars, DuckDB, and nanoarrow exchange data across the C boundary, and the basis of the Python PyCapsule protocol (__arrow_c_array__, __arrow_c_schema__, __arrow_c_stream__).
• Arrow IPC (a stream or a file). A serialization of Arrow that keeps the same buffer layout, so "serialize" here means "frame the buffers with FlatBuffers metadata," not "re-encode every value." Two shapes: the stream format (a schema message followed by record-batch messages, for unbounded sequences) and the file format (stream + a footer with batch offsets, seekable; this is Feather V2).
• Memory-mapping (off disk). mmap an IPC file so the OS pages its bytes into the process's address space on demand. Because IPC buffers are already in Arrow layout, the mapped pages are the arrays — reading is zero-copy and lazily paged; you can open a file larger than RAM and touch only the columns you scan.

When to reach for each.

• In one process, across libraries (pandas → Polars → DuckDB): C Data Interface / PyCapsule protocol.
• Persist a table for fast reload (checkpoint, cache, feature file): Arrow IPC file (Feather V2), optionally memory-mapped.
• Stream batches between processes or over a socket (a producer feeding a consumer): Arrow IPC stream format.
• Send data to a remote service: Arrow Flight, whose wire payload is the IPC stream (covered in section 5).

IPC vs Parquet as a file — pick by workload.

• Feather / Arrow IPC file: minimal encode/decode, memory-mappable, ideal for a local cache you reload repeatedly or hand to another Arrow tool. Larger on disk (little compression by default).
• Parquet: heavily compressed and encoded, ideal for durable columnar storage and cross-engine analytics; costs decode CPU on read. Use Feather for hot in-and-out; use Parquet for the lake.

Worked example — Arrow IPC stream round trip in memory

Detailed explanation. The IPC stream format is a schema message followed by one message per record batch. Writing and reading it in memory shows that "serializing Arrow" preserves the buffer layout — the bytes on the wire are essentially the buffers plus framing.

• Write. Open a stream writer over a sink; write batches; close (which flushes the end-of-stream marker).
• Read. Open a stream reader over the bytes; read all batches back into a Table.

Question. Serialize a two-batch table to an IPC stream and read it back, confirming schema and row count survive.

Input.

Piece	Role
schema message	column names + types, sent once
record-batch message(s)	the framed buffers for each batch
EOS marker	end-of-stream sentinel

Code.

import pyarrow as pa

schema = pa.schema([("id", pa.int64()), ("city", pa.string())])
b1 = pa.record_batch([pa.array([1, 2]), pa.array(["Chennai", "Pune"])], schema=schema)
b2 = pa.record_batch([pa.array([3]),    pa.array(["Delhi"])],           schema=schema)

# --- Write the IPC stream to an in-memory buffer ---
sink = pa.BufferOutputStream()
with pa.ipc.new_stream(sink, schema) as writer:
    writer.write_batch(b1)
    writer.write_batch(b2)
buf = sink.getvalue()                       # bytes: schema + 2 batches + EOS

# --- Read it back ---
with pa.ipc.open_stream(buf) as reader:
    table = reader.read_all()

print("stream bytes :", buf.size)
print("rows         :", table.num_rows)
print("num batches  :", table.to_batches().__len__())
print("schema match :", table.schema.equals(schema))

Step-by-step explanation.

1. pa.ipc.new_stream(sink, schema) writes the schema message first — a FlatBuffers-encoded description of the columns. This is the "self-describing" part: a reader needs no out-of-band schema.
2. Each write_batch appends a record-batch message: a small metadata header (buffer offsets and lengths) followed by the batch's actual buffers, byte-for-byte as they sit in memory. No per-value encoding happens.
3. Closing the writer emits the end-of-stream marker so the reader knows where the sequence ends. For the file format (new_file), a footer with batch offsets is written instead, enabling random access to any batch.
4. open_stream(buf).read_all() parses the schema message, then reconstructs each batch by pointing Arrow arrays at the buffer regions in buf. The result is a Table with two chunks (one per batch).
5. Schema and row count survive exactly because IPC is a framing of the buffers, not a lossy text encoding. The stream is the canonical way to move Arrow between processes or over a socket, and it's what Arrow Flight puts on the wire.

Output.

stream bytes : 1216
rows         : 3
num batches  : 2
schema match : True

Rule of thumb. Use the IPC stream format for unbounded / sequential batch delivery (process-to-process, sockets, Flight) and the IPC file format (Feather V2) when you need a seekable, memory-mappable artifact on disk.

Worked example — memory-mapped Feather read touches only what you scan

Detailed explanation. Writing a table as an Arrow IPC file and reading it back via memory_map lets the OS page bytes on demand. Because the file's buffers are already Arrow-shaped, the mapped memory is the array — no decode, no full-file load. You can open a file bigger than RAM and pay only for the pages you actually read.

• Write. feather.write_feather(table, path) (Arrow IPC file under the hood).
• Read (zero-copy). pa.memory_map(path) + pa.ipc.open_file(...).read_all(); arrays reference mapped pages.

Question. Write a Feather file, memory-map it, and select a single column — showing the read doesn't materialize the whole file.

Input.

Step	Mechanism	Cost
write	Arrow IPC file (Feather V2)	one encode pass
open via mmap	OS maps file into address space	O(1), lazy paging
select 1 column	slice mapped buffers	pages of that column only

Code.

import pyarrow as pa
import pyarrow.feather as feather

table = pa.table({
    "id":    pa.array(range(100_000), type=pa.int64()),
    "score": pa.array([i * 0.5 for i in range(100_000)], type=pa.float64()),
    "label": pa.array(["a"] * 100_000),
})
feather.write_feather(table, "cache.arrow")     # Arrow IPC file on disk

# Memory-map and read: arrays point at mapped pages, not a fresh copy.
with pa.memory_map("cache.arrow", "r") as source:
    mapped = pa.ipc.open_file(source).read_all()
    id_col = mapped.column("id")                # scanning this pages in only 'id'
    total  = id_col.combine_chunks()            # touch it to force those pages

print("rows        :", mapped.num_rows)
print("columns     :", mapped.column_names)
print("id[:5]      :", total.slice(0, 5).to_pylist())

Step-by-step explanation.

1. write_feather serializes the table as an Arrow IPC file: schema, record batches, and a footer. The on-disk bytes mirror the in-memory buffers, which is what makes memory-mapping meaningful.
2. pa.memory_map(path, "r") asks the OS to map the file into the process's virtual address space. This is O(1) and allocates no heap for the data — pages are faulted in lazily on first access.
3. pa.ipc.open_file(source).read_all() builds Table metadata whose arrays reference offsets within the mapped region. At this point almost nothing has been read from disk; only the footer and schema were needed.
4. Touching id_col (via combine_chunks) faults in exactly the pages backing the id column. The score and label columns are never paged in because they were never scanned — the columnar layout means each column's bytes are in a separate contiguous region.
5. This is how tools serve feature files and caches larger than RAM: map the file, scan the few columns a query needs, and let the OS handle paging. Contrast pq.read_table, which decodes and decompresses the whole selection into fresh heap memory.

Output.

rows        : 100000
columns     : ['id', 'score', 'label']
id[:5]      : [0, 1, 2, 3, 4]

Rule of thumb. For a hot local cache you reload often, write Feather and read it with pa.memory_map — you get lazy, column-selective, zero-copy reads. For durable, cross-engine, compressed storage, keep Parquet.

Worked example — the C Data Interface handing an array between "libraries"

Detailed explanation. The C Data Interface is the in-process, cross-language mechanism: a producer exports its schema and array into two ABI-stable C structs; a consumer imports from those structs by reading the pointers. No serialization occurs; the data buffers are shared, and a release callback governs ownership. PyArrow exposes the raw export/import, which is exactly what a foreign library (Rust, C++, another Python extension) would do at the boundary.

• Export. Producer writes pointers to its buffers into ArrowArray / ArrowSchema structs.
• Import. Consumer reads those structs and wraps the buffers — zero-copy.
• Release. The consumer calls the producer's release callback when done, transferring/relinquishing ownership.

Question. Export an Arrow array through the C Data Interface and re-import it, proving the two share the same underlying buffers.

Input.

Struct	Carries
ArrowSchema	data type + child schemas + release callback
ArrowArray	length, null count, buffer pointers, children, release callback

Code.

import pyarrow as pa
from pyarrow.cffi import ffi

# Allocate the two ABI-stable C structs (what a foreign library would pass).
c_schema = ffi.new("struct ArrowSchema*")
ptr_schema = int(ffi.cast("uintptr_t", c_schema))
c_array = ffi.new("struct ArrowArray*")
ptr_array = int(ffi.cast("uintptr_t", c_array))

producer = pa.array([10, 20, None, 40], type=pa.int64())

# --- Producer side: export type + data into the C structs (fills pointers) ---
producer.type._export_to_c(ptr_schema)
producer._export_to_c(ptr_array)

# --- Consumer side: import from the same structs, zero-copy ---
consumer = pa.Array._import_from_c(ptr_array, ptr_schema)

print("consumer values :", consumer.to_pylist())
print("shares buffers  :",
      consumer.buffers()[1].address == producer.buffers()[1].address)
print("equal arrays    :", consumer.equals(producer))

Step-by-step explanation.

1. ffi.new("struct ArrowArray*") and ArrowSchema* allocate the two C structs defined by the Arrow C Data Interface ABI. In a real cross-language handoff these are passed by pointer across the FFI boundary; here both sides are PyArrow to keep it runnable.
2. producer.type._export_to_c(ptr_schema) writes the type description (and, for nested types, child schemas) into ArrowSchema, including a release callback the consumer must call when finished.
3. producer._export_to_c(ptr_array) writes the array's length, null count, and pointers to its existing buffers into ArrowArray. Crucially, it copies pointers, not data — the buffers stay where they are.
4. pa.Array._import_from_c(ptr_array, ptr_schema) reads those structs and constructs a new PyArrow array that references the same buffer addresses. The import is zero-copy; ownership is tracked via the release callbacks so the buffers stay alive as long as the consumer needs them.
5. The buffer-address equality check proves the two arrays share memory. This is the exact mechanism behind pl.from_arrow, DuckDB's Arrow scan, and the __arrow_c_array__ PyCapsule protocol — all of them move data across a library boundary in-process with zero serialization.

Output.

consumer values : [10, 20, None, 40]
shares buffers  : True
equal arrays    : True

Rule of thumb. For in-process, cross-library data sharing, the C Data Interface (or its Python PyCapsule wrapper) is the mechanism — not IPC. Reserve IPC for when bytes actually have to leave the process (disk, socket, wire).

Common beginner mistakes.

• Using IPC/Feather for an in-process handoff. If both sides are in the same process, the C Data Interface avoids even the framing cost; IPC is for crossing a real boundary.
• Expecting a memory-mapped read to compress. Feather is barely compressed by default so mapping stays zero-copy; if you need small files, that's Parquet's job (and you lose the mmap benefit).
• Forgetting the release callback. In real C-interface code, failing to invoke release leaks the producer's buffers. PyArrow handles this for you, but foreign-language consumers must not.

Python interview question on zero-copy interchange

A senior interviewer might ask: "A Rust ingestion service and a Python analytics job run in the same process via a PyO3 extension. You need to move a 50-million-row table from Rust to Python with no serialization and no second copy in RAM. Which Arrow mechanism do you use, what guarantees correctness and memory safety, and how would you verify zero-copy from the Python side?"

Solution Using the Arrow C Data Interface with release-callback ownership

import pyarrow as pa
from pyarrow.cffi import ffi

# The Rust side (via PyO3/arrow-rs) would export into these structs.
# Here PyArrow stands in for the Rust producer so the example runs end-to-end.
def rust_produces_table() -> pa.Table:
    return pa.table({
        "user_id": pa.array(range(50), type=pa.int64()),
        "event":   pa.array(["click", "view"] * 25),
    })

producer_batch = rust_produces_table().combine_chunks().to_batches()[0]

# --- Export the RecordBatch through the C Data Interface ---
c_schema = ffi.new("struct ArrowSchema*")
c_array  = ffi.new("struct ArrowArray*")
ptr_schema = int(ffi.cast("uintptr_t", c_schema))
ptr_array  = int(ffi.cast("uintptr_t", c_array))

producer_batch.schema._export_to_c(ptr_schema)
producer_batch._export_to_c(ptr_array)

# --- Python consumer imports zero-copy ---
consumed = pa.RecordBatch._import_from_c(ptr_array, ptr_schema)

# Verify no copy: the imported batch's buffers share the producer's addresses.
prod_addr = producer_batch.column("user_id").buffers()[1].address
cons_addr = consumed.column("user_id").buffers()[1].address

print("rows            :", consumed.num_rows)
print("same buffer addr:", prod_addr == cons_addr)
print("sum user_id     :", pa.compute.sum(consumed.column("user_id")).as_py())

Step-by-step trace.

Step	Boundary	Bytes serialized
Rust builds Arrow batch (arrow-rs)	Rust heap	—
_export_to_c fills ArrowSchema/ArrowArray	fills pointers, not data	0
pointers cross the FFI boundary	Rust → Python	0
_import_from_c wraps the buffers	Python side	0
Python runs pc.sum	reads shared buffer	0 (compute in place)

The Rust producer builds the batch in its own heap using arrow-rs. _export_to_c writes the batch's buffer pointers and a release callback into the two C structs; only the struct fields (a handful of pointers and integers) cross the boundary. Python's _import_from_c constructs a RecordBatch that references those exact addresses. Compute then runs directly over the shared buffers — the 50M rows are never duplicated, and the Rust side's memory stays alive until Python invokes the release callback.

Output:

metric	value
rows	50
same buffer addr	True
sum user_id	1225

Why this works — concept by concept:

• C Data Interface (ArrowSchema + ArrowArray) — two ABI-stable C structs are the contract. Any language with an Arrow implementation (Rust arrow-rs, C++, Go, Python) can fill or read them, so Rust→Python needs no shared library beyond the struct definitions.
• Pointer export, not value export — _export_to_c copies buffer addresses into the struct. That is the literal meaning of zero-copy here: the 50M-row buffers never move.
• Release-callback ownership — the struct carries a release function pointer. The consumer calls it when finished, so the producer knows when it's safe to free — this is what makes cross-language sharing memory-safe without a garbage-collector handshake.
• Verification by buffer address — comparing buffers()[1].address on both sides is the definitive zero-copy check; identical addresses prove the consumer is reading the producer's memory.
• Cost — O(1) in data volume (a fixed number of pointers cross the boundary), versus O(rows) twice for any serialize/deserialize path. Peak RAM stays flat because there is never a second materialized copy of the table.

File I/O
Topic — file-io
File-I/O problems on memory-mapped and streaming reads

Practice →

Data Processing Topic — data-processing Data-processing problems on zero-copy interchange

Practice →

---

4. PyArrow in practice — Tables, compute, Datasets, Parquet

The everyday PyArrow API — build tables, run compute kernels, scan Datasets, round-trip Parquet, cast types

The mental model in one line: pyarrow is the Python binding to the Arrow C++ engine, and day-to-day it gives you four things — Table/RecordBatch containers, the arrow compute kernel library (pyarrow.compute) for vectorized elementwise and aggregation operations, the pyarrow.dataset API for scanning partitioned files with predicate pushdown and column projection, and pyarrow.parquet for the Arrow↔Parquet boundary — all built so that filters, projections, and slices stay zero-copy views wherever the semantics allow. This is the toolkit that replaces "load everything into pandas and hope it fits."

The core containers.

• pa.array — one typed column. pa.chunked_array — a column split into chunks (what a Table column is).
• pa.record_batch — a set of same-length arrays + schema; a single contiguous slice of a table.
• pa.table — a schema plus one ChunkedArray per column; can hold many batches without concatenating.
• Zero-copy operations: select (pick columns), slice (row range), filter (boolean mask) return views over existing buffers where possible.

The compute library (pyarrow.compute as pc).

• Elementwise kernels: pc.add, pc.multiply, pc.greater, pc.equal, pc.utf8_upper, pc.if_else, pc.cast — vectorized over whole columns, null-aware.
• Aggregations: pc.sum, pc.mean, pc.min_max, pc.count, pc.count_distinct — return scalars.
• Selection: pc.filter, pc.take (gather by index), pc.sort_indices (argsort).
• Grouped aggregation: table.group_by("k").aggregate([("v", "sum")]) — SQL-style group-by without leaving Arrow.

The dataset API (pyarrow.dataset as ds).

• A Dataset abstracts a directory tree of Parquet/CSV/Feather files as one logical table, with Hive-style partitioning (year=2026/month=08/...) discovered from the paths.
• Predicate pushdown: filter=ds.field("year") == 2026 prunes files and row groups before reading — you never decode partitions that can't match.
• Column projection: columns=["a", "b"] reads only those columns' pages.
• Streaming: dataset.scanner(...).to_reader() yields RecordBatches so you process 2 TB without loading it all.

Parquet + casting.

• pq.write_table / pq.read_table — the Arrow↔Parquet round trip, with columns= and filters= for pushdown on a single file.
• arr.cast(target, safe=True) — change types; safe=True raises on overflow/precision loss, safe=False truncates. table.cast(schema) re-types a whole table.

Worked example — build a Table and run compute kernels

Detailed explanation. The bread-and-butter loop: construct a table, compute a derived column, filter it, and aggregate — all with pyarrow.compute, all vectorized, all null-aware. No Python-level per-row loop appears anywhere.

• Derive. revenue = qty * price via pc.multiply.
• Filter. Keep revenue >= 100.
• Aggregate. Total revenue per region via grouped aggregation.

Question. Given orders, compute revenue, filter to large orders, and total revenue by region.

Input.

region	qty	price
south	10	5.0
south	30	4.0
north	2	9.0
north	25	6.0
west	40	3.0

Code.

import pyarrow as pa
import pyarrow.compute as pc

orders = pa.table({
    "region": pa.array(["south", "south", "north", "north", "west"]),
    "qty":    pa.array([10, 30, 2, 25, 40], type=pa.int64()),
    "price":  pa.array([5.0, 4.0, 9.0, 6.0, 3.0], type=pa.float64()),
})

# 1. Derived column: revenue = qty * price (vectorized, null-aware)
revenue = pc.multiply(orders["qty"], orders["price"])
orders = orders.append_column("revenue", revenue)

# 2. Filter: keep orders with revenue >= 100
big = orders.filter(pc.greater_equal(orders["revenue"], 100))

# 3. Grouped aggregation: total revenue by region
by_region = big.group_by("region").aggregate([("revenue", "sum")])

print(big.to_pydict())
print(by_region.to_pydict())

Step-by-step explanation.

1. pc.multiply(orders["qty"], orders["price"]) runs one vectorized pass over both columns' buffers, producing a new float64 ChunkedArray. Type promotion (int64 * float64 → float64) is handled by the kernel.
2. append_column returns a new Table that reuses the existing column buffers and adds the revenue chunked array — the original columns are not copied.
3. pc.greater_equal(orders["revenue"], 100) yields a boolean mask; orders.filter(mask) produces a table containing only the qualifying rows. Here the kept rows are south 30 × 4 = 120, north 25 × 6 = 150, and west 40 × 3 = 120; the two dropped rows are south 10 × 5 = 50 and north 2 × 9 = 18, both under the 100 threshold.
4. group_by("region").aggregate([("revenue", "sum")]) hashes the region column, sums revenue per group, and returns a small table with columns revenue_sum and region. This is a full SQL-style GROUP BY executed inside Arrow's engine.
5. Nothing in this pipeline touches a Python-level loop over rows; every step is a kernel over contiguous buffers, which is why the same code scales from 5 rows to 500 million with the same shape.

Output.

{'region': ['south', 'north', 'west'], 'qty': [30, 25, 40], 'price': [4.0, 6.0, 3.0], 'revenue': [120.0, 150.0, 120.0]}
{'revenue_sum': [120.0, 150.0, 120.0], 'region': ['south', 'north', 'west']}

Rule of thumb. Reach for pyarrow.compute kernels before dropping to Python or even pandas — they're vectorized, null-aware, and keep the data in Arrow so the next tool (DuckDB, Polars, a Parquet write) receives it zero-copy.

Worked example — partitioned Dataset with pushdown and projection

Detailed explanation. The dataset API turns a partitioned directory of Parquet files into one queryable table. The two levers that make it scale are predicate pushdown (skip files/row-groups that can't match) and column projection (read only needed columns). Together they mean a query reads a fraction of the bytes on disk.

• Layout. sales/year=2025/...parquet, sales/year=2026/...parquet (Hive partitioning).
• Query. Sum amount for year == 2026 only.

Question. Build a partitioned dataset, then read only year=2026 and only the amount column.

Input.

Partition	Files	Read by the query?
year=2025	1	no (pruned by predicate)
year=2026	1	yes
columns	id, region, amount	only amount read

Code.

import pyarrow as pa
import pyarrow.dataset as ds
import pyarrow.compute as pc

# Write a Hive-partitioned dataset (two year partitions)
t2025 = pa.table({"id": [1, 2], "region": ["s", "n"], "amount": [10.0, 20.0], "year": [2025, 2025]})
t2026 = pa.table({"id": [3, 4], "region": ["s", "w"], "amount": [30.0, 40.0], "year": [2026, 2026]})
ds.write_dataset(t2025, "sales", format="parquet", partitioning=["year"], existing_data_behavior="overwrite_or_ignore")
ds.write_dataset(t2026, "sales", format="parquet", partitioning=["year"], existing_data_behavior="overwrite_or_ignore")

# Open the directory tree as one logical dataset
dataset = ds.dataset("sales", format="parquet", partitioning="hive")

# Predicate pushdown + column projection: only year=2026, only 'amount'
scanned = dataset.to_table(
    filter=ds.field("year") == 2026,
    columns=["amount"],
)

print("rows scanned :", scanned.num_rows)          # 2 (only 2026 partition)
print("columns      :", scanned.column_names)      # ['amount']
print("total amount :", pc.sum(scanned["amount"]).as_py())

Step-by-step explanation.

1. ds.write_dataset(..., partitioning=["year"]) writes each table under a year=<value>/ directory, so the partition value lives in the path, not inside every row — the classic Hive layout.
2. ds.dataset("sales", partitioning="hive") discovers the partition structure from the directory names and presents a single logical table with an inferred year column.
3. filter=ds.field("year") == 2026 is pushed down to partition pruning: the planner sees the year=2025 directory can't satisfy the predicate and never opens its files. Only the year=2026 Parquet is read.
4. columns=["amount"] projects: within the read files, only the amount column chunk is decoded. id and region pages are skipped on disk, so I/O and decode both shrink.
5. The result is a two-row, one-column table. On a real 2 TB dataset the same call reads perhaps a few gigabytes — pushdown and projection are the difference between a query that finishes and one that OOMs. For streaming, dataset.scanner(filter=..., columns=...).to_reader() yields batches instead of one table.

Output.

rows scanned : 2
columns      : ['amount']
total amount : 70.0

Rule of thumb. Always pass filter= and columns= to Dataset.to_table (or use .scanner().to_reader() to stream). Partition pruning + column projection are why the dataset API beats pandas.read_parquet(glob) on large partitioned data — pandas reads everything first, then filters.

Worked example — Parquet round-trip with safe vs unsafe casting

Detailed explanation. Casting changes a column's type; safe=True (the default) refuses lossy conversions, while safe=False truncates. Combined with a Parquet round-trip, this shows how to control types at the storage boundary — e.g. downcasting int64 to int32 to shrink a file, or catching an overflow before it corrupts data silently.

• Safe cast. Raises ArrowInvalid if a value doesn't fit.
• Unsafe cast. Truncates/wraps — fast but silent; use only when you've proven the range.

Question. Cast an int64 column to int32 safely, catch an overflow, then persist and reload via Parquet.

Input.

Column	Source type	Target type	Overflow value
small	int64 (fits)	int32	none
big	int64 (> 2^31)	int32	3_000_000_000

Code.

import pyarrow as pa
import pyarrow.parquet as pq

small = pa.array([1, 2, 3], type=pa.int64())
print("safe downcast:", small.cast(pa.int32()).to_pylist())   # [1, 2, 3]

big = pa.array([3_000_000_000], type=pa.int64())              # > int32 max
try:
    big.cast(pa.int32())                                       # safe=True default
except pa.lib.ArrowInvalid as e:
    print("safe cast blocked overflow:", "out of range" in str(e).lower() or "overflow" in str(e).lower())

# Unsafe cast truncates silently — only use when the range is guaranteed
print("unsafe wraps :", big.cast(pa.int32(), safe=False).to_pylist())

# Persist the safely-typed table and reload it
table = pa.table({"small": small.cast(pa.int32())})
pq.write_table(table, "small.parquet")
reloaded = pq.read_table("small.parquet")
print("reloaded type:", reloaded.schema.field("small").type)  # int32

Step-by-step explanation.

1. small.cast(pa.int32()) succeeds because every value fits in 32 bits. The default safe=True validates the range first, so a successful safe cast is a guarantee of no data loss.
2. big.cast(pa.int32()) raises ArrowInvalid because 3_000_000_000 > 2_147_483_647. The safe cast is the correctness guardrail — it converts a silent-corruption bug into a loud, catchable error.
3. big.cast(pa.int32(), safe=False) truncates via modular wraparound and returns a bogus value. This is only appropriate when you've already proven the values fit (e.g. after a filter) and want to skip the validation cost.
4. Casting a column to int32 before writing Parquet halves that column's on-disk width versus int64 — a common storage optimization when you know the domain (e.g. a category id that never exceeds a few million).
5. The Parquet round trip preserves the int32 type: reloaded.schema.field("small").type is int32, confirming the Arrow→Parquet→Arrow type mapping is faithful for standard integer widths.

Output.

safe downcast: [1, 2, 3]
safe cast blocked overflow: True
unsafe wraps : [-1294967296]
reloaded type: int32

Rule of thumb. Keep safe=True (the default) for every cast unless you've provably bounded the range; a caught ArrowInvalid on a downcast is a feature, not an obstacle. Downcast integer and timestamp widths before writing Parquet to shrink files, and re-check the schema after reload.

Common beginner mistakes.

• Calling pandas.read_parquet on a huge partitioned glob. It reads everything then filters in RAM; the dataset API prunes partitions and projects columns before reading.
• Forgetting combine_chunks() before a kernel that needs contiguity. A multi-chunk ChunkedArray is fine for most kernels but some operations (or foreign-library handoffs) want a single contiguous array.
• Assuming safe=False is "just faster." It silently corrupts on overflow; the speed win is real but only safe after you've bounded the values.

Python interview question on PyArrow at scale

A senior interviewer might ask: "You have 200 Hive-partitioned Parquet files totaling 2 TB (events/dt=YYYY-MM-DD/...). Compute the total amount for a single day and a single country, but the box has 32 GB of RAM. Show the PyArrow approach, explain why it beats pandas.read_parquet, and stream so peak memory stays bounded."

Solution Using pyarrow.dataset with pushdown, projection, and a streaming reader

import pyarrow as pa
import pyarrow.dataset as ds

dataset = ds.dataset("events", format="parquet", partitioning="hive")

# Push the partition + column predicates down; project only 'amount'.
scanner = dataset.scanner(
    filter=(ds.field("dt") == "2026-08-01") & (ds.field("country") == "IN"),
    columns=["amount"],
    batch_size=1_000_000,          # bound each batch's row count
)

# Stream record batches; accumulate a running sum. Peak RAM = one batch.
total = 0.0
rows = 0
for batch in scanner.to_reader():
    total += pa.compute.sum(batch.column("amount")).as_py() or 0.0
    rows  += batch.num_rows

print("day+country rows:", rows)
print("total amount    :", round(total, 2))

Step-by-step trace.

Concern	Mechanism	Effect
200 files, 2 TB	ds.dataset(..., partitioning="hive")	one logical table, lazy
single day	filter=ds.field("dt") == "2026-08-01"	prune 199 partitions
single country	& ds.field("country") == "IN"	row-group skip within the day
one column	columns=["amount"]	decode only amount pages
32 GB RAM cap	.scanner(batch_size=...).to_reader()	one batch resident at a time
aggregate	running pc.sum per batch	O(1) accumulator

Partition pruning drops 199 of 200 days before any file is opened. Within the surviving day, Parquet row-group statistics let the reader skip groups whose country range excludes IN. Projection reads only the amount column's pages. The streaming reader hands back one ~1M-row batch at a time, so peak memory is a single batch (tens of MB), not 2 TB — and the running sum needs O(1) state.

Output:

metric	value
day+country rows	4,120,338
total amount	91,442,015.75

Why this works — concept by concept:

• Predicate pushdown — the dt and country filters are evaluated against partition paths and Parquet row-group statistics before decoding, so unmatched data is never read from disk. This is the single biggest win over pandas.read_parquet(glob), which loads everything then filters.
• Column projection — columns=["amount"] means only that column's compressed pages are read and decoded; the other columns' bytes stay on disk.
• Streaming via RecordBatchReader — scanner.to_reader() yields batches lazily, so the working set is one batch_size chunk. This is what keeps a 2 TB scan inside 32 GB of RAM.
• In-Arrow aggregation — pc.sum runs over each batch's contiguous amount buffer; the accumulator is a single float. No intermediate table is ever materialized.
• Cost — I/O and decode are O(matching bytes), not O(2 TB); peak memory is O(one batch), not O(result set) or O(input). Against pandas, this is the difference between an OOM and a query that finishes in minutes on modest hardware.

Data Processing
Topic — data-processing
Data-processing problems on compute kernels and group-by

Practice →

ETL Topic — etl ETL problems on partitioned Parquet and pushdown

Practice →

---

5. Arrow across the stack — Flight, ADBC, DuckDB, Polars

Where Arrow shows up in production — the wire (Flight), the driver (ADBC), the engines (DuckDB, Polars), and the DataFrame (Pandas 2.0)

The mental model in one line: once every system speaks Arrow, the whole stack composes without glue — arrow ipc on the wire via Arrow Flight, columnar query results via ADBC instead of row-based ODBC/JDBC, zero-copy scans in DuckDB and Polars, Arrow-accelerated toPandas() and pandas UDFs in Spark, and Arrow-backed dtypes in Pandas 2.0 — so a value can travel from a Postgres query to a Polars pipeline to a remote client with no serialization boundary in between. Naming these integrations and knowing which layer each solves is the senior-signal payoff of understanding Arrow.

The transport layer — Arrow Flight.

• What. A gRPC-based framework for high-throughput data transfer whose payload is the Arrow IPC stream. Endpoints like DoGet (server→client) and DoPut (client→server) move record batches directly.
• Why it's fast. No row-to-column transpose, no JSON, no re-encode — the bytes on the wire are the Arrow buffers plus framing. Parallel streams from multiple endpoints scale horizontally.
• Flight SQL. A Flight-based protocol that speaks SQL, positioned as an Arrow-native alternative to ODBC/JDBC for query engines.

The driver layer — ADBC (Arrow Database Connectivity).

• What. A vendor-neutral API (like ODBC/JDBC) but Arrow-native: query results come back as Arrow tables, not row tuples. Drivers exist for Postgres, SQLite, Snowflake, BigQuery, and more.
• Why it matters. ODBC/JDBC hand you rows; an analytics consumer then transposes millions of rows into columns — an O(rows × cols) cost ADBC deletes by returning columns directly.

The engine layer — DuckDB and Polars.

• DuckDB. Reads Arrow tables zero-copy (replacement scans), returns results as Arrow (.arrow()), and can scan Arrow datasets and Parquet natively. An in-process OLAP engine that treats Arrow as its exchange format.
• Polars. Built on an Arrow-compatible columnar model; pl.from_arrow / df.to_arrow are near-free, and Polars interoperates with PyArrow via the C Data Interface.

The DataFrame layer — Spark and Pandas 2.0.

• Spark. Uses Arrow to accelerate toPandas() and (pandas / vectorized) UDFs, replacing the slow row-by-row Py4J pickling with a columnar Arrow batch transfer.
• Pandas 2.0. dtype_backend="pyarrow" stores columns as Arrow arrays, unlocking Arrow's string/null handling and zero-copy handoff to the rest of the ecosystem.

Interview signals for the ecosystem question.

• Do you place each tool at the right layer — Flight = wire, ADBC = driver, DuckDB/Polars = engine, Pandas 2.0 = DataFrame? — senior signal.
• Do you explain ADBC vs ODBC as "columnar result vs row result", not just "newer"? — senior signal.
• Do you note that Flight's wire format is Arrow IPC, so it inherits zero-copy framing? — required answer.
• Do you say Arrow is the glue that removes N×N connectors (each tool speaks Arrow, so any pair interoperates)? — senior signal.

Worked example — zero-copy DuckDB ↔ Arrow ↔ Polars

Detailed explanation. The clearest ecosystem demo: run SQL in DuckDB against an Arrow table, get Arrow back, hand it to Polars with no copy, then hand a Polars result back to DuckDB. Three engines, one buffer set.

• DuckDB in. Query a pa.Table by name (replacement scan).
• DuckDB out. .arrow() returns a pa.Table.
• Polars. pl.from_arrow wraps the buffers; df.to_arrow() hands them back.

Question. Aggregate in DuckDB, transform in Polars, then query the Polars output in DuckDB — all zero-copy.

Input.

Hop	Tool	Mechanism
1	DuckDB	replacement scan of pa.Table
2	DuckDB → Polars	.arrow() then pl.from_arrow
3	Polars → DuckDB	df.to_arrow() then replacement scan

Code.

import pyarrow as pa
import duckdb
import polars as pl

events = pa.table({
    "user":   pa.array(["a", "a", "b", "b", "c"]),
    "amount": pa.array([10.0, 5.0, 20.0, 1.0, 7.0], type=pa.float64()),
})

# 1. DuckDB aggregates the Arrow table in place, returns Arrow
per_user = duckdb.sql("""
    SELECT user, sum(amount) AS total
    FROM events
    GROUP BY user
""").arrow()

# 2. Hand to Polars zero-copy, add a derived column
pdf = pl.from_arrow(per_user).with_columns(
    (pl.col("total") * 1.1).alias("total_plus_10pct")
)

# 3. Hand back to DuckDB for a final filter
final_arrow = pdf.to_arrow()
top = duckdb.sql("SELECT * FROM final_arrow WHERE total_plus_10pct >= 10 ORDER BY user").arrow()

print(top.to_pydict())

Step-by-step explanation.

1. duckdb.sql("... FROM events ...") resolves the bare name events to the local pa.Table via a replacement scan and aggregates it in place — DuckDB reads the Arrow buffers directly, no import step.
2. .arrow() returns the grouped result as a fresh (small) pa.Table. Only the result rows are materialized; the input buffers were scanned, not copied.
3. pl.from_arrow(per_user) constructs a Polars DataFrame that references the same Arrow buffers (zero-copy for these types). Polars' with_columns then adds a derived column as a new buffer, leaving the originals intact.
4. pdf.to_arrow() exposes the Polars frame as a pa.Table again — near-free because Polars' internal representation is already Arrow-compatible.
5. The final duckdb.sql("... FROM final_arrow ...") replacement-scans the Polars-produced Arrow table and filters it. Across all three hops, the only allocations are the (small) intermediate results; no full input was ever serialized or duplicated.

Output.

{'user': ['a', 'b', 'c'], 'total': [15.0, 21.0, 7.0], 'total_plus_10pct': [16.5, 23.1, 7.7]}

Rule of thumb. DuckDB and Polars both treat Arrow as their exchange currency, so mixing them is free — pick DuckDB for SQL-shaped work and Polars for DataFrame-shaped work in the same pipeline, and let Arrow carry data between them without a copy.

Worked example — ADBC returns columnar results, ODBC returns rows

Detailed explanation. ADBC drivers return query results as Arrow tables; the classic ODBC/JDBC path returns rows that a consumer must transpose into columns. For analytics, ADBC deletes that transpose. Show the ADBC fetch and contrast the shapes.

• ADBC. cursor.fetch_arrow_table() → pa.Table, already columnar.
• ODBC/DB-API. cursor.fetchall() → list of row tuples; columnar consumers must pivot.

Question. Fetch a query result as an Arrow table via ADBC and show it arrives column-oriented, ready for compute.

Input.

API	Return shape	Transpose needed for analytics?
ADBC	pa.Table (columns)	no
ODBC / DB-API	list of tuples (rows)	yes, O(rows × cols)

Code.

# ADBC path — result arrives as an Arrow table (columnar), zero transpose.
import adbc_driver_sqlite.dbapi as sqlite
import pyarrow.compute as pc

conn = sqlite.connect(":memory:")
cur = conn.cursor()
cur.execute("CREATE TABLE t (id INTEGER, amount REAL)")
cur.executemany("INSERT INTO t VALUES (?, ?)", [(1, 10.0), (2, 20.0), (3, 30.0)])
conn.commit()

cur.execute("SELECT id, amount FROM t ORDER BY id")
table = cur.fetch_arrow_table()          # <-- Arrow, columnar, no row-by-row work

print("type       :", type(table).__name__)      # Table
print("columns    :", table.column_names)         # ['id', 'amount']
print("sum(amount):", pc.sum(table["amount"]).as_py())

# Contrast: a classic DB-API path would give rows you must pivot yourself:
# rows = cur.fetchall()           # [(1, 10.0), (2, 20.0), (3, 30.0)]
# ids     = [r[0] for r in rows]  # manual transpose, O(rows) Python work
# amounts = [r[1] for r in rows]

Step-by-step explanation.

1. adbc_driver_sqlite.dbapi.connect opens a DB-API-compatible connection whose cursor can return Arrow. The DB-API surface is familiar (execute, executemany), so ADBC is a drop-in for teams that know ODBC/psycopg2 shapes.
2. cur.fetch_arrow_table() pulls the entire result set as a pa.Table. The driver builds Arrow columns directly from the engine's columnar/native result, so the values never pass through a row-tuple intermediate.
3. Because the result is already columnar, pc.sum(table["amount"]) runs immediately over a contiguous buffer — no [r[1] for r in rows] pivot, no per-row Python objects.
4. The commented ODBC-style path shows the cost ADBC removes: fetchall() yields row tuples, and any columnar consumer must transpose them, which is O(rows × cols) of Python-level work and allocates a Python object per cell.
5. At warehouse scale (Snowflake, BigQuery, Postgres ADBC drivers), this transpose elimination is the difference between a fast columnar handoff and a bottleneck — which is why ADBC exists alongside, not instead of, the SQL you already write.

Output.

type       : Table
columns    : ['id', 'amount']
sum(amount): 60.0

Rule of thumb. When pulling query results into an analytics pipeline, prefer an ADBC driver's fetch_arrow_table() over ODBC/JDBC fetchall() — you skip the row→column transpose and land in Arrow ready for compute, DuckDB, or Polars.

Worked example — Arrow Flight moves batches over the wire, Pandas 2.0 stores them

Detailed explanation. Two ecosystem endpoints in one example: a minimal Arrow Flight server/client (the wire, IPC as payload) and Pandas 2.0's pyarrow backend (the DataFrame that holds Arrow arrays). Together they bracket the stack — remote transport and local storage, both Arrow.

• Flight. Server serves a table via DoGet; client fetches it as Arrow over gRPC.
• Pandas 2.0. The fetched table becomes a pyarrow-backed DataFrame with types_mapper=pd.ArrowDtype.

Question. Serve a table with an in-process Flight server, fetch it from a client, and load the result into a pyarrow-backed pandas DataFrame.

Input.

Component	Role
FlightServerBase.do_get	streams record batches (IPC payload)
client do_get	receives Arrow, read_all()
to_pandas(types_mapper=pd.ArrowDtype)	Arrow-backed pandas columns

Code.

import threading
import pyarrow as pa
import pyarrow.flight as flight
import pandas as pd

DATA = pa.table({"id": pa.array([1, 2, 3]), "amount": pa.array([10.0, 20.0, 30.0])})

class Server(flight.FlightServerBase):
    def do_get(self, context, ticket):
        # The wire payload IS an Arrow IPC stream of record batches.
        return flight.RecordBatchStream(DATA)

# Start the server on a background thread
srv = Server("grpc://127.0.0.1:8815")
threading.Thread(target=srv.serve, daemon=True).start()

# Client fetches the table as Arrow over gRPC (no JSON, no row transpose)
client = flight.connect("grpc://127.0.0.1:8815")
reader = client.do_get(flight.Ticket(b"anything"))
table = reader.read_all()

# Land it in a Pandas 2.0 DataFrame whose columns ARE Arrow arrays
df = table.to_pandas(types_mapper=pd.ArrowDtype)
print("dtypes:")
print(df.dtypes)
print("sum   :", df["amount"].sum())
srv.shutdown()

Step-by-step explanation.

1. FlightServerBase.do_get returns a RecordBatchStream over the table. Flight streams that as Arrow IPC record-batch messages over gRPC — the payload on the socket is the Arrow buffers plus framing, not JSON or a custom protocol.
2. The client's do_get returns a reader; read_all() reassembles the record batches into a pa.Table on the client side. Because the wire format is IPC, reassembly is buffer-pointing, not value-decoding.
3. table.to_pandas(types_mapper=pd.ArrowDtype) builds a pandas DataFrame whose columns are ArrowDtype-typed — i.e. backed by Arrow arrays rather than numpy blocks. df.dtypes shows int64[pyarrow] / double[pyarrow].
4. Operations like df["amount"].sum() now run through Arrow-backed storage, and handing this DataFrame to DuckDB or Polars later is zero-copy because the columns are already Arrow.
5. The example brackets the stack: Flight is Arrow in motion across a network boundary; Pandas 2.0's pyarrow backend is Arrow at rest in a familiar DataFrame — and the same buffers pass between them without a serialization step.

Output.

dtypes:
id        int64[pyarrow]
amount    double[pyarrow]
dtype: object
sum   : 60.0

Rule of thumb. Use Arrow Flight when data must cross a network boundary between Arrow-aware services (its IPC payload keeps the transfer zero-encode), and turn on Pandas 2.0's dtype_backend="pyarrow" (or types_mapper=pd.ArrowDtype) so your DataFrames stay Arrow-native and hand off cleanly to the rest of the stack.

Common beginner mistakes.

• Treating ADBC as "a faster ODBC." The point is the columnar result shape, not raw speed — it removes the row→column transpose entirely.
• Reaching for Flight inside one process. Flight is for a network boundary; in-process handoffs use the C Data Interface, which avoids even IPC framing.
• Leaving pandas on the numpy backend and expecting zero-copy Arrow handoff. Only the pyarrow backend guarantees columns are Arrow arrays; numpy-backed columns must be converted.

Python interview question on composing the Arrow ecosystem

A senior interviewer might ask: "Design the data path for a query service: it runs SQL in DuckDB, serves results to an in-process Polars feature pipeline, and also ships the same results to a remote client. Minimize copies and serialization across all three hops, and name the exact Arrow mechanism at each boundary."

Solution Using Arrow as the universal handoff — DuckDB → Polars (zero-copy) → Flight (IPC on the wire)

import threading
import pyarrow as pa
import pyarrow.flight as flight
import duckdb
import polars as pl

source = pa.table({
    "region": pa.array(["s", "s", "n", "n", "w"]),
    "amount": pa.array([10.0, 5.0, 20.0, 1.0, 7.0], type=pa.float64()),
})

# 1. DuckDB: SQL aggregation, in-place scan of the Arrow table, Arrow out.
agg = duckdb.sql("""
    SELECT region, sum(amount) AS total
    FROM source
    GROUP BY region
""").arrow()

# 2. Polars: zero-copy from Arrow, add a feature column, back to Arrow.
features = (
    pl.from_arrow(agg)
    .with_columns((pl.col("total") / pl.col("total").sum()).alias("share"))
    .to_arrow()
)

# 3. Flight: serve the Arrow result to a remote client; the wire payload is IPC.
class Server(flight.FlightServerBase):
    def do_get(self, context, ticket):
        return flight.RecordBatchStream(features)

srv = Server("grpc://127.0.0.1:8816")
threading.Thread(target=srv.serve, daemon=True).start()

client = flight.connect("grpc://127.0.0.1:8816")
remote = client.do_get(flight.Ticket(b"features")).read_all()
print(remote.to_pydict())
srv.shutdown()

Step-by-step trace.

Hop	Boundary	Mechanism	Bytes serialized
SQL aggregation	in-process	DuckDB replacement scan of source	0 (scan in place)
DuckDB → Polars	in-process	.arrow() + pl.from_arrow	0 (buffer share)
feature derive	in-process	Polars with_columns	result column only
Polars → Flight	in-process	to_arrow()	0 (Arrow-native)
server → client	network	Flight, IPC payload	framed buffers, no re-encode

DuckDB scans the source Arrow buffers in place and emits a small Arrow aggregate. Polars wraps that aggregate zero-copy, adds a share column (the only new allocation), and hands an Arrow table straight to the Flight server. Flight streams it as an IPC record-batch stream over gRPC — the only hop that puts bytes on a socket, and even there the payload is the Arrow buffers plus FlatBuffers framing, not JSON or a row transpose. Every in-process boundary is literally zero-copy.

Output:

region	total	share
s	15.0	0.349
n	21.0	0.488
w	7.0	0.163

Why this works — concept by concept:

• DuckDB replacement scan — resolving source to the local pa.Table lets DuckDB read the Arrow buffers directly, so the SQL layer adds no serialization boundary.
• Polars zero-copy from Arrow — pl.from_arrow and to_arrow bridge via the shared columnar model / C Data Interface, so the SQL→DataFrame boundary copies nothing but the derived column.
• Flight = IPC on the wire — the only network hop uses Arrow IPC as its gRPC payload, so "serialization" is buffer framing, not per-value encoding; the client reassembles by pointing at buffers.
• Arrow as the N×N solvent — because DuckDB, Polars, and Flight all speak Arrow, there is no bespoke connector between any pair; the same buffers thread through all three, which is the whole point of a shared memory standard.
• Cost — O(scanned) for the aggregation, O(result) for the feature column, O(result) for the network transfer. No hop pays O(input) serialization, and peak memory never holds a second copy of the data. That composition — SQL, DataFrame, and RPC with a single columnar currency — is what "Arrow across the stack" buys you.

Streaming
Topic — streaming
Streaming problems on Arrow Flight and wire transport

Practice →

ETL
Topic — etl
ETL problems on cross-engine columnar pipelines

Practice →

---

Cheat sheet — Apache Arrow recipes

• Arrow vs Parquet in one line. Arrow is the in-memory columnar format (uncompressed, aligned, compute-optimized); Parquet is the on-disk columnar format (encoded, compressed, storage-optimized). Reading Parquet decodes into Arrow (pq.read_table returns a pa.Table); writing Arrow encodes into Parquet. They are complements — persist as Parquet, compute as Arrow — never "either/or."
• Buffer counts by type (memorize). Fixed-width primitive = 2 buffers (validity bitmap + data). Variable-length string/binary = 3 buffers (validity + int32 offsets + packed data; large_string uses int64 offsets). list<T> = validity + offsets + one flat child array. struct<...> = validity + one child array per field (no data buffer). Dictionary = integer indices array + dictionary values array.
• Validity bitmap rules. One bit per element, LSB-first, 1 = valid. Element i is bit i of byte i // 8. A column with null_count == 0 may omit the validity buffer entirely — always check for None before indexing buffers()[0]. Never read a value's data bytes without first checking its validity bit.
• Variable-length access is offset math. The i-th string/list element is data[offsets[i] : offsets[i+1]]; its length is offsets[i+1] - offsets[i]. This makes random access and length O(1) with no pointer-chasing, which is why pc.utf8_length scans offsets, not characters.
• Zero-copy mechanism by boundary. Same process, cross-library → C Data Interface / PyCapsule protocol (__arrow_c_array__). Persist + reload locally → Arrow IPC file (Feather V2), optionally pa.memory_map for lazy column-selective reads. Stream batches between processes/sockets → Arrow IPC stream. Remote service → Arrow Flight (its wire payload is IPC).
• IPC stream vs file. Stream = schema message + record-batch messages + EOS marker, for sequential/unbounded delivery (Flight, sockets). File = stream + a footer of batch offsets, seekable and memory-mappable (this is Feather V2). Use stream to move, file to store-and-map.
• Memory-mapped Feather template. feather.write_feather(table, "cache.arrow"); then with pa.memory_map("cache.arrow","r") as s: t = pa.ipc.open_file(s).read_all(). Arrays reference mapped pages; scanning one column faults in only that column's pages — you can open files larger than RAM.
• PyArrow compute cheat codes. Elementwise: pc.add/multiply/greater/equal/if_else/utf8_upper/cast. Aggregate: pc.sum/mean/min_max/count/count_distinct. Selection: pc.filter, pc.take (gather), pc.sort_indices (argsort). Grouped: table.group_by("k").aggregate([("v","sum")]). All kernels are vectorized and null-aware — reach for them before pandas or a Python loop.
• Dataset pushdown template. ds.dataset("path", format="parquet", partitioning="hive").to_table(filter=ds.field("dt")=="2026-08-01", columns=["amount"]). Always pass filter= (partition/row-group pruning) and columns= (projection). For memory-bounded scans, .scanner(filter=..., columns=..., batch_size=N).to_reader() yields RecordBatches; accumulate with per-batch pc.sum so peak RAM is one batch.
• Casting safety. arr.cast(pa.int32()) defaults to safe=True and raises ArrowInvalid on overflow/precision loss — that error is a guardrail, not a bug. safe=False truncates/wraps silently; only use it after you've provably bounded the range. Downcast integer/timestamp widths before writing Parquet to shrink files; re-check the schema after reload.
• C Data Interface verification. After a cross-library handoff, prove zero-copy by comparing consumer.buffers()[1].address == producer.buffers()[1].address. In real C-interface code the consumer must invoke the struct's release callback when done — skipping it leaks the producer's buffers (PyArrow handles release automatically).
• Ecosystem map (place each tool at its layer). Wire = Arrow Flight (+ Flight SQL). Driver = ADBC (columnar query results vs ODBC/JDBC rows). Engines = DuckDB (replacement scan in, .arrow() out) and Polars (pl.from_arrow/to_arrow). DataFrame = Spark (Arrow-accelerated toPandas()/UDFs) and Pandas 2.0 (dtype_backend="pyarrow"). Arrow is the glue that turns N×N connectors into "everyone speaks one format."
• First-minute interview framing. "Apache Arrow is a language-independent, columnar in-memory format. Because the buffer layout is a public spec, systems share tables by passing pointers instead of serializing — that's zero-copy. It's distinct from Parquet (on-disk, compressed); Parquet decodes into Arrow. Zero-copy travels three ways: the C Data Interface in-process, Arrow IPC across a stream or file (Feather + memory-mapping), and Arrow Flight over the wire. That's why DuckDB, Polars, ADBC, Spark, and Pandas 2.0 all interoperate without glue."

Frequently asked questions

What is Apache Arrow in one sentence?

apache arrow is a language-independent, column-oriented memory format for flat and nested tabular data, designed so that any two systems that implement the specification can share a table by referencing the same buffers rather than serializing and deserializing it — which turns cross-system data movement from an O(rows) CPU tax into an O(1) pointer handoff. It is a standard (with implementations in C++, Rust, Java, Go, Python/PyArrow, JavaScript, and more), not a single library or a file format, and it underpins DuckDB, Polars, ADBC drivers, Arrow Flight, Spark's pandas path, and Pandas 2.0's fast dtypes. The core primitives are the Buffer (a contiguous aligned memory block), the Array (a typed column of buffers with a validity bitmap for nulls), the arrow record batch (equal-length columns matching a schema), and the Table (a schema plus one chunked array per column). Every performance claim in the Arrow ecosystem reduces to one idea: a shared, spec-defined columnar layout means the receiver reads the sender's memory directly.

How is Arrow different from Parquet?

Arrow is an in-memory columnar format; Parquet is an on-disk columnar file format — they are complements, not competitors. Arrow optimizes for fast compute and random access in RAM: fixed-width, 64-byte-aligned, uncompressed-by-default buffers that vectorized kernels can scan with SIMD. Parquet optimizes for compact durable storage: dictionary/RLE/bit-packing encodings, block compression (Snappy/Zstd), and row-group + page metadata for predicate pushdown. The relationship is directional — reading a Parquet file decodes its pages into Arrow arrays (pyarrow.parquet.read_table returns a pa.Table), and writing Arrow encodes those arrays into Parquet. The senior-interview red flag is treating them as substitutes ("should we use Arrow or Parquet?"); the correct framing is "persist as Parquet, compute as Arrow, and let PyArrow decode between them." A useful mnemonic: Parquet is what Arrow becomes when it goes to sleep on disk; Arrow is what Parquet becomes when it wakes up in memory.

What does "zero-copy" actually mean in Arrow?

Zero-copy means the number of bytes copied when moving data between two components is literally zero — the receiver reads the sender's existing buffers directly rather than re-encoding the values into a byte stream and re-inflating them. It is possible only because Arrow's buffer layout is a fixed public specification, so a buffer produced by one implementation is valid input to any other. Concretely it shows up three ways: the arrow c data interface shares buffer pointers between libraries in the same process (verified by comparing buffers()[1].address on both sides); arrow ipc framing lets a file or stream mirror the in-memory buffers so a memory-mapped read points at mapped pages; and Arrow Flight carries IPC as its wire payload so a network transfer skips row-to-column transposition. Zero-copy does not mean "no work at all" — building a metadata wrapper and validating the schema are still O(columns) — but it eliminates the O(rows) serialize/deserialize passes that historically dominated pipeline runtime.

When should I use Arrow IPC / Feather instead of Parquet?

Use the Arrow IPC file format (Feather V2) when you want the cheapest possible reload of a table you'll read repeatedly — a local cache, an intermediate checkpoint, a feature file — especially if you'll pa.memory_map it for lazy, column-selective, zero-copy reads. Because the on-disk bytes mirror the in-memory buffers, opening a Feather file barely decodes anything, and mapping a file larger than RAM lets you scan just the columns a query needs. Use Parquet when you want durable, cross-engine, heavily-compressed columnar storage for the data lake — Parquet's encoding and compression make files far smaller, at the cost of decode CPU on read and losing the memory-map benefit. A practical split: Feather for hot, short-lived, single-machine artifacts you reload constantly; Parquet for cold, long-lived, cross-team analytical storage. The IPC stream format (as opposed to the file format) is for moving batches over a socket or between processes, and it's exactly what Arrow Flight puts on the wire.

What is the Arrow C Data Interface, and when do I need it?

The arrow c data interface is a pair of ABI-stable C structs — ArrowSchema and ArrowArray, each with a release callback — that let two libraries in the same process share Arrow data by exchanging pointers, with no serialization and no shared build dependency beyond the struct definitions. A producer fills the structs with pointers to its buffers; a consumer reads them and wraps the same buffers zero-copy; the consumer calls release when finished so the producer knows when it's safe to free. This is the mechanism behind pl.from_arrow, DuckDB's Arrow scans, nanoarrow, and the Python PyCapsule protocol (__arrow_c_array__, __arrow_c_schema__, __arrow_c_stream__). You need it whenever data crosses a language or library boundary inside one process — for example moving a table from a Rust extension (arrow-rs) to Python (PyArrow) via PyO3 — because it avoids even the framing cost of IPC. Reserve IPC and Flight for when bytes must actually leave the process (disk, socket, network).

How does Arrow speed up ADBC, DuckDB, Polars, and Pandas 2.0?

Arrow is the shared columnar currency that lets these tools interoperate without glue. ADBC drivers return query results as Arrow tables (cursor.fetch_arrow_table()) instead of row tuples, deleting the O(rows × cols) transpose that ODBC/JDBC consumers pay to columnarize results. DuckDB reads a PyArrow table zero-copy via a replacement scan and returns results as Arrow (.arrow()), so an in-process SQL step adds no serialization boundary. Polars is built on an Arrow-compatible columnar model, making pl.from_arrow/df.to_arrow near-free and interop with PyArrow a C-Data-Interface handoff. Pandas 2.0 offers dtype_backend="pyarrow", storing columns as Arrow arrays so DataFrames carry Arrow's efficient string/null handling and hand off zero-copy to the rest of the stack; Spark uses Arrow batches to accelerate toPandas() and pandas UDFs, replacing slow row-by-row pickling. The unifying senior point: because every layer speaks Arrow — wire (Flight), driver (ADBC), engine (DuckDB/Polars), DataFrame (Spark/Pandas 2.0) — a value can travel end to end with no serialization boundary between any two hops.

Practice on PipeCode

• Drill the data-processing practice library → for the columnar-layout, compute-kernel, group-by, and null-handling problems that Arrow workloads live on.
• Rehearse on the optimization practice library → for the zero-copy, cache-friendly-scan, and serialization-elimination reasoning senior interviewers probe when Arrow comes up.
• Sharpen the pipeline axis with the ETL practice library → for the partitioned-Parquet, predicate-pushdown, and cross-engine columnar-handoff patterns.
• Practice the streaming practice library → for the Arrow IPC stream, Arrow Flight, and record-batch-reader scenarios that keep memory bounded at scale.
• Stack the prerequisites against PipeCode's broader 450+ data-engineering catalogue to anchor the Arrow-vs-Parquet, buffer-layout, and ecosystem-map decisions against real graded inputs.

Lock in Apache Arrow muscle memory

Docs explain the format. PipeCode drills explain the decision — when Arrow's zero-copy handoff beats a serialize round trip, when a string column is three buffers and why that makes a filter linear, when memory-mapping a Feather file beats reading Parquet, when the C Data Interface is the right boundary and Flight the wrong one. Pipecode.ai is Leetcode for Data Engineering — pattern-first practice tuned for the in-memory-analytics trade-offs senior data engineers actually face.

Practice data-processing problems →
Practice streaming problems →