Most "DuckDB vs Spark" questions are really a sizing question. Spark is a distributed engine: it splits a job across many machines and pays to coordinate them. DuckDB is an in-process engine: it runs the whole job on one machine and pays nothing to coordinate. Which one is faster depends almost entirely on whether one machine can hold the work.
So the useful question is not which engine is better. It is: how many bytes does your job actually touch, and does that fit on one box? Here is the arithmetic, where a cluster earns its cost, and what one node has handled for me. I have not run Spark in production; this is the single-node side of the line, measured, plus what Spark's own documentation says about the other side.
DuckDB vs Spark: what each one is built for
Spark spreads work over a cluster. A driver plans the job, executors on many machines each process part of the data, and a shuffle moves rows between machines when a join or group-by needs them together. If a machine dies, Spark rebuilds the lost piece. Its documentation puts it plainly: if any partition of a cached dataset is lost, "it will automatically be recomputed using the transformations that originally created it." That is what you rent a cluster for: data larger than any one machine, and jobs that keep going when one machine fails.
DuckDB is a columnar SQL engine that runs inside your Python process, a CLI or a service. No driver, no executors, no shuffle over a network. It reads Parquet and CSV in place, uses every core, and spills to disk when a join or sort outgrows memory. The tuning guide lists GROUP BY, JOIN, ORDER BY and window functions as operators that can work larger than memory.
Both read the same Parquet. That matters more than any benchmark: the choice is reversible, because the files don't care which engine opens them.
The sizing math that decides it
The number that matters is the working set: the bytes a job has to process after partition pruning and column selection. It is not the size of the lake.
Three steps get you there.
1. Start from what the job reads, not what exists. A 20 TB events table, partitioned by day, queried for the last 30 days, touching 6 of 60 columns, reads roughly 30/365 × 6/60 of the table. That is about 160 GB on disk before any filter on row values. Parquet compresses, so memory use runs higher than disk size. But most of that work streams through and never has to sit in memory at once.
2. Budget memory by operation. DuckDB's environment guide gives the rule of thumb: about 1–2 GB of memory per thread for aggregation-heavy work and 3–4 GB per thread for join-heavy work. So a 32-thread job needs roughly 32–64 GB for aggregations, or 96–128 GB for joins. That is one ordinary server, not a cluster.
3. Compare with the biggest box you can rent. Single machines are large now. AWS lists an x2iedn.metal with 128 vCPUs and 4,096 GiB of memory. You don't need that. The point is that the ceiling of "one machine" sits far above where most teams assume it is.
Local-First Analytics draws the same line. Chapter 5 suggests DuckDB for up to about 1 TB of partitioned Parquet for aggregation-heavy work, and less for join-heavy work. It also warns that many-to-many joins blow up intermediate state whatever the input size. Its rule of thumb for starting local-first is a working set that fits in 32 GB of RAM and fewer than ten people running queries. Both bars are much lower than where a single node actually gives out.
If your working set after this arithmetic is in the gigabytes or low hundreds of gigabytes, you are running Spark for the cluster's sake, not the data's.
This is the first part. The full post — including the rest of the working details — is on my site: DuckDB vs Spark: which workloads actually need a cluster
Top comments (0)