Running DuckDB on your own infrastructure: a production setup

If you're asking how to run DuckDB on your own infrastructure for faster queries, you've already made the important decision. What's left is mostly avoiding four or five specific mistakes that make a single node look slower than it is.

This is the setup I actually run.

Start with the shape, not the settings

The layout matters more than any tuning flag:

object storage (or a local disk)
  └── warehouse/
      └── events/
          dt=2026-08-11/part-0.parquet
          dt=2026-08-12/part-0.parquet
          dt=2026-08-13/part-0.parquet

one writer process   →  produces new partitions
N reader processes   →  open read-only, query across partitions

One writer. Many readers. Immutable partitions. Everything else is detail.

If you take one thing from this: don't put a mutable .duckdb file at the centre of a multi-process system. DuckDB is single-writer, and the moment two processes want to write, you're fighting the design. Write Parquet, read Parquet.

The settings that actually matter

Most DuckDB tuning advice is noise. These four are not:

-- Leave real headroom. This is DuckDB's budget, not the container's.
SET memory_limit = '12GB';

-- Spilling must land on real disk with real space.
SET temp_directory = '/var/lib/duckdb/tmp';

-- Match the cores you actually have, not the ones the host advertises.
SET threads = 8;

-- Only if you're reading from object storage.
SET preserve_insertion_order = false;

memory_limit should sit meaningfully below your container limit — I use roughly 70–75%. DuckDB accounts for its own buffer pool, not for the Python process around it, the Arrow tables in flight, or the runtime. Set it to the container limit and the orchestrator kills you before DuckDB ever decides to spill.

temp_directory is the one that bites in containers. The default may point at a path on the container's ephemeral layer, which is often small and sometimes memory-backed. A large join then fills it and the pod dies looking like an OOM when it's really a disk problem. Mount a volume and point at it.

threads matters in Kubernetes specifically. DuckDB sees the host's core count, not your CPU limit. On a 64-core node with a 4-core limit, it will happily spawn 64 threads and spend its life being throttled. Set this from your actual limit.

preserve_insertion_order = false lets DuckDB parallelise reads more aggressively when you don't care about row order — which, for aggregate queries over Parquet, you usually don't.

---

This is the first part. The full post — including the rest of the working details — is on my site: Running DuckDB on your own infrastructure: a production setup