Querying Parquet on S3 with DuckDB: secrets, speed and the settings that matter

DuckDB reads Parquet on S3 in one line, and that part is not the problem. The problem shows up a month later, when a query that took two seconds on your laptop takes forty against the bucket and nobody can say why.

The answer is almost never bandwidth. It is the number of requests. Object storage charges you a round trip for every file you open and every column chunk you fetch. A layout that costs nothing on local disk can mean tens of thousands of round trips on S3. This post covers credentials first, because they are what people search for, and then the layout and settings that decide speed.

One honest note. My own production pipeline reads Parquet from an NFS share on a NAS, not from S3. The credential and request-level detail here comes from DuckDB's documentation and from my book. The many-files failure near the end is mine, from that NFS pipeline, and it would have hit the same way on S3.

Can DuckDB read from S3?

Yes. The httpfs extension handles S3, and DuckDB autoloads it the first time you touch an s3:// path. You don't need to install or load anything by hand, though doing so in production scripts makes the dependency explicit.

SELECT count(*)
FROM read_parquet('s3://analytics/events/dt=2026-09-*/*.parquet');

For Parquet, DuckDB does not download whole files. Its docs describe partial reading: it reads the file's metadata, then uses HTTP range requests to fetch only the parts the query needs. A count(*) can be answered from metadata alone. That is why Parquet on S3 works at all, and also why the number of files matters so much: every file needs its own metadata read before anything else happens.

Setting S3 credentials: CREATE SECRET

DuckDB manages credentials as secrets. The S3 docs show two providers.

credential_chain asks the AWS SDK for credentials, the same way the AWS CLI would. That covers environment variables, profiles, SSO, assumed roles and instance or container roles. Use it on servers, so no key ever sits in your code:

CREATE OR REPLACE SECRET aws (
    TYPE s3,
    PROVIDER credential_chain
);

config takes the values directly. Use it when the keys come from somewhere else, such as a secrets manager your app already reads. Pass them in from code rather than typing them into SQL:

import os
import duckdb

con = duckdb.connect()
con.execute(
    "CREATE OR REPLACE SECRET aws "
    "(TYPE s3, PROVIDER config, KEY_ID ?, SECRET ?, REGION 'eu-west-2')",
    [os.environ["AWS_ACCESS_KEY_ID"], os.environ["AWS_SECRET_ACCESS_KEY"]],
)

Three details that save time:

• Scope secrets to buckets. Add SCOPE 's3://raw-bucket' and the secret only applies there. With several matching secrets, the secrets manager picks the longest matching prefix. FROM which_secret('s3://raw-bucket/x.parquet', 's3') tells you which one a path will use, and FROM duckdb_secrets() lists them with sensitive values redacted.
• Persistent secrets are plain files. Secrets live in memory for the session by default. CREATE PERSISTENT SECRET writes them to ~/.duckdb/stored_secrets, and the docs say they are "stored in unencrypted binary format on the disk." On a shared server, use credential_chain instead.
• S3-compatible stores need three more fields. For MinIO and similar, set ENDPOINT, URL_STYLE 'path' and, for a local HTTP endpoint, USE_SSL false.

CREATE OR REPLACE SECRET minio (
    TYPE s3,
    PROVIDER config,
    KEY_ID '',
    SECRET '',
    ENDPOINT 'minio:9000',
    URL_STYLE 'path',
    USE_SSL false,
    SCOPE 's3://analytics'
);

If you have older code that uses SET s3_access_key_id = ..., it still runs. Chapter 9 of Local-First Analytics uses that style for its MinIO example. DuckDB's docs now call it a legacy method and say to avoid it, because settings are easy to print by accident. Move to secrets when you next touch that code.

---

This is the first part. The full post — including the rest of the working details — is on my site: Querying Parquet on S3 with DuckDB: secrets, speed and the settings that matter