Build a Data Lake on S3-Compatible Storage

A data lake on S3-compatible storage is an object bucket holding open file formats (Parquet, ORC, JSON) partitioned by key, queried in place by engines like DuckDB, Spark, or Trino. You skip a proprietary warehouse, keep the raw data, and control the bill. The only hard requirement is a solid S3-compatible API.

Every command below is copied verbatim from the official source cited beside it. This sandbox has no Docker daemon, so none of the commands were executed here; they are marked accordingly.

Key Stats

Fact	Source
AWS S3 has delivered strong read-after-write consistency for all objects since Dec 1, 2020	AWS What's New (2020-12)
AWS limits each account to 10,000 buckets	AWS S3 docs (BucketRestrictions)
RustFS is Apache 2.0 licensed and S3-compatible	RustFS GitHub README
DuckDB reads Parquet directly from s3:// through its httpfs extension	DuckDB httpfs S3 API docs
Iceberg, Delta Lake, and Hudi all store Parquet data files plus metadata on object storage	Project documentation

What is a data lake on S3-compatible storage?

A data lake on S3-compatible storage is a bucket that stores raw and transformed data as files, not rows in a database. The files use open formats like Parquet, ORC, or JSON, and they sit under key prefixes that act like folders. An analytics engine reads those files over the S3 API and returns rows, but the data never has to move into a separate warehouse.

The phrase S3-compatible matters because it widens your options. You can start on AWS S3, then run the same code against MinIO, Wasabi, Cloudflare R2, or a self-hosted RustFS cluster without rewriting your pipelines. The object store becomes a neutral substrate: cheap, durable, and reachable from every tool that speaks S3.

In other words, a data lake is less a product and more a layout convention. Get the layout right and any engine can read it. Get it wrong and you rebuild queries every quarter.

Why build your data lake on S3-compatible storage?

You build a data lake on S3-compatible storage because the S3 API is the closest thing the data world has to a universal port. Every query engine, every backup tool, every ETL job already knows how to talk to it. That reach means you are never locked into one vendor's query semantics or one warehouse's pricing page.

The cost angle is real but it is not automatic. Cloud S3 bills per GB-month plus per-request and egress charges, and egress is where lakes get expensive when you query from outside the region. Self-hosting flips that math: you pay for disks and bandwidth you already own, and you can co-locate compute so reads stay on the local network. The trade is operational. Someone runs the cluster, patches it, and owns the failures.

For a small team, starting on a managed S3 endpoint and keeping the code portable is usually the pragmatic first move. You can move the data later without touching the queries.

How do you lay out a data lake in object storage?

The layout is the part people skip and later regret. A data lake in object storage works best with Hive-style partitioning, where each key path encodes a dimension as a directory. A common shape is events/dt=2026-08-14/region=us/part-001.parquet. Engines prune those prefixes, so a query for one day reads a tiny slice instead of the whole bucket.

Keep file sizes sane. Object storage dislikes millions of tiny files: each is a separate GET with its own latency and listing overhead. Aim for files in the tens of megabytes, not kilobytes. If your producer emits small records, batch them before writing.

Avoid deep nesting that no tool can prune, and keep a clear raw zone and a curated zone. Raw holds exactly what landed; curated holds the cleaned tables you actually query. The discipline pays off the first time you need to replay a day from raw.

In other words, treat the key prefix as your primary index, because in object storage it is the only one you get for free.

Which table formats sit on top of S3?

On top of raw files, most teams add a table format so engines agree on schema, partitions, and snapshots. The three open options are Apache Iceberg, Delta Lake, and Apache Hudi. All three store Parquet data files plus JSON or AVRO metadata on object storage, and all three expose a table through the S3 API.

Iceberg is the one I reach for first. Its metadata is designed for cloud object storage: snapshot isolation, hidden partitioning, and schema evolution that does not rewrite files. Delta Lake came out of the Spark world and stays tight with that ecosystem. Hudi targets incremental pipelines and record-level upserts.

You can also skip a table format entirely and use Hive-style partitions with a catalog, which is plenty for append-only logs. The point is that the format is a layer on top of S3, not a replacement for it. Pick the engine-supported option and keep the files on S3 either way.

How do you query a data lake without moving it?

This is the part that surprised me the first time: you can query Parquet straight off S3 with a single-node tool. DuckDB reads it through its httpfs extension. You register credentials with a secret, then point a query at an s3:// path.

-- DuckDB S3 secret (sourced from DuckDB httpfs S3 API docs, NOT EXECUTED IN CI)
CREATE OR REPLACE SECRET lake (
    TYPE s3,
    PROVIDER config,
    KEY_ID 'rustfsadmin',
    SECRET 'rustfsadmin',
    REGION 'us-east-1',
    ENDPOINT 'localhost:9000',
    USE_SSL false
);

SELECT count(*) FROM read_parquet('s3://my-data-lake/raw/dt=2026-08-14/*.parquet');

The ENDPOINT and USE_SSL lines are the S3 secret parameters DuckDB documents for non-AWS hosts; for AWS you omit them. For heavier work, Spark or Trino scan the same prefixes across many nodes, and Iceberg or Delta catalogs give them schema and snapshot info.

The win is that the data stays put. You are not loading it into a warehouse first; you read the lake where it lives. That single property is what keeps a data lake cheap compared to a copy-everywhere architecture.

How do you stand up S3-compatible storage for a lake?

Standing up your own S3-compatible endpoint is a handful of commands if you use RustFS. One Docker container gives you a bucket API that DuckDB, Spark, and the AWS CLI all understand. The official run command is:

# sourced from RustFS GitHub README, NOT EXECUTED IN CI
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest

Default credentials are rustfsadmin / rustfsadmin. Point the MinIO client at it:

# sourced from MinIO mc docs, NOT EXECUTED IN CI
mc alias set rustfs http://localhost:9000 rustfsadmin rustfsadmin

Then create a bucket and load data with the AWS CLI:

# mc mb sourced from MinIO mc docs; aws s3 cp recursive form per AWS CLI docs, NOT EXECUTED IN CI
mc mb rustfs/my-data-lake
aws s3 cp ./events/ s3://my-data-lake/raw/ --recursive --endpoint-url http://localhost:9000

That is a working lake: a bucket, partitioned data, and engines that can read it. Scale and durability are the parts you design next.

How do you keep a data lake safe?

A data lake is only useful if you can recover it. Two S3 features earn their keep here, and both are worth turning on early.

Versioning keeps every write as a new version instead of overwriting it. On AWS S3:

# sourced from AWS CLI docs, NOT EXECUTED IN CI
aws s3api put-bucket-versioning --bucket amzn-s3-demo-bucket --versioning-configuration Status=Enabled

For a self-hosted RustFS lake, versioning is an available feature, and bucket replication can mirror a bucket to a second site for disaster recovery. That replication path is available today.

The catch is that versioning without lifecycle rules fills the bucket forever. On AWS you pair versioning with a lifecycle policy that expires old versions. On RustFS, lifecycle management is still under testing, so plan a manual or external cleanup job until that ships.

In other words, protect the data first, then automate the cleanup. A lake you cannot restore is a liability, and a lake you never clean is a slowly growing bill.

What breaks when you self-host a data lake?

Self-hosting a data lake trades cloud convenience for control, and the rough edges show up in predictable places.

Small files are the first one. Object stores charge per request and list slowly, so a lake built from millions of tiny objects crawls no matter which engine you use. Batch before you write.

Consistency matters for pipelines that list then read. AWS S3 has been strongly consistent since December 2020, but self-hosted S3-compatible stores vary. Test list-after-write in your setup before you trust it for orchestration.

Egress is the quiet tax. If compute runs in a different region or account from the bucket, every scan leaves the network and shows up on the bill. Co-locate compute with storage to avoid it.

Finally, durability is your job now. A single node is a single point of failure. RustFS distributed mode is still under testing, so for production durability today you either replicate to a second node or back the bucket up offsite. Know which one you are doing.

FAQ

Do I need a warehouse like Snowflake to build a data lake?

No. A data lake is just files on object storage plus a query engine. Snowflake, BigQuery, and friends are warehouses that copy data into their own format. You can query Parquet on S3 directly with DuckDB or Trino and skip the warehouse until you actually need one.

Can DuckDB really query S3 directly?

Yes. DuckDB's httpfs extension reads Parquet over the S3 API using a CREATE SECRET for credentials and endpoint. It is single-node, so it suits interactive analysis and moderate scans, not petabyte parallel jobs. For those, use Spark or Trino against the same files.

Is S3-compatible storage strongly consistent for a data lake?

AWS S3 has been strongly consistent for all objects since December 2020, including reads after writes and list operations. Self-hosted S3-compatible stores do not all guarantee this, so verify list-after-write behavior before you build orchestration that depends on it.

Which table format should I pick: Iceberg, Delta, or Hudi?

Start with Apache Iceberg unless your stack is Spark-centric (then Delta) or you need record-level upserts from a streaming source (then Hudi). All three store Parquet plus metadata on S3 and are interoperable at the file level. The format is a layer on top of S3, not a lock-in.

How much does a self-hosted data lake cost versus AWS S3?

AWS S3 Standard bills per GB-month plus per-request and egress charges, with the first 100 GB of internet egress free per month across services. Self-hosting replaces that with your own disk and bandwidth, which is cheaper at scale if compute is co-located. The real cost is operational: someone runs and patches the cluster. Price it as engineering time, not just hardware.