DEV Community

Cover image for Apache Iceberg & Lakehouse: RustFS S3 Tables in Practice
Ethan Carter
Ethan Carter

Posted on

Apache Iceberg & Lakehouse: RustFS S3 Tables in Practice

#apacheiceberg #lakehouse #s3 #objectstorage

A lakehouse is not a product; it is a three-layer split — a table format (Apache Iceberg), a catalog that knows which tables exist, and object storage that holds the bytes. RustFS's S3 Tables feature contributes the middle layer directly: every bucket can expose a built-in Apache Iceberg REST catalog, on the same server that stores the data. This article walks the stack layer by layer, with results from a hands-on session on RustFS 1.0.1, including the part that failed, which is the part worth reading.

Fact (sourced or first-hand, checked 2026-10-12) Value
Table format in RustFS S3 Tables Apache Iceberg, format-version: 2 (first-hand: create-table response)
Catalog Built-in Iceberg REST catalog, enabled per bucket (/iceberg/v1/<bucket>)
Metadata and data location Same bucket, reserved prefix .rustfs-table/ (first-hand: metadata-location)
Minimum RustFS version 1.0.0 (GA 2026-09-16)
License Apache 2.0
PyIceberg 0.12.0 against it (first-hand) Read path (list/load) works; append commit rejected (missing x-amz-content-sha256 header)

What a lakehouse actually is

A lakehouse is an architecture, and the fastest way to understand it is to name the three layers it separates. The table format — Apache Iceberg in this article — defines how a table's rows, schema, and snapshots are represented as ordinary files. The catalog is the entry point that answers one question reliably: where is the current metadata for table X. Object storage holds both the data files and the metadata files, durably and cheaply.

None of the three layers needs to know about the others' internals, and that is the point. A query engine speaks to the catalog, the catalog points at metadata, the metadata points at data files on object storage. Swap any one layer and the others keep working, which is exactly what has made this split the default design for analytics on top of object storage, and why an S3-compatible store with a built-in catalog changes the deployment math.

One implementation detail is worth knowing before the architecture diagram hardens in your head: in RustFS's S3 Tables the catalog has no separate database behind it. The default object catalog backing persists catalog state in the same object store, under the reserved .rustfs-table/ prefix. The layers are conceptually separate and physically colocated — lose the bucket and the catalog state goes with it, which moves backup planning from optional to load-bearing.

Why the table format became the battleground

The table format is the layer where lakehouse projects compete, and Apache Iceberg is an open table format hosted by the Apache Software Foundation. Its design goal is treating a directory of Parquet files as a full database table: schema evolution, hidden partitioning, time travel via snapshots, and atomic swaps of metadata pointers so readers never see half-written state. Version 2 of the spec added row-level deletes, which is what let merge-on-read engines adopt it for upsert workloads.

RustFS creates Iceberg tables with format-version: 2, confirmed in the create-table response during the hands-on run below. The version 3 specification exists in the Iceberg project, and RustFS's S3 Tables documentation states v3 is not yet supported. That is a plain fact of current scope, the kind worth knowing before you design a partition spec against features that are not there.

Where object storage fits in the stack

Object storage is the layer the lakehouse leans on hardest and trusts the least visibly. Every Iceberg table read touches the catalog, then one metadata file, then the manifest list, the manifest files it names, and finally the data files — a fan-out of small reads against the store. The Iceberg spec asks little of the file system: in-place writes, seekable reads, deletes. Consistency leans on the catalog's atomic metadata swap, not on the store. Engines nonetheless perform many small reads and listings per query, so storage that serves metadata-sized objects quickly is what keeps the architecture responsive.

The storage layer also owns the failure modes. A lakehouse on object storage is only as durable as the store underneath it, which is precisely why the choice of store is an infrastructure decision, not a checkbox. Running it on hardware you administer means the durability story is your replication and backup plan, and no table format changes that arithmetic.

What RustFS S3 Tables provides

RustFS ships an Iceberg REST catalog as a built-in feature, enabled per bucket, on an Apache 2.0 licensed server (requires RustFS 1.0.0+). In a hands-on session on 2026-10-11 against RustFS 1.0.1 (single node, Windows x86_64), the full control-plane chain ran clean: enabling the catalog on a bucket returned enabled: true with catalog-uri: /iceberg/v1/my-bucket; creating a namespace and then a table both returned 200; and the table's metadata-location pointed into the same bucket under the reserved prefix .rustfs-table/, meaning metadata and data live together, with per-table UUID prefixes for isolation.

Two facts from the same session matter for architecture planning. First, the table response reported rustfs.credential-vending: unsupported. Vending is disabled by default (the documentation describes an opt-in path negotiated via X-Iceberg-Access-Delegation), so this session's clients authenticated with long-lived keys and SigV4. The consequence is operational: every engine client holds a full set of S3 credentials, and rotation, scoping, and auditing of those keys are your key-management workload rather than something the catalog hands out per session. The documentation states the preconditions (a compatible client that negotiates the header, a caller permitted to request table credentials) and stops there; its own PyIceberg walkthrough still authenticates with explicitly configured credentials, and the 1.0.1 build behind the hands-on session reported the field as unsupported outright. If vended credentials are a deciding factor for your deployment, verify the full credential path on your build before committing to it. Second, namespaces are organizational, not security boundaries — a tenant cannot be fenced with a namespace alone. Isolation has to come from the storage layer: bucket policies, prefix-scoped IAM, and the table-scoped admin actions the documentation defines. Design multi-team deployments accordingly.

Which engines connect — and the honest status

The catalog speaks the Iceberg REST protocol, so any Iceberg client should be able to point at it. What was actually verified, on the record: PyIceberg 0.12.0 with a SigV4 adapter completed the read path (namespace listing and table load) against the RustFS catalog. The append path failed: the commit was rejected with a missing x-amz-content-sha256 header, because botocore's generic signer only sets that header for empty bodies. That failure is recorded here as-is; if you build on PyIceberg, pin your versions and run one append before you commit to the stack.

The project's own support matrix tracks engine status: PyIceberg and DuckDB marked automated, Spark and Trino covered by a live test harness. Third-party signal cuts both ways. An issue against the catalog documented Spark hitting a SigV4 double-encoding 403 (Iceberg's signer double-encodes the URI path); it was filed in September 2026 and closed on October 5. Deploying a Spark-Iceberg release that contains the fix matters — older releases still exhibit the 403, so pin the version deliberately and verify it, rather than assuming a recent install includes it. Engines connect; the edges are exactly where you would expect them, and testing your engine against your storage version is not optional.

One protocol-level note keeps expectations straight. The Iceberg REST catalog spec leaves authentication to implementation choice — the reference flow uses OAuth2, while this catalog authenticates requests with SigV4 signing, which is exactly why the PyIceberg session above needed a signing adapter. Authorization is not carried by the REST protocol here either: namespace and table operations map to RustFS admin actions enforced at the storage layer, and the documentation states plainly that full AWS S3 Tables control-plane compatibility is not claimed. Treat the REST interface as the common dialect between engines and catalogs, not a promise of identical semantics everywhere.

What this stack does not do

The honest list is short but load-bearing. RustFS S3 Tables includes no SQL engine. There is no embedded query processor; you bring DuckDB, Spark, or Trino, and they speak to the catalog. There are no multi-table transactions: each table commits independently, so cross-table invariants are your application's problem. Table creation is documented as direct create with v2 by default; staged table creation is listed as unsupported. Dropping a table removes the catalog entry but keeps the underlying objects in place, so cleanup of unreferenced files is a maintenance step you run separately.

None of these are hidden defects; they are the documented scope of the feature, and every lakehouse stack makes similar trade-offs somewhere. The difference between a stack you can defend and one that pages you at 2 a.m. is whether the limitations were priced in before the first table was created.

The ops reality of metadata-heavy tables

Iceberg tables accumulate metadata by design: every commit writes a new metadata file, manifests multiply with partition counts, and snapshots pile up until something expires them. The maintenance verbs are part of the format (expire_snapshots to bound history, remove_orphan_files to collect abandoned writes), and both carry footguns documented in the Iceberg project itself: expiring snapshots destroys the time-travel window you may be relying on for recovery, and orphan-file removal with an aggressive older_than window can race with slow in-flight commits and delete live data.

On self-hosted storage, add the capacity angle: metadata from many small commits is a real storage cost, and it lands on the disks you provision. Budget for the metadata prefix the way you budget for the data, and put snapshot expiry on a schedule from day one, not after the first "why is this bucket 3x my dataset" investigation.

The cost is not only capacity. Commit-heavy workloads produce torrents of small reads and writes against the storage layer's metadata path, and the single-node Windows session behind this article exercises that path gently — a clustered deployment with concurrent writers is a different load profile, which is one more reason the verification note below lists multi-node and concurrent-commit testing as not performed. Where the ceiling sits is a fair question with no published answer: no RustFS small-object benchmark at metadata scale exists to cite, and the honest range depends on drives, erasure-set width, network, and client mix. The practical move is to run a small-object mixed-workload benchmark against the exact cluster hardware before a lakehouse lands on it, and treat the measured number, not a datasheet, as the design input.

Backup deserves its own line in the runbook. Because catalog state and table metadata are ordinary objects under .rustfs-table/, a bucket-level backup covers the catalog automatically; a plan that copies only data files does not. It also pays to separate what the storage layer already gives you from what a backup adds. A distributed RustFS cluster stores every object as Reed-Solomon erasure-coded shards across an erasure set, with per-shard checksums and healing, which protects against disk, node, and bitrot loss inside the cluster. It does not protect against a DELETE that should not have happened, a pipeline writing garbage, or corruption applied consistently across shards; those are logical losses, and an independent copy is the only answer. Copy strategies differ in recovery point and blast radius: a scheduled snapshot copy bounds RPO at the schedule interval and restores by re-upload, while a continuously synced second bucket bounds RPO near zero at the cost of propagating mistakes just as fast, deletes included, unless the target keeps versions. The documentation reviewed for this article does not cover a cross-site replication feature, so treat that leg as something to verify on your build or to assemble with an S3-sync tool against the API. Two more facts from the documentation are worth pricing before you need them under pressure. Recovery has an escape hatch: Iceberg clients can register an existing metadata location as a table, so a lost catalog entry is recoverable when the metadata files survive. And there is an explicit upgrade red line: after the durable-strong snapshot format v2 is persisted, rolling back to a binary that only reads v1 fails closed — check the cutover runbook before upgrading any deployment that carries data you cannot lose.

The two catalog backings are a deliberate choice, not a default to forget. The default object backing keeps catalog state in the bucket, and the documentation describes its concurrency model in one line: a commit validates its base metadata and referenced objects before conditionally updating the metadata pointer, a conflicting writer must reload the table and resolve the conflict, and the transaction boundary is one table. The durable-strong mode is the alternative, and the documentation prices its adoption rather than advertising it — a cutover runbook with migration preflight, coordinated writer fencing, and the rollback red line above. No threshold for making the switch is published, so the honest sequence is to run the default backing, watch how conflict retries behave under your real write concurrency, and pay the cutover cost only when the reload-and-resolve path is measurably in the way.

Monitoring has a documentation anchor too. The counters that matter here are derivable from documented behavior: commit conflicts (the reload-and-resolve path), snapshot count per table, and object count plus byte size under the .rustfs-table/ prefix. One more fact raises the stakes: metadata deletion and background maintenance are disabled by default, and RustFS does not run a built-in periodic maintenance scheduler — expiry jobs you schedule yourself only happen if something is watching them happen.

When a lakehouse on self-hosted object storage makes sense

The stack earns its keep in three situations. First: your data governance or cost profile already points at self-hosted storage, and adding a table format plus catalog turns that store into an analytics substrate instead of a backup target. Second: you want Iceberg's engine portability without a hyperscaler's managed-table billing; the same table reads from DuckDB locally and Spark on a cluster. Third: your tables are already large enough that the metadata fan-out matters, which is where a storage layer you can size and monitor yourself pays off.

The built-in catalog is also not the only catalog. The documentation ships a second, explicit pattern: RustFS as the object storage under a separate Iceberg REST catalog service (the external-catalog integration, demonstrated with Spark and a standalone REST container). The built-in catalog is scoped per bucket — its warehouse is a single bucket name — so if one catalog needs to span several buckets or stores, the external pattern is the documented route. The trade is one more stateful service that you run, upgrade, and back up yourself; which pattern fits depends mainly on whether your catalog needs to outlive one bucket.

If none of those hold (small tables, one engine, no storage constraints), a plain bucket with files works, and the lakehouse layer is complexity without a customer. The format is not the product; the workload is.

Frequently Asked Questions

What is an Apache Iceberg lakehouse?

An analytics architecture that separates three layers: the Apache Iceberg table format (how tables are represented as files), a catalog (where table metadata lives), and object storage (where data and metadata are durably kept). The separation lets engines like DuckDB, Spark, and Trino read the same tables without owning the storage.

Do I need a separate catalog server for Iceberg on RustFS?

No. RustFS S3 Tables enables a built-in Iceberg REST catalog per bucket — the hands-on run enabled it on an existing bucket with a single request, and the catalog URI was served from the same server at /iceberg/v1/<bucket>. Metadata files land in the same bucket under the .rustfs-table/ prefix.

Can PyIceberg write to RustFS S3 Tables?

Reads were verified working in a first-hand session (PyIceberg 0.12.0, SigV4). Appends were rejected with a missing x-amz-content-sha256 header error, a signing quirk between botocore's generic signer and the catalog endpoint. Test one append with your pinned versions before building on it.

Does RustFS include a SQL query engine?

No. RustFS S3 Tables provides the table format layer and the catalog; query engines are external — DuckDB, Spark, Trino, or any Iceberg REST client. The project's support matrix tracks which engines are exercised against the catalog.

How is this different from AWS S3 Tables?

AWS S3 Tables is a managed AWS feature: Amazon describes it as fully managed, meaning AWS operates the catalog and its maintenance and you never manage metadata infrastructure yourself. RustFS's implementation is the self-hosted path — the same Iceberg REST interface, on an Apache 2.0 server you run, with the default object catalog backing persisting catalog state in your own bucket. That shifts the metadata failure domain and the backup plan onto your side of the table, which is the deployment trade being made, not a defect ranking.

A note on how this was verified

Article type: architecture analysis anchored on a first-hand verification session. The hands-on results quoted here (catalog enable, namespace and table creation, PyIceberg read success and append failure) were run on 2026-10-11 against RustFS 1.0.1, single node on Windows, with curl SigV4 and PyIceberg 0.12.0; the full method and raw outputs are documented in the companion article on the S3 Tables catalog. Not verified for this article: Spark or Trino against the live catalog, multi-node clusters, concurrent commits, snapshot expiry behavior, and production-scale metadata volume. Iceberg spec and maintenance behaviors are cited from the Apache Iceberg project documentation; check current versions before designing partition specs.

Sources

Top comments (0)