RustFS S3 Tables: A Built-In Iceberg REST Catalog in an Apache-2.0 Object Store

RustFS S3 Tables is an Apache Iceberg REST Catalog built directly into the RustFS object storage kernel, so one process on one port serves both standard S3 and the Iceberg REST Catalog protocol. That removes the separate Hive Metastore or Glue catalog a self-hosted lakehouse normally needs. It is Apache 2.0 with no separate commercial edition.

Item	Value
Feature	Iceberg REST Catalog embedded in the object-store kernel
License	Apache 2.0, no separate commercial edition
Ships in	RustFS 1.0.0
Endpoint	Same port as S3 (9000)
Status	Built into the object-store kernel; fully open source (Apache 2.0)
Iceberg format	v1 and v2 supported (v2 default); v3 unsupported

What is RustFS S3 Tables?

RustFS S3 Tables is a catalog service embedded in the storage server rather than deployed next to it. An Iceberg client uses the REST catalog to discover tables and commit metadata changes. It uses the S3 API to read and write the underlying Parquet files. RustFS serves both interfaces on the S3 API port.

That collapses one moving part: a typical self-hosted lakehouse pairs the object store with a catalog such as Hive Metastore, AWS Glue, Nessie, or Polaris, and RustFS folds the catalog into the store. Because the catalog speaks the open Iceberg REST protocol, engines with an Iceberg REST client need no vendor plugin — Apache Spark, PyIceberg, and DuckDB are documented as connecting without one, and enabling a table bucket does not require registering existing Parquet files first. One consequence worth stating plainly: the catalog's metadata lives inside RustFS itself (Iceberg metadata JSON and manifests written to a reserved prefix on the same bucket), so there is no separate metadata failure domain. Your RustFS durability and backup story is also your catalog backup story, and you cannot fall back to an independent metastore.

The control plane and data plane split

Writes separate into two paths. Table creation and snapshot commits travel through the control plane, which is the Iceberg REST Catalog. Parquet data files are written by the compute engine straight into the bucket over the S3 API, with no intermediate layer in the data path.

Two details keep that boundary safe. Metadata pointers live under a reserved internal prefix, so ordinary S3 requests cannot create, overwrite, or delete catalog metadata; a stray aws s3 cp will not corrupt table state. On commits, the catalog runs compare-and-swap version checks to reject stale concurrent writers and uses idempotency keys to absorb duplicate submissions. The transaction boundary is one table. RustFS does not claim multi-table atomic transactions, and the limit is concrete: you cannot commit a schema change to one table and a data load to a second table in a single atomic operation. Workloads that need cross-table consistency — updating a dimension table and a fact table together, for example — must coordinate that in application logic or accept eventual consistency.

From an empty bucket to a working table

The official walkthrough starts with a dedicated bucket, then flips it into a table bucket with a signed request that carries an empty body. Using my-bucket on localhost:9000:

export RUSTFS_ENDPOINT="http://localhost:9000"
export AWS_ACCESS_KEY_ID="<your-access-key>"
export AWS_SECRET_ACCESS_KEY="<your-secret-key>"
export AWS_DEFAULT_REGION="us-east-1"

aws --endpoint-url "$RUSTFS_ENDPOINT" s3api create-bucket --bucket my-bucket

curl --fail-with-body --silent --show-error \
  --aws-sigv4 "aws:amz:us-east-1:s3" \
  --user "$AWS_ACCESS_KEY_ID:$AWS_SECRET_ACCESS_KEY" \
  --header "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" \
  --request PUT "$RUSTFS_ENDPOINT/iceberg/v1/buckets/my-bucket"

Both the enable call and the GET status check return HTTP 200, with enabled: true and catalog-type: iceberg-rest in the response. I confirmed this end to end on a single-node RustFS 1.0.1 run; see the note at the end for exactly what was executed.

Namespaces and tables are plain REST calls under the warehouse prefix. Namespace creation with curl:

curl --fail-with-body --silent --show-error \
  --aws-sigv4 "aws:amz:us-east-1:s3" \
  --user "$AWS_ACCESS_KEY_ID:$AWS_SECRET_ACCESS_KEY" \
  --header "Content-Type: application/json" \
  --header "x-amz-content-sha256: UNSIGNED-PAYLOAD" \
  --request POST "$RUSTFS_ENDPOINT/iceberg/v1/my-bucket/namespaces" \
  --data '{"namespace":["my-namespace"]}'
# {"namespace":["my-namespace"],"properties":{}}

One signing detail cost me a retry: the enable call signs an empty body, so it carries the well-known empty-sha256 header, but reusing that header on the namespace POST (which carries a JSON body) is rejected with Sha256Mismatch (HTTP 400). Body-carrying requests need x-amz-content-sha256: UNSIGNED-PAYLOAD or the actual body hash instead.

Table creation follows the same pattern against /namespaces/my-namespace/tables and returns the full Iceberg metadata document. Three fields in that response are worth reading: metadata-location points at the metadata JSON under a reserved .rustfs-table/ prefix on the same bucket, which is visible confirmation that catalog state lives in ordinary S3 objects next to your data; metadata.format-version is 2, matching the documented v2 default; and config reports rustfs.credential-vending: unsupported, so the client must hold its own S3 credentials rather than receiving temporary ones from the catalog.

From there an Iceberg client needs four settings. The REST catalog URI is http://localhost:9000/iceberg, and the client appends /v1. The warehouse is the bucket name, not an S3 URI or an AWS ARN. REST auth is SigV4 with signing name s3 and region us-east-1. The S3 file endpoint is the same host, with path-style addressing.

The docs flag one trap that first-timers hit: REST request signing and S3 file access are configured separately, even for the same account. Configure only the S3 side and the client returns 403 until you add the catalog's SigV4 credentials.

Which engines work today?

RustFS maintains a support matrix with different confidence levels per client. Two are automated. PyIceberg covers namespace and table creation, append, reload, scan, and catalog operations. DuckDB Iceberg 1.5.5 covers single-table create, insert, update, delete, and merge, plus schema evolution, snapshots, and concurrent writes.

Apache Spark has an opt-in live harness, so validate your exact Spark and Iceberg versions. Trino has a manual read-only probe and write compatibility is not claimed. StarRocks is documented as an external-catalog read path only. Databend has a live harness limited to an S3-stage read probe of table data files, with no claimed Iceberg REST integration.

At the endpoint level, /iceberg/v1 is the canonical prefix and /_iceberg/v1 is a MinIO AIStor-style compatibility alias. RustFS supports both.

My own run puts one caveat on the matrix's confidence levels. With PyIceberg 0.12.0 against RustFS 1.0.1, the catalog read path works over SigV4: namespace listing and table load return clean responses. The append commit does not. PyIceberg signs catalog requests with botocore's generic SigV4 signer, which sets the S3-style x-amz-content-sha256 header on empty-body requests but not on body-carrying ones; RustFS requires that header on its catalog endpoints, so the commit is rejected with missing header: x-amz-content-sha256. The matrix lists PyIceberg as automated-tested, so treat this as a version-sensitive area rather than a permanent gap: pin your PyIceberg version and run one append before you build on it. Signing is where client compatibility actually breaks today. A separate, still-open issue (#8103, filed September 2026) reports Spark failing metadata-table lookups for a different signing reason: Iceberg's signer double-encodes a URI path that RustFS canonicalizes the S3 way, turning what should be a 404 into a 403.

What's in scope, and what isn't

Before you build on it, here is the scope RustFS documents. Iceberg format v1 and v2 are supported with v2 as the default. Staged table creation, purge-on-drop, and Iceberg v3 are not supported. There is no SQL execution engine, no multi-table atomic transactions, and no independent active-active writes across regions. RustFS implements the open Iceberg REST Catalog protocol and does not claim full AWS S3 Tables control-plane compatibility, so "open-source S3 Tables" means a free, open-protocol Iceberg catalog rather than a drop-in AWS S3 Tables clone. Metadata deletion and background maintenance are off by default, with no built-in periodic scheduler. Dropping a table removes the catalog entry but keeps the underlying objects, so recursive deletion of paths still referenced by snapshots would lose data.

Where the catalog state lives, and how it fails closed

The default catalog backing is object-backed: RustFS persists catalog state as ordinary S3 objects — Iceberg metadata JSON, manifests, and the current pointer file — under a reserved internal prefix on the same bucket, not in a separate embedded database. That is why the backup story for your data is also the backup story for your catalog. The backing is configurable with RUSTFS_TABLE_CATALOG_BACKING; a durable-strong mode exists for deployments that want stronger durability than the default object-backed state, at the cost of a more involved cutover.

Because the catalog is objects plus a CAS-driven pointer, it fails closed by design. The support matrix tracks concrete cases: a stale commit token or stale base metadata returns a conflict without advancing the table pointer; missing metadata, manifest, data, or delete objects fail closed before a commit or maintenance step can advance state; concurrent writers resolve to a single winning CAS with retryable conflicts for the losers; catalog and ordinary S3 permission denials both block a data-plane bypass; and a stale maintenance plan fails closed before any object deletion. Across those cases the direction is the same: the system refuses to advance rather than risk corruption, and each guarantee is tied to a specific RustFS build and client version, not something to assume without testing your exact stack.

Upgrade risk concentrates in the durability-strong cutover. Moving to durable-strong backing requires a preflight reporting SNAPSHOT_MATERIALIZED with no blockers, an exclusive migration fence that drains in-flight writes, and a restart before admitting writers. Snapshot format uses a two-gate switch (RUSTFS_TABLE_CATALOG_STRONG_SNAPSHOT_V2 plus a fleet-confirmed gate); once a version-2 snapshot is persisted, rolling catalog writers back to a binary that only reads version 1 fails closed rather than replacing in-process state. The rollback red line is explicit: after v2 is on disk, do not revert to a v1-only binary. Keep the object-backed catalog backup in your restore set until durable-strong has cleared your retention window, and rehearse the disaster-recovery path on an isolated bucket before it carries data you cannot lose.

RustFS S3 Tables next to the alternatives

MinIO AIStor Tables. MinIO offers a comparable embedded-catalog capability, but only inside AIStor, its commercially licensed product line. AIStor is a separate binary from the AGPL-3.0 MinIO Object Store, and MinIO prices AIStor by capacity at roughly $0.02 per GB per month. The MinIO Object Store entered maintenance mode in December 2025 and no longer takes new features, so the open-source line does not carry this capability. RustFS ships the same class of capability in its Apache-2.0 core, with no community-versus-enterprise split. Licensing and deployment model are where they differ, not feature scope: if you want to drop the separate catalog without taking on a commercial license, RustFS is the lower-friction option, while AIStor brings a more structured commercial-support and compliance story.

Standalone open-source catalogs (Apache Polaris, Project Nessie). Polaris and Nessie are independent, Apache-2.0 REST catalogs you deploy and operate yourself, and both sit in front of any S3-compatible store. They are the genuine alternative if you want catalog/object separation rather than embedding: one catalog can serve several storage backends, and you control its versioning and access policy directly. The cost is exactly the moving part RustFS removes — you run, secure, back up, and upgrade a second service, and you own the wire between catalog and storage. RustFS trades that separation for a single process with one upgrade surface and one backup domain.

On maturity the trade is real: Polaris and Nessie are established projects with broad production adoption and active communities, while RustFS S3 Tables shipped with the 1.0.0 GA and has a thinner third-party deployment track record so far. If an existing operational playbook and catalog maturity weigh more than avoiding a second service, the standalone catalogs are the lower-risk starting point today; RustFS is the bet on a smaller operational footprint that is still closing that gap.

How to choose. The line is "embed vs separate," not "RustFS vs everyone else." If your priority is not running an extra service, RustFS or AIStor fit. If you already standardize on Polaris or Nessie across several backends, RustFS can sit underneath as just the S3 layer. If you need a managed, vendor-run catalog, the comparison shifts to AWS Glue or a cloud warehouse's native catalog rather than self-hosted Polaris/Nessie or RustFS.

Before you take this to production

Pin the version. Use RustFS 1.0.0 or later with a fixed tag rather than latest, and test in an isolated bucket. Give the table bucket its own lifecycle thinking, because table buckets are excluded from ordinary lifecycle expiration and enabling the mode on an existing bucket changes how those rules apply.

Configure both credential paths. Use least-privilege IAM for the catalog actions (admin:SetTableBucket, admin:GetTableBucket, and the namespace and table actions) plus separate S3 permissions for table file reads and writes. Leave metadata deletion and background maintenance off until you have reviewed the retained references, and never recursively delete S3 paths that snapshots may still point to. Start with DuckDB or PyIceberg on a single node to prove create, write, read, and drop before you scale out.

Namespaces group tables, but most of the isolation that matters lands on S3 prefixes, not on the namespace itself. The bucket-level enable response reports credential-scope: warehouse-prefix, and each table's config narrows it to a per-table UUID prefix (s3://my-bucket/tables/<uuid> in my run), which is the scope the catalog expects client credentials to cover. Two consequences follow. You cannot pre-provision a data-access policy for a table's prefix before that table exists, so namespace-level separation gives you organization rather than enforcement. And the realistic granularity options are coarser than the object model suggests: one credential set per producing application, strict prefix conventions, or a bucket per tenant when boundaries must be hard. Keep the catalog admin actions on a separate identity from data-plane writers either way.

Plan metadata maintenance up front. Metadata deletion and background maintenance are off by default and there is no built-in periodic scheduler, so dropping a table removes only the catalog entry and leaves the underlying Parquet files in place — live snapshots keep referencing them and keep occupying storage. Before a table carries real volume, set up a periodic cleanup job (your engine's expire_snapshots / remove_orphan_files, or a scheduled sweep of unreferenced paths) and never recursively delete S3 paths that live snapshots still reference.

The cleanup jobs themselves have teeth. remove_orphan_files deletes any data file current metadata does not reference, so a writer that crashes or pauses mid-commit leaves uncommitted files behind, and a cleanup pass with an aggressive older_than window can delete files a writer is about to commit — the Iceberg docs default that window to days, not minutes, for exactly this reason. expire_snapshots shrinks history on purpose, which is precisely what breaks point-in-time recovery: expire snapshots before your backup has copied them, and a restore lands on a newer state than the one you meant to keep. And every commit writes a fresh metadata JSON plus manifest list, so a table fed by many small commits accumulates metadata objects even while its data barely grows; until snapshots expire, all of it stays in the prefix your catalog reads on every table load.

Treat the catalog's reserved prefix as part of your backup set, since there is no independent metadata store to restore from.

FAQ

Is RustFS S3 Tables free and open source?

Yes. RustFS is Apache 2.0, and S3 Tables ships as a built-in feature of the storage server with no separate commercial edition.

Do I still need Hive Metastore or AWS Glue?

No. The Iceberg REST Catalog runs inside RustFS, so you do not deploy or operate a separate catalog service.

Which query engines work with RustFS S3 Tables?

Any client that speaks the Iceberg REST Catalog protocol. PyIceberg and DuckDB are automated in the RustFS test matrix; Spark and Trino have live harnesses; StarRocks and Databend are narrower.

Is RustFS S3 Tables production-ready?

It is a built-in feature of the Apache-2.0 storage server with no separate commercial edition. As with any feature, validate it against your own workload and keep a rollback path before it carries critical data.

How does RustFS S3 Tables differ from AWS S3 Tables?

AWS S3 Tables is a managed commercial service. RustFS implements the open Iceberg REST Catalog protocol on a self-hosted, Apache-2.0 stack and does not claim full AWS S3 Tables control-plane compatibility.

A note on how this was verified

Article type: hands-on control-plane walkthrough plus a systematic read of the official RustFS documentation, GA announcement, and published support matrix.

The walkthrough above was executed first-hand on a single-node RustFS 1.0.1 (Windows x86_64 binary, release build dated 2026-10-03) on 2026-10-11: bucket creation, table-bucket enablement, status check, namespace creation, table creation, and table load. A PyIceberg 0.12.0 client was also connected over SigV4: catalog reads (namespace listing, table load) succeeded, and the first append commit was rejected with the missing header: x-amz-content-sha256 error quoted in this article. The responses quoted here are taken from those runs, including the Sha256Mismatch rejection. Not exercised: successful engine-side data writes or scans (the commit path blocked them), DuckDB, concurrency benchmarking of metadata commits, large-volume snapshot or metadata-growth stress testing, failure-injection recovery drills, and the durable-strong cutover with its v2 snapshot gate. The operational cautions reflect documented behavior and known limits, not first-hand war stories. Validate against your own workload and keep a rollback path before it carries data you cannot lose.

Sources

1. RustFS GA announcement, "RustFS 1.0.0 GA" (2026-09-16): https://rustfs.com/blog/announcing-rustfs-1-0-0-ga/
2. RustFS S3 Tables documentation (EN): https://docs.rustfs.com/en/administration/data/s3-tables
3. RustFS S3 Tables support matrix (main): https://github.com/rustfs/rustfs/blob/main/docs/architecture/s3-tables-support-matrix.md
4. RustFS blog, "RustFS Launches S3 Tables" (2026-09-10): https://rustfs.com/blog/rustfs-s3-tables-iceberg-rest-catalog-quickstart
5. MinIO AIStor feature set and pricing: https://www.min.io/blog/aistor-overview
6. RustFS release 1.0.1 binaries (Windows x86_64, used for the hands-on run): https://github.com/rustfs/rustfs/releases/tag/1.0.1
7. rustfs/rustfs issue #8103, Spark SigV4 metadata-table failure (open as of 2026-10-11): https://github.com/rustfs/rustfs/issues/8103
8. Apache Iceberg maintenance procedures (remove_orphan_files / expire_snapshots semantics and defaults): https://iceberg.apache.org/docs/latest/spark-procedures/