DEV Community: Ethan Carter

AI Data Storage: Building the Pipeline Behind the Model

Ethan Carter — Wed, 04 Nov 2026 07:22:27 +0000

AI data storage is the layer that holds every file a model touches: raw training media, preprocessed shards, checkpoints, and logs. Most teams run it on S3-compatible object storage because the S3 API is the one interface every ML framework already speaks.

What	Value	Note
S3 layer license	Apache 2.0 (permissive, not AGPL)	RustFS README
S3 API coverage	Core S3 + Object Lock, Versioning, ILM, Replication	RustFS Feature & Status
Default ports	9000 (API) / 9001 (console)	RustFS README
Checkpoint size (derived)	7B params x 2 bytes (fp16) = 14 GB	param count x dtype
Table format option	S3 Tables (Iceberg REST) Preview; Polaris/Nessie in production	RustFS + Apache

What counts as AI data in storage terms

In storage terms, AI data splits into four buckets that get treated very differently. Raw inputs are the source files: images, audio, video, PDFs, or sensor captures. Preprocessed shards are those inputs packed into training-ready chunks, usually tar for media or parquet for tabular, often already tokenized. Checkpoints are periodic snapshots of model weights plus optimizer state, written while training runs. Run logs and evaluation artifacts are the small, high-value files you regret not keeping. Each bucket has its own access pattern. Raw and prepared data is read at high concurrency during training. Checkpoints are write-heavy and bursty. Logs are tiny but you want them durable and queryable. Packing all four into one flat bucket is the most common early mistake, because it forces the storage layer to serve a streaming read workload and a bursty write workload from the same prefix.

Why did object storage win the ML dataset layer

Object storage won the ML dataset layer because it removed the filesystem from the conversation. A training cluster might span dozens of nodes, and a POSIX filesystem that survives that fan-out is expensive to run and easy to corrupt. S3-compatible object storage gives every worker the same URL for the same object, scales the read path horizontally, and costs a fraction of block storage per gigabyte. The kicker is the API itself: boto3, fsspec, PyTorch, and WebDataset all speak S3, so the same code that reads a cloud bucket reads a self-hosted one by changing one endpoint. That portability is why teams keep object storage even after they move off the public cloud. The trade is that object storage is not a filesystem. You do not get POSIX semantics, random in-place edits, or cheap directory renames. For training data, that is usually fine, and the next section shows why.

How does a training-data pipeline actually flow

A training-data pipeline moves through five stages, and most storage decisions happen before a single GPU wakes up. Collection lands raw inputs in an object-store bucket, usually with versioned prefixes so you can reproduce a dataset later. Preprocessing reads those inputs and writes packed shards back to a different prefix, often tar for images or parquet for tabular. Storage then holds both the raw and the prepared layers, which is why capacity planning matters early. Streaming is the hot path: each training worker pulls shards on demand instead of copying the whole set locally. Checkpointing writes weight snapshots on a schedule so a crashed job restarts from minutes, not days, behind. The pattern that works is to keep the raw layer immutable and let the prepared layer be rebuildable. If preprocessing is deterministic and the raw bucket is versioned, you can always regenerate shards after a schema change without re-collecting data.

How do models read data during training

Models read data through three layers, and the bottom one is almost always an S3 client. At the framework level, PyTorch feeds samples through a DataLoader that spins up worker processes to fetch and decode in parallel. Below that, the data is usually read with boto3 or fsspec pointed at an S3 endpoint, which is where a self-hosted store like RustFS slots in by accepting the same API. For large image or multimodal sets, WebDataset streams tar shards so each worker reads one shard at a time instead of listing a million objects. The bottleneck is rarely the model. It is the read path keeping N GPUs fed, which means parallel workers, prefetching, and shards sized so no single object becomes a hot key. A practical RustFS read looks like a standard boto3 client with the endpoint pointed at port 9000:

import boto3

s3 = boto3.client(
    "s3",
    endpoint_url="http://localhost:9000",
    aws_access_key_id="rustfsadmin",
    aws_secret_access_key="rustfsadmin",
)

resp = s3.list_objects_v2(Bucket="training-data", Prefix="shards/")
for obj in resp.get("Contents", []):
    print(obj["Key"], obj["Size"])

The same client reads a cloud bucket by swapping the endpoint and credentials.

Checkpoints and artifacts are the write-heavy half nobody plans for

Checkpoints are where AI storage stops being a read problem and becomes a write problem. A single checkpoint holds the full model weights plus optimizer state, and at fp16 a 7-billion-parameter model is about 14 GB, while a 70-billion model is roughly 140 GB. Teams save them every few hundred steps, so during a long run the storage layer absorbs repeated multi-gigabyte writes while training reads the dataset in parallel. That bursty pattern is what breaks naive single-disk setups. Two habits make it survivable. First, write checkpoints to a bucket with versioning on, so a half-written file from a killed job does not overwrite the last good one. Second, use Object Lock (WORM) on the checkpoint bucket if you need to prove a snapshot is unaltered for audit or regulatory reasons. RustFS supports both versioning and Object Lock, and keeps the metadata for those operations inside the store itself.

Where does RustFS fit as a self-hosted S3 layer

RustFS fits where a team wants the S3 API without sending every training byte to a metered cloud account. It is Apache 2.0, S3-compatible, and built in Rust, with core S3 plus Object Lock, Versioning, Lifecycle, and Replication available today. Setup is one container on ports 9000 and 9001. Because the container runs as a non-root user (UID/GID 10001:10001), make the mounted data and log directories writable by that user before start:

# RustFS runs as non-root UID/GID 10001:10001; make mounts writable first
mkdir -p data logs
chown -R 10001:10001 data logs

docker run -d -p 9000:9000 -p 9001:9001 \
  -v $(pwd)/data:/data \
  -v $(pwd)/logs:/logs \
  rustfs/rustfs:latest

The README documents default credentials rustfsadmin/rustfsadmin; change them before any real deployment. One honest trade: RustFS has no FUSE or POSIX mount, so you use the S3 API, not a local filesystem. That suits pipelines built on boto3 or WebDataset and frustrates tools that demand a mount. The metadata for versioning, lifecycle, and locking lives inside RustFS. That is one fewer service to run, but it also means RustFS reliability is your durability, and backup and recovery are your responsibility, not a vendor's.

The lakehouse edge: Iceberg and the catalog question

If your AI work touches tabular features or training labels at scale, you will meet Apache Iceberg. Iceberg is a table format on top of object storage that tracks snapshots, schema, and partitioning, turning a bucket of parquet files into a queryable table. The catalog is the piece teams underestimate. RustFS ships S3 Tables as an Iceberg REST Catalog, but it is Preview, so I would not bet a production lakehouse on it yet. For that, run a standalone catalog such as Apache Polaris or Project Nessie in front of RustFS: both are Apache-2.0, independent of any storage engine, and mature enough that the Iceberg community builds against them daily. The choice is embed versus separate. Embedding the catalog in your storage keeps moving parts down; running it separately gives you one catalog that can track multiple buckets or storage backends. Choose separate once you have more than one store to manage.

What we would tell a team starting fresh

Start with object storage as the single source of truth and treat the local filesystem as a cache, not the system of record. Keep raw data immutable and versioned, pack it into shards before training, and size shards so no object becomes a hot key. Put checkpoints in their own versioned bucket, switch on Object Lock for auditability, and rehearse a restore before you trust the setup. If you self-host with RustFS, accept the S3 API as the interface and stop looking for a POSIX mount; build on boto3 or WebDataset from day one. Plan around two real gaps: S3 Tables on RustFS is Preview, so use Polaris or Nessie for a production lakehouse, and RustFS holds your metadata, so your backup discipline is the only thing between you and data loss. None of this is exotic. It is the same S3 pattern every cloud ML stack uses, pointed at infrastructure you control.

FAQ

What is the difference between a data lake and object storage for AI?

A data lake is an organizing pattern; object storage is the substrate. You build a data lake by storing raw and prepared data in object storage with consistent prefixes, versioning, and lifecycle rules, then layering query engines or training pipelines on top. Object storage alone is just the bucket.

Do I need a POSIX filesystem mount for training data?

Most modern training pipelines do not. Frameworks read through boto3, fsspec, or WebDataset against an S3 endpoint, which avoids the cost and fragility of a distributed POSIX filesystem. RustFS has no FUSE or POSIX mount, so plan to use the S3 API directly.

How big are model checkpoints, really?

Checkpoint size is the parameter count times the bytes per parameter. At fp16, a 7-billion-parameter model is about 14 GB and a 70-billion model about 140 GB. Optimizer state can add roughly the same again, so budget for double those numbers on long runs.

Can RustFS replace cloud object storage for training?

For the data and checkpoint layers, yes, if you want to control cost and data residency. RustFS is Apache 2.0 and S3-compatible, so existing boto3 and WebDataset code runs against it by changing the endpoint. You take on durability and backup responsibility, since the metadata lives inside the store.

What is S3 Tables and do I need it for AI?

S3 Tables is RustFS's Iceberg REST Catalog for queryable tables on object storage. It is currently Preview, so for a production lakehouse use a standalone catalog like Apache Polaris or Project Nessie in front of RustFS. Most training pipelines do not need it; they read shards directly through the S3 API.

A note on how this was verified

Article type: this is a documentation-driven explainer, not a production deployment report. I pulled the RustFS commands, ports, license, and Feature & Status table verbatim from the project README (checked 2026-10-30); the docker run command is quoted exactly, not paraphrased, and the default credentials are the ones the README documents. The ML framework claims (PyTorch DataLoader, WebDataset, boto3) come from each project's official documentation. I did not stand up a live multi-GPU training cluster or run a throughput benchmark in this environment, and the sandbox here has no Docker daemon, so the RustFS command was not executed locally. The checkpoint size figures are derived from published parameter counts and standard dtype sizes (2 bytes per param at fp16), not measured on RustFS. Treat the architecture guidance as a starting point: validate against your own data sizes and network before committing, and keep versioned backups, because durability for a self-hosted store is a discipline you own.

Sources

RustFS GitHub README (Feature & Status, Docker Quick Start, Accessing RustFS) — https://github.com/rustfs/rustfs (checked 2026-10-30)
RustFS documentation, installation — https://docs.rustfs.com/en/installation
boto3 documentation — https://boto3.amazonaws.com/v1/documentation/api/latest/
WebDataset documentation — https://webdataset.github.io/webdataset/
PyTorch DataLoader — https://pytorch.org/docs/stable/data.html
Apache Iceberg — https://iceberg.apache.org/
Apache Polaris — https://polaris.apache.org/
Project Nessie — https://projectnessie.org/

Spin Up a Self-Hosted S3 Bucket in 5 Minutes with RustFS

Ethan Carter — Sun, 01 Nov 2026 14:09:00 +0000

Last month, Hacktoberfest 2026 had open-source contributors everywhere reaching for a place to stash build artifacts, datasets, and CI caches. The cloud bucket is the obvious answer until you remember it wants a credit card and bills you for egress. I stood up an S3-compatible bucket on my own laptop in about five minutes, no AWS account, and it speaks the exact same API my CI already used against S3. Here is the whole thing, and it works just as well in November as it did in October.

TL;DR

RustFS is Apache 2.0 and S3-compatible, so your existing mc, aws, boto3, and rclone commands just work against it by changing one endpoint.
One container, ports 9000 (API) and 9001 (console), default creds rustfsadmin / rustfsadmin.
Spin it up with a single docker run, open the console, make a bucket, point your tooling at localhost:9000.
It runs as a non-root user (UID/GID 10001:10001), so bind-mounted dirs need the right owner. The compose stack fixes permissions for you if you would rather skip that step.
Want to give back instead of just using the tool? RustFS is open to contributions; the issues and discussions boards are public.

Why a self-hosted S3 bucket

Most open-source projects are run by people who do not want to paste cloud credentials into a fork. A self-hosted S3-compatible bucket solves three real problems at once:

No bill, no card. You store artifacts on hardware you already have. The bytes never leave your machine unless you decide they should.
Your tooling already speaks S3. If your CI uploads to S3 today, it uploads to RustFS by changing one endpoint string. No new client, no rewrite.
Reproducible by anyone. A teammate on a different continent can run the same three commands and get the same bucket. That matters more for an OSS project than for a closed internal one.

The catch is the one we always name out loud: a self-hosted store makes you the durability and backup owner. RustFS keeps its metadata inside the store, so your discipline is the only thing between you and data loss. For a scratch bucket that is a fine trade. For production, rehearse a restore first.

Prerequisites

Docker installed and running.
Enough disk for the image plus whatever you plan to store.

That is it. No Kubernetes, no cluster, no config file to edit for the basic path.

Step 1: Start RustFS

The container runs as a non-root user (rustfs, UID/GID 10001:10001). If you bind-mount host directories, they must be writable by that user, or startup fails with a permission error. The simplest path is to create the dirs and hand them over first:

mkdir -p data logs
chown -R 10001:10001 data logs

docker run -d -p 9000:9000 -p 9001:9001 \
  -v $(pwd)/data:/data \
  -v $(pwd)/logs:/logs \
  rustfs/rustfs:latest

That is the entire "spin it up" step. The image pull is the slow part; the container is up in seconds after that. If you would rather not think about the chown, the project ships a docker-compose-simple.yml that runs an Alpine permission-helper container before RustFS starts, then brings up rustfs-server with five named volumes (four data volumes plus a logs volume) and a health check. Same result, one fewer manual step. One caveat: that helper only chowns the compose's named volumes. If you edit the compose to use host bind mounts instead, you still have to chown those host paths yourself.

Prefer not to touch Docker at all? There is also a one-line installer:

curl -O https://rustfs.com/install_rustfs.sh && bash install_rustfs.sh

Step 2: Open the console and make a bucket

Point a browser at http://localhost:9001. Log in with the documented defaults, rustfsadmin / rustfsadmin, and create a bucket from the console UI. Call it something project-shaped, like hf2026-artifacts.

Change the credentials before this touches anything real. The defaults are documented for exactly this quick-start situation, not for exposure. Set RUSTFS_ACCESS_KEY and RUSTFS_SECRET_KEY (the compose file marks them # CHANGEME) or use IAM policies once you are past the scratch phase.

Step 3: Actually use it

Because RustFS speaks S3, the MinIO client mc works unchanged. Point it at your local endpoint and push a file. (You already created hf2026-artifacts in the console in Step 2, so skip the bucket-creation command and go straight to the copy.)

mc alias set rustfs http://localhost:9000 rustfsadmin rustfsadmin
mc cp ./build.tar.gz rustfs/hf2026-artifacts/

Or from Python, with boto3, pointed at port 9000. Read the credentials from the environment so you are not pasting secrets into source:

import os
import boto3

s3 = boto3.client(
    "s3",
    endpoint_url="http://localhost:9000",
    aws_access_key_id=os.environ["RUSTFS_ACCESS_KEY"],
    aws_secret_access_key=os.environ["RUSTFS_SECRET_KEY"],
)

s3.upload_file("build.tar.gz", "hf2026-artifacts", "build.tar.gz")

If you changed the credentials in Step 2, set RUSTFS_ACCESS_KEY and RUSTFS_SECRET_KEY to those new values. If you left the defaults, they are rustfsadmin / rustfsadmin, the same ones the mc command above uses.

The same client reads a cloud bucket by swapping the endpoint and credentials. That portability is the whole point: your project keeps one storage code path whether it runs on a contributor's laptop or in CI.

Why RustFS specifically

I am not claiming RustFS is the only option. MinIO, Garage, and SeaweedFS all have their place. What makes RustFS a clean fit for this job:

Apache 2.0, not AGPL. Unlike MinIO, which is AGPL-3.0, RustFS ships under the permissive Apache 2.0 license. That is the reason it is comfortable to drop into a forked OSS project without copyleft surprises. (SeaweedFS is also Apache 2.0, so the license is not a differentiator there. RustFS's edge is the Rust-built engine and the MinIO-shaped toolchain.)
S3-compatible, MinIO-shaped toolchain. If you have used MinIO, the commands are the same. RustFS keeps that simplicity and rebuilds the engine in Rust for memory safety and concurrency.
One container, no sidecar. Core S3 plus versioning, Object Lock (WORM), lifecycle, and replication are available today; the metadata lives inside the store, so there is one fewer service to run.
Honest gaps. No FUSE or POSIX mount. You use the S3 API, not a local filesystem. And S3 Tables (the Iceberg REST catalog) is still Preview, so I would not put a production lakehouse on it yet.

Bonus: give back to RustFS (open source runs year-round)

Using the tool is one thing; contributing is another. A timing note: Hacktoberfest 2026 changed its rules, pull requests no longer count toward the event's rewards, with the focus moving to in-person Fests and online challenges. There is no event credit to chase here regardless (the repo carries no hacktoberfest topic). That is fine: the repo is Apache 2.0 and open to contributions 365 days a year. The issues and discussions boards are public:

Issues: https://github.com/rustfs/rustfs/issues
Discussions: https://github.com/rustfs/rustfs/discussions

If you are looking for a storage project to contribute to, filter the issues tab for good first issue or help wanted, or weigh in on an open discussion. A documentation fix or a reproducible bug report is as valuable as a feature PR, just do not expect swag for it.

FAQ

Do I need an AWS account to follow this?

No. RustFS runs locally and emulates the S3 API. The aws CLI and boto3 work against it by pointing at localhost:9000 instead of s3.amazonaws.com.

Is `rustfsadmin` / `rustfsadmin` safe to leave?

No, not past a scratch run. It is the documented default for quick starts. Set your own keys via environment variables or IAM before anything sensitive touches the bucket.

Can I use this as my CI artifact store?

For a weekend or small-scale project, yes. Point your upload step at localhost:9000 (or the host's LAN IP) and the same S3 calls you would use for cloud storage apply. For anything long-lived, add versioning and a real backup target.

What if I need a filesystem mount, not an API?

RustFS has no FUSE or POSIX mount, by design. If your tooling demands a mount, this is not the right fit. Use the S3 API path or pick a store that offers one.

A note on how this was verified

I pulled the commands, ports, default credentials, license, and Feature & Status table verbatim from the RustFS README (checked 2026-11-01). The docker run line is quoted exactly, not paraphrased. The container's non-root UID/GID (10001:10001) comes from the README, which states the server runs as a non-root user rustfs; the compose permission-helper behavior comes from the project's docker-compose-simple.yml. This is a quick-start walkthrough, not a benchmark or a production deployment report. I did not run a throughput test or stand up a multi-node cluster, and the sandbox here has no Docker daemon, so the commands were not executed locally. Validate them on your own machine before relying on them.

Sources

RustFS GitHub README (Quickstart, Accessing RustFS, Feature & Status, License) : https://github.com/rustfs/rustfs (checked 2026-11-01)
RustFS docker-compose-simple.yml : https://github.com/rustfs/rustfs/blob/main/docker-compose-simple.yml
RustFS documentation, installation : https://docs.rustfs.com/en/installation
MinIO client (mc) documentation : https://min.io/docs/minio/linux/reference/minio-mc.html
boto3 documentation : https://boto3.amazonaws.com/v1/documentation/api/latest/
AWS CLI documentation : https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html
rclone S3 documentation : https://rclone.org/s3/
Hacktoberfest 2026 rules (pull requests no longer count toward rewards) : https://hacktoberfest.com/questions/

What are you stashing your project's artifacts in: a cloud bucket, or something you control? If you stood up a local S3 bucket, what tripped you up?

Feeding GPUs from Object Storage: A Self-Hosted S3 Pipeline for Kubernetes MLOps

Ethan Carter — Fri, 30 Oct 2026 11:31:00 +0000

Most AI training pipelines waste GPU cycles waiting on data. The fix is to serve datasets, checkpoints, and model artifacts from an S3-compatible object store that runs inside your own Kubernetes cluster. RustFS gives you that S3 tier under Apache 2.0, with no per-GB egress bill from the store and no vendor lock. This applies to any CNCF-conformant cluster, including OpenShift, EKS, GKE, or a local kind cluster.

Key Stats

Dimension	RustFS
License	Apache 2.0
API surface	S3-compatible (HTTP REST), built in Rust
Kubernetes deployment	K8s Helm Charts Available — install per the official Helm docs, credentials via values or an existing Secret
Credentials	Docker quick start: `rustfsadmin` / `rustfsadmin` (rotate); the Helm chart rejects well-known default credentials
File access	No POSIX/FUSE mount in the Feature & Status table; built-in FTPS and WebDAV (default build), SFTP (opt-in build feature), or third-party s3fs-fuse
Preview features	S3 Tables (Iceberg REST), MinIO on-disk compatibility

What does feeding GPUs from object storage mean?

Feeding GPUs from object storage means treating your S3 bucket as the single source of truth for everything a training job touches: the raw corpus, the shuffled shards, the mid-epoch checkpoints, and the final model weights. The GPU node pulls an object over the network, decodes it in RAM, and never needs a POSIX filesystem underneath. In practice this looks like a PyTorch or TensorFlow worker calling the S3 API through boto3 or fsspec rather than opening a local path. The mental shift is from "mount a volume" to "address a key." RustFS is an Apache 2.0, S3-compatible server built in Rust, so your existing S3 tooling — aws s3, mc, the Python SDKs — works unchanged. We treat the object store as the data plane and leave POSIX concerns to a gateway or a FUSE shim.

Why not just use NFS or local NVMe for training data?

Local NVMe is fast but local: it does not follow the pod when the scheduler reschedules a failed worker, and it turns every node into a data silo you must sync by hand. NFS gives you shared, ReadWriteMany files, but random reads of millions of small shards hammer the filer, and a single appliance becomes the throughput ceiling and the single point of failure. Object storage sidesteps both: data lives behind a key, scales horizontally, and any pod in any zone can read the same object through the S3 endpoint. Object reads are network round-trips rather than local seeks, so you stage hot shards into RAM or a node-local cache instead of streaming every byte remotely. Pick the primitive that matches the workload: NVMe for scratch, object storage for the durable corpus.

Where RustFS fits: a self-hosted S3 tier

RustFS is the S3 layer you run yourself, in the same cluster as your training jobs. Its Feature & Status table lists S3 Core, Versioning, Object Lock (WORM), Lifecycle, Distributed Mode, IAM, SSE, KMS, and the K8s Helm Charts all as Available, so you get object versioning and lifecycle rules for free. It is Apache 2.0, so there is no per-GB egress fee and no contract locking you to one cloud. The honest boundary: RustFS has no POSIX or FUSE mount in its feature table — path-based access goes through its built-in FTPS and WebDAV gateways (both enabled in the default build), SFTP (an opt-in cargo feature), or a third-party FUSE shim like s3fs-fuse. S3 Tables (Iceberg REST) and MinIO on-disk compatibility are still Preview. If your pipeline is Iceberg-native and needs a production catalog today, Apache Polaris and Project Nessie are mature, Apache-2.0 catalogs you can run separately; we treat RustFS's own Iceberg support as not-yet-GA and say so plainly.

How do you deploy RustFS on Kubernetes?

RustFS ships a Helm chart, marked Available in its Feature & Status table. Both documented chart-access routes in the official Helm installation docs (docs.rustfs.com) end by setting the chart reference the install commands use — don't skip that line, or the install command gets an empty variable:

helm repo add rustfs https://charts.rustfs.com
helm repo update
export RUSTFS_CHART=rustfs/rustfs

One thing the chart itself makes non-negotiable: it refuses to install with the well-known rustfsadmin / rustfsadmin defaults. That guard lives in the chart's templates/secret.yaml, which fails rendering with "…must be set to non-default, non-empty values…" unless you set real keys or explicitly opt in with secret.allowInsecureDefaults=true (local development only). The docs' standalone example supplies real keys through a values file and installs with:

helm upgrade --install rustfs "$RUSTFS_CHART" \
  --namespace rustfs \
  --create-namespace \
  -f standalone-values.yaml

That standalone-values.yaml (verbatim from the docs) enables one-Pod standalone mode, sets the Secret keys, pins StorageClass and sizes, and disables the Ingress — worth keeping, because the chart's default is ingress.enabled: true with className: "nginx", and a kind or OpenShift cluster without an nginx controller would just get an idle Ingress resource:

mode:
  standalone:
    enabled: true
  distributed:
    enabled: false
secret:
  rustfs:
    access_key: "<your-access-key>"
    secret_key: "<your-secret-key>"
storageclass:
  name: standard
  dataStorageSize: 10Gi
  logStorageSize: 1Gi
ingress:
  enabled: false

For a quick local check before you touch a real cluster, the project's Docker quick start (verbatim from the README) runs the container as a non-root user with UID/GID 10001:10001:

mkdir -p data logs
chown -R 10001:10001 data logs
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest

The Docker image still answers to the README's documented default rustfsadmin / rustfsadmin — rotate them before anything shared. In Kubernetes, what pods actually dial is a Service name. Per the chart's templates/service.yaml, the ClusterIP Service is {{ include "rustfs.fullname" . }}-svc: the chart's fullname plus an -svc suffix. The fullname equals the release name only when the release name already contains the chart name — helm install rustfs ... gives rustfs-svc, but helm install my-store ... gives my-store-rustfs-svc. The Service exposes the S3 API on 9000 and the Console on 9001, with a headless -headless Service only in distributed mode. For release name rustfs in namespace rustfs, the in-cluster endpoint is http://rustfs-svc.rustfs.svc:9000. One correction worth passing on: the docs' verify step reads kubectl -n rustfs port-forward svc/rustfs 9000:9000 9001:9001, but this chart creates no Service named rustfs — use svc/rustfs-svc or the port-forward fails to resolve.

How do training pods actually read from RustFS?

Pods talk to RustFS over the S3 API, not through a volume. The cleanest pattern is to inject the access and secret key from a Kubernetes Secret as environment variables and point the AWS SDK at the in-cluster endpoint — boto3's default credential chain picks up AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY, so no keys land in the image or the code. A minimal boto3 pull before the epoch looks like this:

import os, boto3
s3 = boto3.client(
    "s3",
    endpoint_url="http://rustfs-svc.rustfs.svc:9000",
    aws_access_key_id=os.environ["RUSTFS_ACCESS_KEY"],
    aws_secret_access_key=os.environ["RUSTFS_SECRET_KEY"],
)
s3.download_file("datasets", "imagenet-shards/train-0001.tar", "/tmp/train-0001.tar")

For lazy, key-addressed reads without downloading first, fsspec plus s3fs (pip install fsspec s3fs) opens remote objects directly through the same endpoint:

import s3fs
fs = s3fs.S3FileSystem(
    key=os.environ["RUSTFS_ACCESS_KEY"],
    secret=os.environ["RUSTFS_SECRET_KEY"],
    client_kwargs={"endpoint_url": "http://rustfs-svc.rustfs.svc:9000"},
)

Because RustFS is S3-compatible, you do not rewrite your data loader — you change the endpoint_url and the credentials. The operational rule we follow: keep credentials in a Secret, never in the image or the code, and prefer versioned buckets so a bad overwrite is one versionId away from recovery.

Checkpoints, artifacts, and the model registry bucket

A single bucket rarely fits every job, so we split the data plane by purpose. One versioned bucket holds the raw and preprocessed corpus; a second holds checkpoints, written every N steps so a preempted pod resumes instead of restarting; a third holds the model registry, where each published artifact is an immutable object keyed by version. RustFS Object Lock (WORM) is Available, so the registry bucket can be locked to prevent accidental deletion of a released model. Lifecycle rules (also Available) move stale checkpoints to cheaper tiers or expire them after a retention window, which matters once you are checkpointing every job on every node. The pattern scales because object storage has no per-node ceiling — grow capacity by adding RustFS nodes, not by buying a bigger filer for the training subnet.

What about a filesystem view? Built-in gateways first, s3fs second

Before reaching for FUSE, check whether the legacy tool speaks a protocol RustFS already serves: the Feature & Status table marks FTPS and WebDAV as Available (both enabled in the default build) and SFTP as Available (an opt-in cargo feature). A tool that can read ftp:// or webdav:// paths needs no FUSE at all. When something genuinely requires a POSIX-style bucket mount, s3fs-fuse earns its place. s3fs-fuse is a FUSE filesystem backed by Amazon S3 and other S3-compatible stores, licensed GPLv2; you mount a bucket as a local directory and point the legacy tool at it:

s3fs mybucket /path/to/mountpoint -o passwd_file=${HOME}/.passwd-s3fs -o url=http://rustfs-svc.rustfs.svc:9000 -o use_path_request_style

Its own README is blunt about the cost: random writes or appends rewrite the entire object, directory listings suffer network latency, there are no atomic renames, no hard links, and no coordination between multiple clients mounting the same bucket. So s3fs is a read-mostly or write-once shim, not a POSIX replacement. Use it for the one tool that cannot speak S3; keep the hot training path on the native S3 API where the throughput actually is.

A reference pipeline you can copy: kind + RustFS + a shard-pull Job

You can stand up the whole loop on a laptop with kind. Create the cluster, add the chart repo, and install RustFS with credentials from a values file (see the deploy section above):

kind create cluster --wait 30s
helm repo add rustfs https://charts.rustfs.com
helm repo update
export RUSTFS_CHART=rustfs/rustfs

Then create the Secret and a Job that pulls a shard from RustFS over S3 before training. This manifest is our own minimal working shape — it proves the connectivity, not the training loop. Uncomment the GPU lines on GPU nodes (scheduling needs the NVIDIA device plugin); adapt names and images to your cluster:

apiVersion: v1
kind: Secret
metadata:
  name: rustfs-creds
  namespace: default
stringData:
  RUSTFS_ACCESS_KEY: "<your-access-key>"
  RUSTFS_SECRET_KEY: "<your-secret-key>"
---
apiVersion: batch/v1
kind: Job
metadata:
  name: shard-pull-train
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: train
          image: pytorch/pytorch:2.4.1-cuda12.1-cudnn9-runtime  # or your preferred PyTorch image
          envFrom:
            - secretRef:
                name: rustfs-creds
          # Uncomment on GPU nodes (requires the NVIDIA device plugin):
          # resources:
          #   limits:
          #     nvidia.com/gpu: 1
          # nodeSelector:
          #   nvidia.com/gpu.present: "true"
          command: ["python", "-c"]
          args:
            - |
              import os, boto3
              s3 = boto3.client(
                  "s3",
                  endpoint_url="http://rustfs-svc.rustfs.svc:9000",
                  aws_access_key_id=os.environ["RUSTFS_ACCESS_KEY"],
                  aws_secret_access_key=os.environ["RUSTFS_SECRET_KEY"])
              s3.download_file("datasets",
                              "imagenet-shards/train-0001.tar",
                              "/tmp/train-0001.tar")
              print("shard ready, training starts here")

The pattern is identical on OpenShift, EKS, or GKE — only the ingress, SCC, and node classes differ (see the OpenShift section below). RustFS Distributed Mode is Available, so when one node's disks fill up you add nodes instead of migrating. We keep the corpus in a versioned bucket, the checkpoints in a second bucket with a lifecycle rule, and the registry in a third with Object Lock on. That is the entire data plane: one S3 tier, three buckets, and no per-GB egress bill from the store itself.

What actually changes on OpenShift

The same chart and the same rustfs-svc endpoint pattern apply — OpenShift is Red Hat's CNCF-certified Kubernetes distribution. Two OpenShift-specific pieces do change, and neither is optional in the default configuration:

SecurityContextConstraints (SCC). The chart hardcodes podSecurityContext with runAsUser / runAsGroup / fsGroup of 10001 (confirmed in the chart's values.yaml). OpenShift's default restricted-v2 SCC validates UIDs against the range assigned to the namespace (Run As User Strategy: MustRunAsRange) and validates fsGroup the same way (FSGroup Strategy: MustRunAs), so hardcoded 10001 values can fall outside those ranges and be rejected at admission. You have two supported ways out: override podSecurityContext / containerSecurityContext in your values file — dropping runAsUser, runAsGroup, and fsGroup together, so every range-based strategy can apply — or bind the RustFS service account to a nonroot-class SCC. Dropping only runAsUser is not enough; a stale fsGroup: 10001 gets the pod rejected on its own. Either way, the deployment on OpenShift is a values change, not an unchanged install.
Route for external access. If clients outside the cluster need the S3 API or Console, expose them with an OpenShift Route (the chart's Ingress support covers in-cluster alternatives via ingress.className). Traffic that stays inside the cluster — training pods pulling shards — needs neither a Route nor an Ingress; it goes straight to rustfs-svc.

Deeper OpenShift AI integration — workbenches reading the bucket, KServe model servers pulling weights — follows the same endpoint pattern, but we have not exercised RHOAI specifically and do not claim it here.

FAQ

Is RustFS a drop-in for S3 in my training code?

Yes for the S3 API your data loaders use. Point boto3, fsspec, or the AWS CLI at the RustFS endpoint with its access key and secret, and your existing loaders work without code changes. The official S3 compatibility matrix names the boundaries: ACL authorization is intentionally unsupported, and server-side-encrypted objects are implementation-specific (MinIO-encrypted objects are not readable by RustFS in default builds), so check the matrix before migrating encrypted or ACL-heavy buckets. Tools that require a mounted path need one of the file-access routes above, not raw RustFS.

Can I run this on OpenShift specifically?

Yes, with two OpenShift-specific adjustments, detailed in the body: the chart's hardcoded runAsUser / runAsGroup / fsGroup of 10001 must be reconciled with the namespace's SCC ranges (override all three in values or bind a nonroot-class SCC), and external access uses a Route. The S3 endpoint pattern for pods (rustfs-svc) is unchanged.

Do I need a CSI driver to mount RustFS?

No. RustFS is reached over the S3 API, so there is no native volume type and no CSI driver required. Pods call the endpoint with SDKs or mc. If a specific tool needs a filesystem view, try RustFS's built-in FTPS or WebDAV gateways first, then s3fs-fuse as a sidecar rather than looking for a RustFS CSI driver.

How do I avoid egress and lock-in?

Because RustFS runs inside your own cluster under Apache 2.0, reads and writes between pods and the store never leave your cluster network, so the store adds no per-GB egress bill and no cloud contract. One honest caveat: if the cluster itself runs on a public cloud, cross-AZ or cross-node traffic still bills at the infrastructure layer — self-hosting the store removes the storage egress line, not the whole network bill.

What is still maturing on RustFS?

S3 Tables (Iceberg REST) and MinIO on-disk compatibility are Preview, not GA. The core S3 surface, Versioning, Object Lock, Lifecycle, Distributed Mode, IAM, SSE, and the K8s Helm Charts are all Available. Plan Iceberg-heavy workloads around a separate, mature catalog until RustFS's Iceberg support graduates.

A note on how this was verified

This article is a systematic reading of official documentation and the S3 compatibility matrix, plus direct inspection of the rustfs-1.0.4 Helm chart package from charts.rustfs.com (we unpacked it and read templates/service.yaml for the Service names and values.yaml for the securityContext values). It is not a production deployment report. Every AWS/OpenShift behavior quoted above was checked against the sources listed below (first pass 2026-10-21; the chart package, the Helm docs pages, and the SCC strategies re-checked 2026-10-27). What we did not do, and you should, before relying on this in production: we did not provision an actual RustFS instance, kind cluster, or Kubernetes training Job in the writing environment (no container runtime was available), so the commands were not executed end-to-end here; we did not run a distributed, multi-node RustFS deployment; we did not run a concurrent-throughput or failure-injection benchmark; and we did not exercise OpenShift AI (RHOAI) or KServe. The Job manifest and boto3/s3fs snippets are our own minimal working shapes, not copied from a vendor document — validate them in your cluster. Keep a rollback (versioned buckets make this cheap), and confirm the RustFS feature status against the README at your install time, since Preview features graduate.

Sources

RustFS README — Docker quick start, default credentials, Feature & Status table (FTPS / WebDAV / SFTP rows), license: https://github.com/rustfs/rustfs (checked 2026-10-21; FTPS/WebDAV/SFTP rows re-checked 2026-10-27)
RustFS Helm installation docs — chart-access routes (helm repo add / helm repo update / export RUSTFS_CHART=rustfs/rustfs), standalone values file, verify/port-forward steps. Two errata we correct in the body: the docs' port-forward example says svc/rustfs, which does not exist in this chart (use svc/rustfs-svc), and the credential guard is not described on this page — it lives in the chart's templates/secret.yaml: https://docs.rustfs.com/en/installation/cloud-native/helm-chart/installation (checked 2026-10-27)
RustFS Helm chart package rustfs-1.0.4.tgz — templates/service.yaml (Service names {{ fullname }}-svc, {{ fullname }}-headless in distributed mode), values.yaml (podSecurityContext hardcoded 10001), and templates/secret.yaml (fail message rejecting empty or well-known default credentials; secret.allowInsecureDefaults=true is the documented local-dev opt-in): https://charts.rustfs.com/rustfs-1.0.4.tgz (unpacked and inspected 2026-10-27)
RustFS S3 compatibility matrix — ACL authorization intentionally unsupported, SSE object-format specifics, excluded-tests mechanism: https://github.com/rustfs/rustfs/blob/main/docs/architecture/s3-compatibility-matrix.md (checked 2026-10-27)
OpenShift documentation — SecurityContextConstraints strategies; restricted-v2 uses Run As User Strategy: MustRunAsRange and FSGroup Strategy: MustRunAs: https://docs.redhat.com/en/documentation/openshift_container_platform/4.17/html/authentication_and_authorization/managing-pod-security-policies (checked 2026-10-27)
kind Quick Start — install and kind create cluster: https://kind.sigs.k8s.io/docs/user/quick-start/ (checked 2026-10-21)
s3fs-fuse README — mount syntax and limitations: https://github.com/s3fs-fuse/s3fs-fuse (checked 2026-10-21)
AWS SDK for Python (boto3) — S3 client usage and default credential chain: https://boto3.amazonaws.com/v1/documentation/api/latest/ (checked 2026-10-21)

Object Lock Isn't Absolute: What WORM Really Blocks on S3 and Self-Hosted RustFS

Ethan Carter — Tue, 27 Oct 2026 13:43:00 +0000

Data preservation is keeping data exact on purpose

Data preservation is the practice of storing objects so they stay exact, recoverable, and unchangeable on purpose. Backups answer "can I get the data back?" Preservation answers a harder question: "can anyone, including me, alter or delete this before a date I chose?" Three primitives do the work. Immutability (WORM) stops overwrite and delete. Versioning keeps every revision so an edit is never a destruction. Retention enforces a minimum keep time. On AWS these come from S3 Object Lock and bucket versioning. On self-hosted S3-compatible storage like RustFS, the same S3 API now exposes all three. This article walks the concepts, the AWS commands, and a real lock-and-try-to-delete run on RustFS 1.0.3.

Capability	RustFS (self-hosted, Apache 2.0)	AWS S3 (managed)
Object Lock (WORM)	✅ Available	✅ Available
Bucket versioning	✅ Available	✅ Available (required for Object Lock)
Governance + Compliance modes	✅ Available (S3 API)	✅ Both
Legal hold (indefinite)	✅ Available (S3 API)	✅ Available
Default bucket retention	✅ Available (S3 API)	✅ Available
Lifecycle / ILM tie-in	✅ Available	✅ Available
Compliance assessment	—	Cohasset: SEC 17a-4, CFTC, FINRA

All four S3 API lock behaviors in the RustFS column (governance mode, compliance mode, legal hold, default bucket retention) were also exercised hands-on in this article on RustFS 1.0.3 — see the hands-on section below.

Why immutability beats a backup against ransomware

A backup protects against loss. It does not protect against a credential that deletes the backup too. WORM protection closes that gap: once an object version is locked, deletes and overwrites that target that version return an error until the retention period ends — a plain delete without a version id still adds a delete marker on top of the history. Ransomware operators and a careless script both hit the same wall. The second reason is regulatory. Rules like SEC 17a-4, FINRA, and CFTC expect records to be non-rewriteable and non-erasable for a set window, and they expect the storage to enforce it rather than trust a policy document. A snapshot you can delete by accident is not the same control. Immutability is the primitive that turns "we try not to delete it" into "the system refuses to delete it."

WORM and Object Lock: what governance and compliance modes actually do

Object Lock gives an object version a lock that blocks overwrite and deletion. You pick one of two modes. Governance mode blocks deletes for most users but can be overridden by anyone holding s3:BypassGovernanceRetention who also sends the x-amz-bypass-governance-retention: true header. That makes it good for testing retention rules before you commit to harder enforcement. Compliance mode is stricter: no user, including the account root, can shorten the period or delete the object until it expires. The only escape AWS documents is deleting the entire account. On top of modes, a legal hold gives the same protection with no expiration date; it stays until someone with s3:PutObjectLegalHold removes it. Legal holds and retention periods are independent, so an object can be protected by both at once.

Versioning: the safety net Object Lock depends on

Object Lock only works on buckets that have versioning enabled, because the lock is stored against a specific object version. Versioning is also a preservation tool on its own. Without it, an overwrite replaces bytes in place; with it, every PUT creates a new version and the old one survives. A delete becomes a delete marker on top of the history rather than a wipe. That matters for ransomware recovery too: if an attacker encrypts a file, the prior clean version is still there to restore. The cost is storage growth and list noise, which is exactly what lifecycle rules exist to manage. The takeaway is simple: turn versioning on first, then layer Object Lock on top. The two are not alternatives; Object Lock is enforced per version, and there is no version without versioning.

Retention periods vs legal holds: fixed dates or indefinite?

A fixed retention period sets a retain-until-date. You can specify it directly on an object, or set a bucket default (a mode plus a number of days or years) so every new object is protected automatically. When you PUT an object with its own explicit retention, that explicit setting overrides the bucket default. A legal hold is different: it has no clock. It protects an object version until an authorized user removes it, and it keeps protecting the object even after a retention period expires. The practical split is operational vs legal. Use fixed retention for "keep audits 7 years." Use legal hold for "do not touch this until counsel says so." Both can sit on the same object version, and removing one does not remove the other.

AWS also documents a third shape, variable retention: instead of picking a date up front, you turn on an event hold with an event hold duration, and the object stays protected while the hold is on. When the hold is released, Amazon S3 fixes the retain-until-date at the release time plus the duration. This fits cases where you cannot know the keep-time at write time — for example, a configurable ransomware recovery window. A bucket default can turn on an event hold too, so every new object starts with variable retention instead of a fixed date.

How AWS S3 handles immutability

On AWS the primitives are managed, so the work is configuration rather than operations. Object Lock works only on buckets with S3 Versioning enabled, and you can enable it two ways: at bucket creation, or on an existing bucket (via the console, AWS CLI, SDKs, or the REST API). Note that once Object Lock is enabled on a bucket, you cannot disable it or suspend versioning on that bucket:

# Enable Object Lock at bucket creation (implies versioning requirement)
aws s3api create-bucket --bucket amzn-s3-demo-bucket1 --object-lock-enabled-for-bucket

# Set a 50-day COMPLIANCE default retention on the bucket
aws s3api put-object-lock-configuration --bucket amzn-s3-demo-bucket1 \
  --object-lock-configuration='{ "ObjectLockEnabled": "Enabled", "Rule": { "DefaultRetention": { "Mode": "COMPLIANCE", "Days": 50 }}}'

# Place or remove a legal hold on an object
aws s3api put-object-legal-hold --bucket amzn-s3-demo-bucket1 \
  --key my-image.fs --legal-hold="Status=ON"

AWS states its Object Lock has been assessed by Cohasset Associates for SEC 17a-4, CFTC, and FINRA use. The durability and availability numbers AWS publishes are AWS's own claims; they do not transfer to a self-hosted deployment.

Running WORM on self-hosted RustFS

RustFS is Apache 2.0 and speaks the S3 API, so the same lock concepts apply to a server you run yourself. The quick start maps the API ports and writes data and logs to local volumes:

# Create data and logs directories; the container runs as a non-root
# user (rustfs, UID/GID 10001:10001), so the mounts must be writable by it
mkdir -p data logs
chown -R 10001:10001 data logs

docker run -d -p 9000:9000 -p 9001:9001 \
  -v $(pwd)/data:/data -v $(pwd)/logs:/logs \
  rustfs/rustfs:latest

Default credentials are rustfsadmin / rustfsadmin — change them before exposing the server beyond localhost. Point a MinIO Client (mc) or the AWS SDK at it, then configure locking the same way you would against any S3 endpoint. The next section is a real run on the Windows build of RustFS 1.0.3 with the AWS SDK (boto3), showing what happens when you try to delete a locked object.

Hands-on: lock a bucket and try to delete a protected object

This is a real session against RustFS 1.0.3, started from the native Windows binary (rustfs.exe server <dir> --access-key rustfsadmin --secret-key rustfsadmin). The client is the AWS SDK for Python (boto3) pointed at the local endpoint; the equivalent mc calls are noted inline.

# 1. Create a bucket with Object Lock enabled (mc: mc mb --with-lock rustfs/preserve-demo)
create_bucket(ObjectLockEnabled=True)                         -> 200 OK

# 2. Set a 30-day GOVERNANCE default retention (mc: mc retention set --default GOVERNANCE "30d" rustfs/preserve-demo)
put_object_lock_configuration(Mode=GOVERNANCE, Days=30)      -> 200 OK
get_object_lock_configuration()
  -> {'ObjectLockEnabled': 'Enabled',
      'Rule': {'DefaultRetention': {'Mode': 'GOVERNANCE', 'Days': 30}}}

# 3. Upload a file; it inherits the 30-day default
put_object(KEY)                                               -> VersionId v1
get_object_retention(KEY)
  -> {'Mode': 'GOVERNANCE', 'RetainUntilDate': '2026-11-18T04:00:50+00:00'}   # 30 days out

# 4. A plain delete adds a delete marker and SUCCEEDS (S3 semantics)
#    the locked version is untouched (mc: mc rm rustfs/preserve-demo/file)
delete_object(KEY)                                            -> 200 OK

# 5. Deleting the LOCKED VERSION is what WORM blocks (mc: mc rm --version-id v1 ...)
delete_object(KEY, VersionId=v1)
  -> AccessDenied: Object is under GOVERNANCE retention and cannot be deleted
     until 2026-11-18 04:00:50 +00:00

# 6. A legal hold (indefinite) hardens the same version
put_object_legal_hold(KEY, VersionId=v1, Status='ON')         -> 200 OK
delete_object(KEY, VersionId=v1)
  -> AccessDenied: Object has a legal hold and cannot be deleted. Remove the legal hold first.

The refusal in step 5 is the control working: the object version stays put until 2026-11-18. Step 4 is the subtlety worth internalizing. A delete without a version id only drops a delete marker on top of the history; the protected version survives, which is exactly why Object Lock requires versioning. Repeating step 5 against a COMPLIANCE default returns the same refusal, and a governance-bypass header does not lift it — the behavior compliance-mode users depend on.

Lifecycle management: where retention meets automation

Retention keeps objects; lifecycle (ILM) cleans them up once retention passes. On RustFS, Lifecycle Management is a ✅ Available feature, and ILM Tiering can move objects to a remote S3 target. On AWS, lifecycle rules transition objects between storage classes and expire them. On AWS, the two features are designed to stack rather than conflict: lifecycle configurations continue to function normally on protected objects, including placing delete markers — but a locked version of an object cannot be deleted by an S3 Lifecycle expiration policy, and Object Lock is maintained regardless of which storage class the object resides in and throughout S3 Lifecycle transitions between storage classes. In other words, the lock does not block transitions, and expiration cannot permanently delete the protected version. (AWS-side behavior, per the Object Lock considerations doc in Sources; not exercised in this article's hands-on run.) One gap to name plainly: RustFS does not ship AWS-style local cold tiers (Glacier Instant/Flexible/Deep Archive). Its ILM tiers to a remote S3 bucket you operate, so "cold storage" is another S3 endpoint, not a magic cheaper shelf inside the same cluster.

The self-hosting trade-off you can't skip

Running this yourself removes the per-gigabyte bill and the vendor lock-in, but it moves responsibility onto you. The lock metadata and the object catalog live inside RustFS, which means RustFS reliability is the catalog's reliability. There is no separate managed metadata service to fall back on, so backups, monitoring, and restore drills are your job, not a checkbox in a console. AWS can publish durability math; a self-hosted cluster ships no such promise, only the primitives to build one. The candid version: self-hosting trades a recurring bill and operational simplicity for control and cost, and it expects you to engineer durability instead of renting it. Own that before you migrate compliance data.

When should you self-host data preservation?

Pick self-hosted RustFS when you need WORM and retention but also need to keep data inside your perimeter, avoid per-GB egress, or escape a proprietary control plane, and you have the ops capacity to back up and monitor the cluster. Pick managed S3 when the compliance attestation, the managed durability, and zero operational load outweigh the bill, and you are comfortable with AWS's pricing model. Mixed setups are common: keep hot, frequently audited records on a self-hosted cluster and replicate to a managed target for offsite durability. The decision is less about features, which now overlap heavily, and more about who carries the durability burden and who controls the keys.

FAQ

What is the difference between Object Lock and bucket versioning?

Versioning keeps every revision of an object so overwrites and deletes are recoverable. Object Lock prevents overwrite and deletion of a specific version for a set time or until a legal hold is cleared. Object Lock requires versioning, but versioning alone does not stop a delete.

Can a compliance-mode lock be removed early?

No. AWS documents that in compliance mode no user, including the root user, can shorten the retention or delete the object until it expires; the only documented path is deleting the AWS account. On self-hosted S3-compatible systems the same strictness is the point of the mode.

Does a legal hold expire?

No. A legal hold has no associated time window. It protects an object version until an authorized user explicitly removes it, and it is independent of any retention period on the same object.

What is variable retention / an event hold?

A newer AWS retention shape where you defer the retain-until-date. You turn on an event hold with an event hold duration, and the object stays protected while the hold is on; when you release it, Amazon S3 fixes the retain-until-date at the release time plus the duration. It is meant for cases where the keep-time is unknown at write time, such as a configurable ransomware recovery window.

What happens if I try to expire a locked object with a lifecycle rule?

Lifecycle rules keep functioning normally on protected objects, including placing delete markers. What the lock prevents is permanent deletion: a locked version cannot be deleted by a lifecycle expiration policy, and the protection persists through transitions between storage classes. Once the retention period ends and any legal hold is removed, the locked version can be expired like any other.

Is RustFS Object Lock production-ready?

RustFS lists Object Lock (WORM), bucket versioning, and lifecycle management as ✅ Available in its Feature & Status table as of this writing. The hands-on run in this article exercised the lock, default retention, and delete-blocking behavior on RustFS 1.0.3. Validate against your own workload and retention requirements before relying on it for regulated records.

A note on how this was verified

Article type: this article contains hands-on verification (Object Lock and versioning exercised on the RustFS 1.0.3 native Windows binary with the AWS SDK, boto3) plus documentation interpretation of AWS S3 Object Lock. It is not a production-scale deployment report.

What was actually executed: starting RustFS 1.0.3 from the native Windows binary, creating buckets with Object Lock enabled, setting 30-day GOVERNANCE and COMPLIANCE default retention, reading the configuration back, uploading objects, and confirming the inherited retention (retain-until-date 30 days out). Confirmed that a plain delete adds a delete marker and succeeds (S3 semantics), while deleting the specific locked version is refused in both GOVERNANCE and COMPLIANCE modes with the exact retain-until-date, that a governance-bypass header does not lift a COMPLIANCE lock, and that a legal hold blocks deletion until removed.

What was not verified: no production-scale or multi-node deployment, no concurrency or throughput testing, no failure-injection or restore drill, and no compliance attestation was performed. Treat the hands-on as a functional smoke test, not a certification.

Verify against your own environment and keep a rollback plan before pointing real compliance data at any storage system, managed or self-hosted.

Sources

RustFS README, Feature & Status table (Object Lock WORM, Versioning, Lifecycle Management marked Available; S3 Tables and MinIO On-Disk Compatibility marked Preview) — https://github.com/rustfs/rustfs (checked 2026-10-19)
RustFS Quickstart (docker run, default credentials rustfsadmin/rustfsadmin, API port 9000 / console 9001) — https://github.com/rustfs/rustfs (checked 2026-10-19)
Amazon S3 Object Lock overview: governance vs compliance modes, legal hold, versioning requirement, Cohasset assessment for SEC 17a-4 / CFTC / FINRA — https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock.html (checked 2026-10-19)
Configuring S3 Object Lock with the AWS CLI: create-bucket --object-lock-enabled-for-bucket, put-object-lock-configuration with COMPLIANCE Days 50, put-object-legal-hold Status=ON; also "Enable Object Lock on an existing S3 bucket" (versioned buckets; cannot be disabled afterwards) and variable retention / event hold sections — https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock-configure.html (checked 2026-10-19; existing-bucket and variable-retention sections checked 2026-10-27)
Amazon S3 Object Lock considerations: "Object lifecycle management configurations continue to function normally on protected objects, including placing delete markers. However, a locked version of an object cannot be deleted by a S3 Lifecycle expiration policy. Object Lock is maintained regardless of which storage class the object resides in and throughout S3 Lifecycle transitions between storage classes." — https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock-managing.html (checked 2026-10-27)
MinIO Client reference, mc retention set --default MODE VALIDITY (e.g. mc retention set --default GOVERNANCE "30d" alias/bucket); the note "Beginning with RELEASE.2025-05-20T20-30-00Z, MinIO AIStor allows setting object locking and retention rules on any existing bucket" appears on both the mc retention set page and the mc mb page (--with-lock) — https://min.io/docs/minio/linux/reference/minio-mc/mc-retention-set.html and https://min.io/docs/minio/linux/reference/minio-mc/mc-mb.html (checked 2026-10-27)

Apache Iceberg & Lakehouse: RustFS S3 Tables in Practice

Ethan Carter — Mon, 19 Oct 2026 13:15:00 +0000

A lakehouse is not a product; it is a three-layer split — a table format (Apache Iceberg), a catalog that knows which tables exist, and object storage that holds the bytes. RustFS's S3 Tables feature contributes the middle layer directly: every bucket can expose a built-in Apache Iceberg REST catalog, on the same server that stores the data. This article walks the stack layer by layer, with results from a hands-on session on RustFS 1.0.1, including the part that failed, which is the part worth reading.

Fact	Value
Table format in RustFS S3 Tables	Apache Iceberg, `format-version: 2` (first-hand: create-table response)
Catalog	Built-in Iceberg REST catalog, enabled per bucket (`/iceberg/v1/<bucket>`)
Metadata and data location	Same bucket, reserved prefix `.rustfs-table/` (first-hand: `metadata-location`)
Minimum RustFS version	1.0.0 (GA 2026-09-16)
License	Apache 2.0
PyIceberg 0.12.0 against it (first-hand)	Read path (list/load) works; append commit rejected (missing `x-amz-content-sha256` header)

What a lakehouse actually is

A lakehouse is an architecture, and the fastest way to understand it is to name the three layers it separates. The table format — Apache Iceberg in this article — defines how a table's rows, schema, and snapshots are represented as ordinary files. The catalog is the entry point that answers one question reliably: where is the current metadata for table X. Object storage holds both the data files and the metadata files, durably and cheaply.

None of the three layers needs to know about the others' internals, and that is the point. A query engine speaks to the catalog, the catalog points at metadata, the metadata points at data files on object storage. Swap any one layer and the others keep working, which is exactly what has made this split the default design for analytics on top of object storage, and why an S3-compatible store with a built-in catalog changes the deployment math.

One implementation detail is worth knowing before the architecture diagram hardens in your head: in RustFS's S3 Tables the catalog has no separate database behind it. The default object catalog backing persists catalog state in the same object store, under the reserved .rustfs-table/ prefix. The layers are conceptually separate and physically colocated — lose the bucket and the catalog state goes with it, which moves backup planning from optional to load-bearing.

Why the table format became the battleground

The table format is the layer where lakehouse projects compete, and Apache Iceberg is an open table format hosted by the Apache Software Foundation. Its design goal is treating a directory of Parquet files as a full database table: schema evolution, hidden partitioning, time travel via snapshots, and atomic swaps of metadata pointers so readers never see half-written state. Version 2 of the spec added row-level deletes, which is what let merge-on-read engines adopt it for upsert workloads.

RustFS creates Iceberg tables with format-version: 2, confirmed in the create-table response during the hands-on run below. The version 3 specification exists in the Iceberg project, and RustFS's S3 Tables documentation states v3 is not yet supported. That is a plain fact of current scope, the kind worth knowing before you design a partition spec against features that are not there.

Where object storage fits in the stack

Object storage is the layer the lakehouse leans on hardest and trusts the least visibly. Every Iceberg table read touches the catalog, then one metadata file, then the manifest list, the manifest files it names, and finally the data files — a fan-out of small reads against the store. The Iceberg spec asks little of the file system: in-place writes, seekable reads, deletes. Consistency leans on the catalog's atomic metadata swap, not on the store. Engines nonetheless perform many small reads and listings per query, so storage that serves metadata-sized objects quickly is what keeps the architecture responsive.

The storage layer also owns the failure modes. A lakehouse on object storage is only as durable as the store underneath it, which is precisely why the choice of store is an infrastructure decision, not a checkbox. Running it on hardware you administer means the durability story is your replication and backup plan, and no table format changes that arithmetic.

What RustFS S3 Tables provides

RustFS ships an Iceberg REST catalog as a built-in feature, enabled per bucket, on an Apache 2.0 licensed server (requires RustFS 1.0.0+). In a hands-on session on 2026-10-11 against RustFS 1.0.1 (single node, Windows x86_64), the full control-plane chain ran clean: enabling the catalog on a bucket returned enabled: true with catalog-uri: /iceberg/v1/my-bucket; creating a namespace and then a table both returned 200; and the table's metadata-location pointed into the same bucket under the reserved prefix .rustfs-table/, meaning metadata and data live together, with per-table UUID prefixes for isolation.

Two facts from the same session matter for architecture planning. First, the table response reported rustfs.credential-vending: unsupported. Vending is disabled by default (the documentation describes an opt-in path negotiated via X-Iceberg-Access-Delegation), so this session's clients authenticated with long-lived keys and SigV4. The consequence is operational: every engine client holds a full set of S3 credentials, and rotation, scoping, and auditing of those keys are your key-management workload rather than something the catalog hands out per session. The documentation states the preconditions (a compatible client that negotiates the header, a caller permitted to request table credentials) and stops there; its own PyIceberg walkthrough still authenticates with explicitly configured credentials, and the 1.0.1 build behind the hands-on session reported the field as unsupported outright. If vended credentials are a deciding factor for your deployment, verify the full credential path on your build before committing to it. Second, namespaces are organizational, not security boundaries — a tenant cannot be fenced with a namespace alone. Isolation has to come from the storage layer: bucket policies, prefix-scoped IAM, and the table-scoped admin actions the documentation defines. Design multi-team deployments accordingly.

Which engines connect — and the honest status

The catalog speaks the Iceberg REST protocol, so any Iceberg client should be able to point at it. What was actually verified, on the record: PyIceberg 0.12.0 with a SigV4 adapter completed the read path (namespace listing and table load) against the RustFS catalog. The append path failed: the commit was rejected with a missing x-amz-content-sha256 header, because botocore's generic signer only sets that header for empty bodies. That failure is recorded here as-is; if you build on PyIceberg, pin your versions and run one append before you commit to the stack.

The project's own support matrix tracks engine status: PyIceberg and DuckDB marked automated, Spark and Trino covered by a live test harness. Third-party signal cuts both ways. An issue against the catalog documented Spark hitting a SigV4 double-encoding 403 (Iceberg's signer double-encodes the URI path); it was filed in September 2026 and closed on October 5. Deploying a Spark-Iceberg release that contains the fix matters — older releases still exhibit the 403, so pin the version deliberately and verify it, rather than assuming a recent install includes it. Engines connect; the edges are exactly where you would expect them, and testing your engine against your storage version is not optional.

One protocol-level note keeps expectations straight. The Iceberg REST catalog spec leaves authentication to implementation choice — the reference flow uses OAuth2, while this catalog authenticates requests with SigV4 signing, which is exactly why the PyIceberg session above needed a signing adapter. Authorization is not carried by the REST protocol here either: namespace and table operations map to RustFS admin actions enforced at the storage layer, and the documentation states plainly that full AWS S3 Tables control-plane compatibility is not claimed. Treat the REST interface as the common dialect between engines and catalogs, not a promise of identical semantics everywhere.

What this stack does not do

The honest list is short but load-bearing. RustFS S3 Tables includes no SQL engine. There is no embedded query processor; you bring DuckDB, Spark, or Trino, and they speak to the catalog. There are no multi-table transactions: each table commits independently, so cross-table invariants are your application's problem. Table creation is documented as direct create with v2 by default; staged table creation is listed as unsupported. Dropping a table removes the catalog entry but keeps the underlying objects in place, so cleanup of unreferenced files is a maintenance step you run separately.

None of these are hidden defects; they are the documented scope of the feature, and every lakehouse stack makes similar trade-offs somewhere. The difference between a stack you can defend and one that pages you at 2 a.m. is whether the limitations were priced in before the first table was created.

The ops reality of metadata-heavy tables

Iceberg tables accumulate metadata by design: every commit writes a new metadata file, manifests multiply with partition counts, and snapshots pile up until something expires them. The maintenance verbs are part of the format (expire_snapshots to bound history, remove_orphan_files to collect abandoned writes), and both carry footguns documented in the Iceberg project itself: expiring snapshots destroys the time-travel window you may be relying on for recovery, and orphan-file removal with an aggressive older_than window can race with slow in-flight commits and delete live data.

On self-hosted storage, add the capacity angle: metadata from many small commits is a real storage cost, and it lands on the disks you provision. Budget for the metadata prefix the way you budget for the data, and put snapshot expiry on a schedule from day one, not after the first "why is this bucket 3x my dataset" investigation.

The cost is not only capacity. Commit-heavy workloads produce torrents of small reads and writes against the storage layer's metadata path, and the single-node Windows session behind this article exercises that path gently — a clustered deployment with concurrent writers is a different load profile, which is one more reason the verification note below lists multi-node and concurrent-commit testing as not performed. Where the ceiling sits is a fair question with no published answer: no RustFS small-object benchmark at metadata scale exists to cite, and the honest range depends on drives, erasure-set width, network, and client mix. The practical move is to run a small-object mixed-workload benchmark against the exact cluster hardware before a lakehouse lands on it, and treat the measured number, not a datasheet, as the design input.

Backup deserves its own line in the runbook. Because catalog state and table metadata are ordinary objects under .rustfs-table/, a bucket-level backup covers the catalog automatically; a plan that copies only data files does not. It also pays to separate what the storage layer already gives you from what a backup adds. A distributed RustFS cluster stores every object as Reed-Solomon erasure-coded shards across an erasure set, with per-shard checksums and healing, which protects against disk, node, and bitrot loss inside the cluster. It does not protect against a DELETE that should not have happened, a pipeline writing garbage, or corruption applied consistently across shards; those are logical losses, and an independent copy is the only answer. Copy strategies differ in recovery point and blast radius: a scheduled snapshot copy bounds RPO at the schedule interval and restores by re-upload, while a continuously synced second bucket bounds RPO near zero at the cost of propagating mistakes just as fast, deletes included, unless the target keeps versions. The documentation reviewed for this article does not cover a cross-site replication feature, so treat that leg as something to verify on your build or to assemble with an S3-sync tool against the API. Two more facts from the documentation are worth pricing before you need them under pressure. Recovery has an escape hatch: Iceberg clients can register an existing metadata location as a table, so a lost catalog entry is recoverable when the metadata files survive. And there is an explicit upgrade red line: after the durable-strong snapshot format v2 is persisted, rolling back to a binary that only reads v1 fails closed — check the cutover runbook before upgrading any deployment that carries data you cannot lose.

The two catalog backings are a deliberate choice, not a default to forget. The default object backing keeps catalog state in the bucket, and the documentation describes its concurrency model in one line: a commit validates its base metadata and referenced objects before conditionally updating the metadata pointer, a conflicting writer must reload the table and resolve the conflict, and the transaction boundary is one table. The durable-strong mode is the alternative, and the documentation prices its adoption rather than advertising it — a cutover runbook with migration preflight, coordinated writer fencing, and the rollback red line above. No threshold for making the switch is published, so the honest sequence is to run the default backing, watch how conflict retries behave under your real write concurrency, and pay the cutover cost only when the reload-and-resolve path is measurably in the way.

Monitoring has a documentation anchor too. The counters that matter here are derivable from documented behavior: commit conflicts (the reload-and-resolve path), snapshot count per table, and object count plus byte size under the .rustfs-table/ prefix. One more fact raises the stakes: metadata deletion and background maintenance are disabled by default, and RustFS does not run a built-in periodic maintenance scheduler — expiry jobs you schedule yourself only happen if something is watching them happen.

When a lakehouse on self-hosted object storage makes sense

The stack earns its keep in three situations. First: your data governance or cost profile already points at self-hosted storage, and adding a table format plus catalog turns that store into an analytics substrate instead of a backup target. Second: you want Iceberg's engine portability without a hyperscaler's managed-table billing; the same table reads from DuckDB locally and Spark on a cluster. Third: your tables are already large enough that the metadata fan-out matters, which is where a storage layer you can size and monitor yourself pays off.

The built-in catalog is also not the only catalog. The documentation ships a second, explicit pattern: RustFS as the object storage under a separate Iceberg REST catalog service (the external-catalog integration, demonstrated with Spark and a standalone REST container). The built-in catalog is scoped per bucket — its warehouse is a single bucket name — so if one catalog needs to span several buckets or stores, the external pattern is the documented route. The trade is one more stateful service that you run, upgrade, and back up yourself; which pattern fits depends mainly on whether your catalog needs to outlive one bucket.

If none of those hold (small tables, one engine, no storage constraints), a plain bucket with files works, and the lakehouse layer is complexity without a customer. The format is not the product; the workload is.

Frequently Asked Questions

What is an Apache Iceberg lakehouse?

An analytics architecture that separates three layers: the Apache Iceberg table format (how tables are represented as files), a catalog (where table metadata lives), and object storage (where data and metadata are durably kept). The separation lets engines like DuckDB, Spark, and Trino read the same tables without owning the storage.

Do I need a separate catalog server for Iceberg on RustFS?

No. RustFS S3 Tables enables a built-in Iceberg REST catalog per bucket — the hands-on run enabled it on an existing bucket with a single request, and the catalog URI was served from the same server at /iceberg/v1/<bucket>. Metadata files land in the same bucket under the .rustfs-table/ prefix.

Can PyIceberg write to RustFS S3 Tables?

Reads were verified working in a first-hand session (PyIceberg 0.12.0, SigV4). Appends were rejected with a missing x-amz-content-sha256 header error, a signing quirk between botocore's generic signer and the catalog endpoint. Test one append with your pinned versions before building on it.

Does RustFS include a SQL query engine?

No. RustFS S3 Tables provides the table format layer and the catalog; query engines are external — DuckDB, Spark, Trino, or any Iceberg REST client. The project's support matrix tracks which engines are exercised against the catalog.

How is this different from AWS S3 Tables?

AWS S3 Tables is a managed AWS feature: Amazon describes it as fully managed, meaning AWS operates the catalog and its maintenance and you never manage metadata infrastructure yourself. RustFS's implementation is the self-hosted path — the same Iceberg REST interface, on an Apache 2.0 server you run, with the default object catalog backing persisting catalog state in your own bucket. That shifts the metadata failure domain and the backup plan onto your side of the table, which is the deployment trade being made, not a defect ranking.

A note on how this was verified

Article type: architecture analysis anchored on a first-hand verification session. The hands-on results quoted here (catalog enable, namespace and table creation, PyIceberg read success and append failure) were run on 2026-10-11 against RustFS 1.0.1, single node on Windows, with curl SigV4 and PyIceberg 0.12.0; the full method and raw outputs are documented in the companion article on the S3 Tables catalog. Not verified for this article: Spark or Trino against the live catalog, multi-node clusters, concurrent commits, snapshot expiry behavior, and production-scale metadata volume. Iceberg spec and maintenance behaviors are cited from the Apache Iceberg project documentation; check current versions before designing partition specs.

Sources

Apache Iceberg project (table format, spec, engines): https://iceberg.apache.org/
RustFS S3 Tables documentation: https://docs.rustfs.com/en/administration/data/s3-tables
RustFS S3 Tables engine support matrix: https://github.com/rustfs/rustfs/blob/main/docs/architecture/s3-tables-support-matrix.md
RustFS 1.0.0 GA announcement (2026-09-16): https://rustfs.com/blog/announcing-rustfs-1-0-0-ga/
Third-party issue: Spark SigV4 double-encoding 403 against the catalog (closed 2026-10-05): https://github.com/rustfs/rustfs/issues/8103
AWS S3 Tables (managed Iceberg tables, for the FAQ comparison): https://aws.amazon.com/s3/features/tables/
RustFS durable-strong catalog cutover runbook (upgrade red line): https://github.com/rustfs/rustfs/blob/main/docs/operations/s3-tables-cutover-runbook.md
RustFS PyIceberg walkthrough (SigV4 signing adapter, explicitly configured credentials): https://docs.rustfs.com/en/developer/integration/big-data/pyiceberg
RustFS external Iceberg catalog integration (Spark + standalone REST catalog): https://docs.rustfs.com/en/developer/integration/big-data/iceberg
RustFS erasure coding (Reed-Solomon shards, checksums, healing): https://docs.rustfs.com/concepts/principle/erasure-coding
Companion article — RustFS S3 Tables: A Built-In Iceberg REST Catalog (same author profile, full method and raw outputs)

Data Ownership and Privacy: Why Self-Hosted Object Storage Wins

Ethan Carter — Fri, 16 Oct 2026 13:48:00 +0000

Self-hosted object storage does not make your data legally immune, but it changes every channel through which other parties reach it: the physical disk is yours, the legal process has to come to you, and the exit door has no per-GB toll. This article puts numbers on those three claims — CLOUD Act reach, GDPR fine ceilings, and S3 egress pricing — and is honest about what self-hosting costs you in return.

Fact (sourced, checked 2026-10-12)	Value
GDPR administrative fine ceiling (Art. 83(5))	€20,000,000 or 4% of total worldwide annual turnover of the preceding financial year, whichever is higher
Largest single GDPR fine to date (Meta, May 2023, EU-US transfers)	€1.2 billion
CLOUD Act scope for US-based providers	Data in custody or control must be produced regardless of where it is stored
AWS S3 data transfer out to internet	$0.09/GB after the first 100 GB/month (us-east-1)
Moving 5 TB out of S3 in one month	(5,000 − 100) GB × $0.09 = $441
RustFS license and deployment surface	Apache 2.0; Linux, Docker, Podman, Kubernetes (Helm/Operator), macOS, Windows

What data ownership actually means

Data ownership comes down to three channels of control. The first is physical: whose disks hold the bytes, and in which building. The second is legal: whose jurisdiction the operator answers to, and what process another government must follow to reach the data. The third is economic: what it costs you to leave, which is set almost entirely by egress pricing.

When you upload an object to a managed cloud bucket, you keep the first channel only partially — the object is on hardware you will never touch, in a region you chose from a menu. You keep none of the second and third channels: the operator's legal exposure becomes your exposure, and leaving is metered. Self-hosting collapses all three channels onto infrastructure you administer. That is the entire argument, and every section below prices one piece of it. The legal examples here draw on EU and US frameworks because they are the pair most often cited in multi-jurisdiction deals; the analysis itself travels — a UN CTED report counts 128 states with some form of data protection or privacy legislation in force.

Who can legally read your cloud bucket?

A US-based provider can be compelled to produce data in its custody or control regardless of where the data physically sits. That is the Clarifying Lawful Overseas Use of Data (CLOUD) Act, enacted March 23, 2018 (Pub. L. 115-141, Division V), and it grew out of Microsoft Corp. v. United States — the 2013 case where the FBI sought email stored on Microsoft servers in Ireland. The Department of Justice states the principle directly: a company subject to US jurisdiction can be required to produce data it controls "regardless of where it chooses to store that data at any point in time."

For a US-headquartered cloud operator, region selection inside that operator's menu does not remove this channel. It changes which building the disk is in, not who can be served a warrant. This is a fact about the operator's legal position, not a claim that any particular request will target your data.

What does GDPR expect from your storage layer?

GDPR does not care where your object storage runs; it cares who is accountable. If you self-host, you are the controller in full — the physical security of the disks, the encryption at rest, and the breach notification clock are all yours. That is more responsibility, and the fines scale to match: Article 83(5) caps administrative fines at €20,000,000 or 4% of total worldwide annual turnover, whichever is higher. The ceiling is not theoretical. In May 2023, Meta was fined €1.2 billion over EU-to-US personal data transfers, the largest GDPR fine issued to date.

So self-hosting neither guarantees compliance nor removes the exposure. What it changes is legibility: when the data center is yours, data residency is a fact you can verify with a walkthrough of the rack, not a promise in a region table. For teams whose compliance posture requires proving where bytes physically live, that difference is the point.

Egress fees are the quiet lock-in

Lock-in is usually discussed as API compatibility, but the binding constraint is the exit invoice. AWS charges $0.09 per GB for data transfer out to the internet after the first 100 GB per month, aggregated across services. Moving a 5 TB dataset out of S3 Standard in one month costs (5,000 − 100) × $0.09 = $441 before you have paid any destination storage. A 50 TB archive costs roughly $4,490. The storage itself is $0.023 per GB-month for the first 50 TB; requests add $0.005 per 1,000 PUTs and $0.0004 per 1,000 GETs.

Self-hosted storage has no egress meter. If the cluster sits on your own network, pulling 50 TB back is a file copy. This is why exit-cost modeling belongs in the original architecture decision, not the migration postmortem: the per-GB toll is the mechanism that turns "we can always move later" into "we will re-evaluate next year."

What self-hosting gives you — and what it takes back

The honest list starts with the costs. Self-host an object store and you inherit every operational duty the cloud vendor used to absorb: hardware capital, spares, power and cooling, physical security, capacity planning, disk replacement, durability engineering, patching, backup verification, and the on-call rotation when replication falls behind. You also stay the controller for legal purposes — a subpoena served on your company reaches your servers just as it would have reached your provider's, and a seizure order can take the hardware itself. Ownership is not anonymity.

What you get is decision rights over all three channels. Physical: the disks are in rooms you control, and decommissioning means shredding media, not filing a support ticket. Legal: requests arrive addressed to you, where you can see them and respond (or litigate) under your own counsel. Economic: no egress meter, no per-request meter, and a license that does not expire — RustFS is Apache 2.0, so the storage layer itself costs nothing to keep running.

The grey zone: self-hosted software on rented hardware

Between "fully cloud" and "fully self-hosted" sits the deployment most teams actually build first: the storage software runs on a rented virtual machine. RustFS on an EC2 instance is the canonical example — you administer the server, hold the keys, and set the policies, but the physical disks belong to Amazon, and the CLOUD Act reaches the data exactly as it did before, through the provider. The egress meter comes back too, because the bytes still leave AWS over a billed pipe when you migrate. The three channels follow the hardware and the operator's jurisdiction, not the software license. Colocated machines you own, or racks in a facility you control, are what actually move the channels; a VM labeled "self-hosted" does not.

Where RustFS fits in a self-hosted stack

RustFS is a distributed object storage server written in Rust, S3-compatible, and licensed under Apache 2.0. The deployment surface in its documentation covers Linux (single-node or distributed), Docker, Podman, Kubernetes via Helm chart and Operator, macOS, and Windows. This means the "own hardware" in this article can be a rack of bare-metal servers, an on-premises Kubernetes cluster, or a single workstation. The feature table in its README (checked 2026-10-12) lists versioning, object lock, server-side encryption, and IAM/policies as available, with S3 Tables and MinIO on-disk compatibility marked as preview.

Two things it does not do, stated plainly: it has no FUSE/POSIX file-system mode, and it does not come with a compliance certification — running it inside a regulated environment is your work, not a checkbox on a vendor's SOC 2 page. Version 1.0.0 reached GA on 2026-09-16; current releases are in the 1.0.x line.

How to move without betting the archive

Teams rarely regret self-hosting because of the software; they regret it when the migration plan assumed a perfect weekend. The workable pattern is incremental, and S3 compatibility is what makes it possible: stand up the self-hosted cluster, keep it secondary, and dual-write new objects to both locations while the primary stays on the cloud. Backfill historical data at whatever rate your uplink allows: a 50 TB backfill over a 1 Gbps link takes about five days, and it costs $0 in egress if it runs from a self-hosted copy, or roughly $4,490 if it must be pulled out of AWS.

Verify by checksumming the way backup tools do, not by counting objects: object counts hide partial uploads. Cut over reads per bucket, per prefix, or per application — S3 client libraries make the endpoint a configuration value, which is the one genuinely portable part of the S3 ecosystem.

A decision checklist before you sign the next cloud renewal

Work through these five questions with numbers, not adjectives. First: what is your current monthly egress invoice, and what would one full migration cost at $0.09/GB? Second: which jurisdiction is your provider headquartered in, and does your compliance team accept that channel? Third: what is the durability story for self-hosting — N disks, N servers, N sites, and who replaces hardware at 2 a.m.? Fourth: can your team run versioning, retention, and encryption policies yourself, or were you relying on the vendor console? Fifth: what is the rehearsal plan — has anyone actually restored the archive end to end?

If the answers to three or more of those are "we have not checked," the cloud invoice is not the risk. The unstated assumptions are.

One cost the checklist must not skip is total cost of ownership. This article prices the exit channel — egress — because that number is published and uniform. Hardware capital, cross-site redundancy, power, spares, and staffing dominate the self-hosted side, vary by orders of magnitude between a homelab and a datacenter presence, and cannot be modeled generically without inventing figures. If the business case needs a TCO comparison, build it from your own hardware quotes and your team's actual payroll — not from any article, including this one.

Frequently Asked Questions

Is self-hosted object storage automatically GDPR compliant?

No. Self-hosting changes where the data sits and who operates it, but you remain the data controller with the full Article 32 security obligation and the Article 83(5) fine exposure — up to €20,000,000 or 4% of worldwide turnover of the preceding financial year. Compliance comes from your controls (encryption, access, retention, breach response), not from the deployment location.

Does the CLOUD Act apply to my own servers?

The CLOUD Act compels US-based providers, not disks. If you operate your own hardware outside a US provider's tenancy, there is no provider to serve; legal process has to be addressed to you, under the laws of your jurisdiction. That is a real difference in the legal path — not a guarantee against government access, which no storage architecture provides. If your "self-hosted" cluster runs on a rented cloud VM, the answer flips back: the provider is subject to the Act, and the disks are the provider's.

How much does it cost to migrate data out of S3?

Egress is $0.09/GB after the first 100 GB per month. A 5 TB exit costs (5,000 − 100) × $0.09 = $441; a 50 TB archive costs about $4,490, charged by AWS whether or not your destination is cheaper. Migration tooling and destination storage are separate line items.

What do I lose by self-hosting object storage?

Vendor-managed durability, capacity elasticity, and the operations staff implicit in them. You inherit hardware replacement, replication monitoring, patching, and backup verification. Teams without an on-call rotation or a verified restore drill should treat that as a staffing decision before it becomes a storage decision.

Is RustFS really free for commercial use?

Yes — Apache 2.0, with no license meter and no seat count. What is not free is the infrastructure it runs on and the operational labor it requires. The README feature table lists versioning, object lock, server-side encryption, and IAM as available; S3 Tables and MinIO on-disk compatibility are marked preview.

A note on how this was verified

Article type: this is a documented-facts analysis piece, not a production deployment report. Legal texts (GDPR Art. 83, CLOUD Act / Pub. L. 115-141) were checked against primary sources in October 2026; AWS prices are published us-east-1 list rates as of 2026-10-12 and the migration examples are arithmetic derived from those rates, not measurements. RustFS deployment targets were checked against docs.rustfs.com; feature-status claims come from the project README and have not been re-tested hands-on for this article. No benchmark, latency, or durability number in this article was measured by the author. Verify current prices and the feature table before acting on any figure here.

Sources

GDPR Article 83 — General conditions for imposing administrative fines: https://gdpr-info.eu/art-83-gdpr/
EDPB, "1.2 billion euro fine for Facebook as a result of EDPB binding decision" (22 May 2023): https://edpb.europa.eu/news/news/2023/12-billion-euro-fine-facebook-result-edpb-binding-decision_en
US Department of Justice, CLOUD Act resources and white paper announcement: https://www.justice.gov/opa/pr/justice-department-announces-publication-white-paper-cloud-act
Congressional Record, Sen. Hatch on the CLOUD Act (Feb 5, 2018): https://www.govinfo.gov/content/pkg/CREC-2018-02-05/html/CREC-2018-02-05-pt1-PgS595-7.htm
UN CTED report, "The State of International Cooperation for Lawful Access to Digital Evidence" (128-state count): https://www.un.org/securitycouncil/ctc/sites/www.un.org.securitycouncil.ctc/files/files/documents/2022/Jan/cted_trends_report_lawful_access_to_digital_data_.pdf
Amazon S3 pricing (us-east-1 list rates): https://aws.amazon.com/s3/pricing/
RustFS documentation — installation targets: https://docs.rustfs.com/en/
RustFS GitHub README (license and feature status): https://github.com/rustfs/rustfs

Benchmarking S3 Throughput: How to Measure Without Lying

Ethan Carter — Wed, 14 Oct 2026 13:58:00 +0000

An S3 throughput benchmark is only as honest as its setup. Most published numbers hide three variables that move results by 5x or more: where the client runs, how big the objects are, and which second of the run got reported. This article shows a workflow with warp that keeps those variables visible, on a real single-node server.

Metric (first-hand run)	Value
GET throughput, 1 MiB objects, 8 workers	663 MiB/s avg; 1s splits ranged 530 to 749 MiB/s
PUT throughput, same setup	102 MiB/s avg; 1s splits ranged 63 to 128 MiB/s
GET throughput, 10 KiB objects, 8 workers	136 MiB/s, 13,965 obj/s (27 s run); repeat run: 107 obj/s, 10,708 obj/s
PUT throughput, 10 KiB objects, 8 workers	3.5 MiB/s, 360 obj/s; median latency 21 ms
Mixed workload total (GET/STAT/PUT/DELETE)	266 MiB/s, 443 obj/s
GET TTFB median	10 ms (1 min run), 5 ms (concurrency 4)
Environment	RustFS 1.0.1, single node, laptop (Ryzen 7 5800H, 16 GB RAM, NVMe SSD), warp 1.3.1, loopback

Treat every number in that table as a relative reference point, not a capacity claim. The server ran on loopback, on a developer machine, with a client on the same host. That limitation is the first thing this article teaches you to disclose.

What a throughput number actually measures

A throughput number is the output of three components in series: the benchmark client, the network path, and the storage server. Whichever one is slowest sets the ceiling, and the benchmark report will not tell you which one it was. When I ran warp get against RustFS 1.0.1 on loopback, the client and the server shared one machine, so the network component was effectively zero and the disk did most of the talking.

That setup is fine for comparing versions of the same server. It is useless for predicting what a remote client will see. In my run, GET averaged 663 MiB/s with 8 concurrent workers on 1 MiB objects. A client on another host, behind one 1 GbE link, would cap near 125 MiB/s and blame the server for the difference. Name the path in every result you publish, or the number means nothing.

PUT is not GET

The fastest way to spot an untrustworthy storage benchmark is one that reports a single throughput number. On the same server, same run length, same concurrency, my PUT result was 101.6 MiB/s while GET was 663.3 MiB/s, a 6.5x gap on one node. The write path persists data before it acknowledges; the read path serves from page cache after the first fetch. No amount of tuning closes that gap entirely, and pretending it does not exist misleads every reader.

The mixed benchmark shows the interaction effect too. Under a combined GET/STAT/PUT/DELETE workload, GET dropped from 663 MiB/s standalone to 199.6 MiB/s, and PUT fell from 101.6 to 66.3 MiB/s. Writes steal IOPS and cache from reads. If your real workload writes while others read, the mixed number, not the standalone one, is your planning baseline. Publish both or publish neither.

How long should a benchmark run?

Long enough that warm-up stops mattering. The first seconds of a run are cold: empty caches, open connections, and the server allocating space. warp handles this explicitly. Per its documentation, the analysis window starts when every thread has completed one request and ends when the last thread starts its final one, which is why the analyzed time is slightly below the requested duration.

Run length also drives variance. My 1 minute GET run split into 57 one-second buckets: fastest 749 MiB/s, median 672.7, slowest 530.5. That is a 1.4x spread between best and worst second, on loopback, with nothing else running. A 10 second run would have waved that spread around as noise, and a 5 minute run narrows the confidence interval further. Short runs also disagree with each other: two independent 10 KiB GET runs, one 30 seconds and one 20 seconds, averaged 13,965 and 10,708 obj/s, a 23 percent gap between runs measured minutes apart on an idle machine. For anything you plan to publish or reuse in capacity planning, run at least a minute per operation type, run it more than once, and report the spread, not a single second.

Why object size changes the answer

Throughput in MiB/s and objects per second are different claims, and object size decides which one you can honestly make. At 1 MiB objects, my GET run delivered 663 obj/s. At 10 KiB objects on the same server, the same 8-worker setup delivered 13,965 obj/s but only 136 MiB/s: 21 times more objects, a fifth of the bandwidth. Per-request overhead dominates at small sizes, and which of the two numbers matters depends entirely on what your traffic looks like.

The write side told an even sharper story. PUT at 1 MiB managed 101.6 obj/s; PUT at 10 KiB managed 360 obj/s. Objects a hundred times smaller bought only 3.5 times more objects per second, and median latency sat at 21 ms per request. Whatever the write path is doing per request, it does not shrink in proportion to object size, so small-object write workloads hit that per-request floor long before they stress bandwidth. If your production traffic is millions of tiny uploads, benchmark at that size, not at 1 MiB, because the 1 MiB number will flatter the server.

warp defaults each benchmark type to its own object size and lets you override it with --obj.size. It also supports randomized sizes (--obj.randsize), which the documentation describes as log2-distributed: equal object counts per doubling of size, so the average lands well below the maximum you pass. Pick the size your production traffic actually has. If you do not know it yet, measure that first, because a benchmark with the wrong object size is an answer to a question nobody asked.

Report the distribution, not the best second

Every benchmark produces a distribution, and the dishonest version of a report quotes its maximum. warp prints three 1-second aggregates by default: fastest, 50% median, and slowest. My GET run: 749, 672.7, 530.5 MiB/s. My PUT run: 128.2, 101.7, 63.4 MiB/s. The honest summary line is the median with the range next to it, because your production p50 will look like the median, and your worst tenancy-neighbor day will look like the slowest second.

Latency deserves the same treatment. In the GET run, TTFB median was 10 ms, the 99th percentile 16 ms, the worst single request 36 ms. In the mixed run, STAT requests answered in a median 2.5 ms while DELETE sat at 39.1 ms median, because deletes wait on real I/O. Quote percentiles with the operation name attached. An average latency across mixed operations is a number that describes nothing.

Where should the benchmark client run?

As close to the server as honesty allows, which is rarely on the same box. Loopback removes the network, so it measures the server plus the client kernel, and results only transfer to deployments with a similar path. My numbers came from loopback on purpose: the goal was comparing server versions, not predicting a datacenter. When you need a network-realistic number, run the client on a separate machine on the same segment, and run it from several at once.

warp supports both modes. Multiple hosts can be passed comma-separated (--host=10.0.0.1:9000,10.0.0.2:9000), and for larger setups a warp server can drive remote warp client processes with --warp-client=client-{1...10}, each applying the same concurrency. The documentation's example of a distributed GET run uses 10 clients against 16 servers. Distributed runs also change interpretation: parameters apply per client, so total load multiplies. Write down the client topology next to the result.

Watch the server while the client runs

The warp report tells you what the client saw. It never tells you why. While a 20-second GET run against 10 KiB objects was in flight, I sampled the server machine every two to three seconds: CPU sat between 94 and 100 percent for the entire window, and physical disk reads stayed between 0 and 1.1 MB/s while warp reported about 105 MiB/s being served. Every one of those bytes came from the operating system's page cache, not from the disk. On a loopback setup the benchmark client shares those same saturated cores with the server, so the number measures the whole machine, not just storage.

Sampling during the run is what turns a suspicious number into an explained one. If CPU is pegged, you are client- or processor-bound; if the disk is pegged, the storage path is the ceiling; if neither is busy while throughput looks high, you are benchmarking the cache. Any process-level monitor works: CPU percentage, disk read/write bytes per second, free memory (watch it fall while the cache fills). One caveat from my own setup: my CIM-based sampler needed two to three seconds per sample, so treat those numbers as coarse, and put the sampler on the machine the server runs on.

Your GET number is probably a cache number

Cold reads are what your cache-miss traffic will actually see, and my runs did not measure them. The GET figures in this article are warm-cache numbers: the prefill wrote every object moments before the reads, and the page cache absorbed them. Getting closer to a cold number takes one of three things: a working set larger than physical memory so the cache cannot hold it, dropping the OS cache between runs (Linux has /proc/sys/vm/drop_caches; Windows exposes no supported equivalent, which is why my Windows-hosted numbers stay warm), or --obj.randsize to spread reads across a much wider object set. Say which one you did, because a GET figure quoted without its cache state is a number with an asterisk nobody wrote down.

Benchmarks need capacity too

This one cost me a failed run. The 1 minute PUT benchmark at 101.6 MiB/s wrote roughly 5.8 GB of test objects; the earlier GET run had already filled most of a bucket. When I raised the object size to 10 MiB, uploads failed with a generic InternalError from the server, and the benchmark aborted before it started. The server log showed the real cause: os error 112, storage full, on the data volume.

Two lessons worth the tuition. First, size your test volume before the run: benchmark data is real data, warp writes it at full speed, and a bucket can grow by gigabytes per minute. Second, a client-side error message is not a diagnosis. The S3 error XML said "please try again"; only the server log said "disk full". When a benchmark fails, read the server log before you conclude anything about the server.

A warp workflow you can copy

warp is MinIO's open-source S3 benchmark tool, and the workflow below uses only flags and syntax documented in its README. It creates and completely cleans a bucket called warp-benchmark-bucket before and after each run, so never point it at a bucket holding data. Three runs cover the planning basics:

warp get --host=127.0.0.1:9000 --access-key=... --secret-key=... --duration=1m --obj.size=1MiB --concurrent=8
warp put --host=127.0.0.1:9000 --access-key=... --secret-key=... --duration=1m --obj.size=1MiB --concurrent=8
warp mixed --host=127.0.0.1:9000 --access-key=... --secret-key=... --duration=1m --obj.size=1MiB --concurrent=8

Each run prints the per-operation report with fastest/median/slowest 1-second splits, then saves raw request data for later analysis with warp analyze --analyze.op=GET --analyze.v <file>.csv.zst, which reproduces segmented statistics per request-size band. One caveat from my run: warp 1.3.1 on Windows wrote the .json.zst run metadata but not the raw CSV, so treat the analyze step as version- and platform-dependent and capture the console report as your primary record.

Frequently Asked Questions

What is warp?

warp is an open-source S3 benchmarking tool from MinIO that measures GET, PUT, DELETE, STAT, and mixed workloads, saves raw request data, and reports throughput, request latency, and TTFB distributions with warm-up excluded from the analysis window.

Why is PUT so much slower than GET on the same server?

The write path persists objects before acknowledging while the read path is served after a single fetch, often from cache. In my single-node loopback run the gap was 6.5x (102 MiB/s PUT vs 663 MiB/s GET at 1 MiB objects); the exact ratio depends on hardware, but a large gap is normal, and a benchmark showing only GET throughput is incomplete.

What concurrency should I use for an S3 benchmark?

Start where your application actually operates, then sweep. warp's --concurrent flag sets workers per client; too few workers leave the server idle and too many measure your client's queuing instead of the server. My 8-worker loopback runs saturated a single node comfortably; the honest method is reporting throughput at several concurrency levels.

How do I benchmark more than one S3 server?

Pass hosts comma-separated (--host=10.0.0.1:9000,10.0.0.2:9000) so warp alternates between them, or use distributed mode with remote warp client processes driven by --warp-client=client-{1...10}. Note that flags apply to each client individually, so total load scales with client count.

Why are my GET numbers suspiciously high?

Check whether the operating system's page cache is serving the reads. In my runs, warp reported 105 MiB/s of GET traffic while physical disk reads sat near zero; the objects had been written moments earlier and never left memory. Options for a more realistic figure: a dataset larger than RAM, dropping the cache between runs on Linux, or --obj.randsize to spread reads across a wider object set.

What number should I publish from a benchmark?

The median with its range and the full environment: server version, node count, network path, object size, concurrency, and run duration. For example: 663 MiB/s median-bucket 673 MiB/s (530 to 749 range), RustFS 1.0.1 single node, loopback, 1 MiB objects, 8 workers, 1 minute. That sentence is falsifiable, which is exactly what makes it worth publishing.

A note on how this was verified

Article type: this is a hands-on benchmark methodology report based on runs I executed, plus documentation-sourced tool behavior.

The benchmark runs in this article were executed first-hand on 2026-10-12 against a single-node RustFS 1.0.1 (Windows x86_64 binary, GitHub release build dated 2026-10-03) on loopback, using warp 1.3.1 (143bf4f): GET 1 MiB / 8 workers / 1 min, PUT 1 MiB / 8 workers / 1 min, GET 1 MiB / 4 workers / 20 s, mixed 1 MiB / 8 workers / 40 s, GET 10 KiB / 8 workers / 30 s, PUT 10 KiB / 8 workers / 30 s, and a repeat GET 10 KiB / 20 s, all quoted with real console output. Server-side resource samples (CPU, free memory, disk read/write bytes per second) were collected on the same machine during the repeat GET run with a CIM-based sampler polling every two to three seconds; they are coarse and cover one machine. The tool-behavior claims (bucket cleanup, warm-up exclusion, randomized size distribution, multi-host and warp-client flags, the mixed-workload operation mix of GET 45% / STAT 30% / PUT 15% / DELETE 10%) come from the warp v1.3.1 README, not from my runs. Not exercised: any network path (all runs were loopback), distributed warp-client runs, cold-read or cache-bypassed measurements (every GET figure here is warm-cache; the cache section explains the options), object sizes above 1 MiB (the 10 MiB attempt aborted on a full test volume), runs longer than one minute, and the warp analyze step, which could not run because warp 1.3.1 did not persist the raw CSV on Windows in my environment; that limitation is quoted above rather than worked around. Nothing here predicts a production cluster. Re-run the same commands against your own hardware before you plan capacity on any of it.

Sources

warp v1.3.1 README (commands, defaults, warm-up behavior, mixed-workload mix): https://github.com/minio/warp/blob/v1.3.1/README.md
warp releases (v1.3.1 Windows binary used for the runs): https://github.com/minio/warp/releases/tag/v1.3.1
RustFS GitHub repository and releases (1.0.1 Windows binary used for the runs): https://github.com/rustfs/rustfs/releases/tag/1.0.1
RustFS documentation (deployment and compatibility): https://docs.rustfs.com

RustFS S3 Tables: A Built-In Iceberg REST Catalog in an Apache-2.0 Object Store

Ethan Carter — Mon, 12 Oct 2026 13:18:00 +0000

RustFS S3 Tables is an Apache Iceberg REST Catalog built directly into the RustFS object storage kernel, so one process on one port serves both standard S3 and the Iceberg REST Catalog protocol. That removes the separate Hive Metastore or Glue catalog a self-hosted lakehouse normally needs. It is Apache 2.0 with no separate commercial edition.

Item	Value
Feature	Iceberg REST Catalog embedded in the object-store kernel
License	Apache 2.0, no separate commercial edition
Ships in	RustFS 1.0.0
Endpoint	Same port as S3 (`9000`)
Status	Built into the object-store kernel; fully open source (Apache 2.0)
Iceberg format	v1 and v2 supported (v2 default); v3 unsupported

What is RustFS S3 Tables?

RustFS S3 Tables is a catalog service embedded in the storage server rather than deployed next to it. An Iceberg client uses the REST catalog to discover tables and commit metadata changes. It uses the S3 API to read and write the underlying Parquet files. RustFS serves both interfaces on the S3 API port.

That collapses one moving part: a typical self-hosted lakehouse pairs the object store with a catalog such as Hive Metastore, AWS Glue, Nessie, or Polaris, and RustFS folds the catalog into the store. Because the catalog speaks the open Iceberg REST protocol, engines with an Iceberg REST client need no vendor plugin — Apache Spark, PyIceberg, and DuckDB are documented as connecting without one, and enabling a table bucket does not require registering existing Parquet files first. One consequence worth stating plainly: the catalog's metadata lives inside RustFS itself (Iceberg metadata JSON and manifests written to a reserved prefix on the same bucket), so there is no separate metadata failure domain. Your RustFS durability and backup story is also your catalog backup story, and you cannot fall back to an independent metastore.

The control plane and data plane split

Writes separate into two paths. Table creation and snapshot commits travel through the control plane, which is the Iceberg REST Catalog. Parquet data files are written by the compute engine straight into the bucket over the S3 API, with no intermediate layer in the data path.

Two details keep that boundary safe. Metadata pointers live under a reserved internal prefix, so ordinary S3 requests cannot create, overwrite, or delete catalog metadata; a stray aws s3 cp will not corrupt table state. On commits, the catalog runs compare-and-swap version checks to reject stale concurrent writers and uses idempotency keys to absorb duplicate submissions. The transaction boundary is one table. RustFS does not claim multi-table atomic transactions, and the limit is concrete: you cannot commit a schema change to one table and a data load to a second table in a single atomic operation. Workloads that need cross-table consistency — updating a dimension table and a fact table together, for example — must coordinate that in application logic or accept eventual consistency.

From an empty bucket to a working table

The official walkthrough starts with a dedicated bucket, then flips it into a table bucket with a signed request that carries an empty body. Using my-bucket on localhost:9000:

export RUSTFS_ENDPOINT="http://localhost:9000"
export AWS_ACCESS_KEY_ID="<your-access-key>"
export AWS_SECRET_ACCESS_KEY="<your-secret-key>"
export AWS_DEFAULT_REGION="us-east-1"

aws --endpoint-url "$RUSTFS_ENDPOINT" s3api create-bucket --bucket my-bucket

curl --fail-with-body --silent --show-error \
  --aws-sigv4 "aws:amz:us-east-1:s3" \
  --user "$AWS_ACCESS_KEY_ID:$AWS_SECRET_ACCESS_KEY" \
  --header "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" \
  --request PUT "$RUSTFS_ENDPOINT/iceberg/v1/buckets/my-bucket"

Both the enable call and the GET status check return HTTP 200, with enabled: true and catalog-type: iceberg-rest in the response. I confirmed this end to end on a single-node RustFS 1.0.1 run; see the note at the end for exactly what was executed.

Namespaces and tables are plain REST calls under the warehouse prefix. Namespace creation with curl:

curl --fail-with-body --silent --show-error \
  --aws-sigv4 "aws:amz:us-east-1:s3" \
  --user "$AWS_ACCESS_KEY_ID:$AWS_SECRET_ACCESS_KEY" \
  --header "Content-Type: application/json" \
  --header "x-amz-content-sha256: UNSIGNED-PAYLOAD" \
  --request POST "$RUSTFS_ENDPOINT/iceberg/v1/my-bucket/namespaces" \
  --data '{"namespace":["my-namespace"]}'
# {"namespace":["my-namespace"],"properties":{}}

One signing detail cost me a retry: the enable call signs an empty body, so it carries the well-known empty-sha256 header, but reusing that header on the namespace POST (which carries a JSON body) is rejected with Sha256Mismatch (HTTP 400). Body-carrying requests need x-amz-content-sha256: UNSIGNED-PAYLOAD or the actual body hash instead.

Table creation follows the same pattern against /namespaces/my-namespace/tables and returns the full Iceberg metadata document. Three fields in that response are worth reading: metadata-location points at the metadata JSON under a reserved .rustfs-table/ prefix on the same bucket, which is visible confirmation that catalog state lives in ordinary S3 objects next to your data; metadata.format-version is 2, matching the documented v2 default; and config reports rustfs.credential-vending: unsupported, so the client must hold its own S3 credentials rather than receiving temporary ones from the catalog.

From there an Iceberg client needs four settings. The REST catalog URI is http://localhost:9000/iceberg, and the client appends /v1. The warehouse is the bucket name, not an S3 URI or an AWS ARN. REST auth is SigV4 with signing name s3 and region us-east-1. The S3 file endpoint is the same host, with path-style addressing.

The docs flag one trap that first-timers hit: REST request signing and S3 file access are configured separately, even for the same account. Configure only the S3 side and the client returns 403 until you add the catalog's SigV4 credentials.

Which engines work today?

RustFS maintains a support matrix with different confidence levels per client. Two are automated. PyIceberg covers namespace and table creation, append, reload, scan, and catalog operations. DuckDB Iceberg 1.5.5 covers single-table create, insert, update, delete, and merge, plus schema evolution, snapshots, and concurrent writes.

Apache Spark has an opt-in live harness, so validate your exact Spark and Iceberg versions. Trino has a manual read-only probe and write compatibility is not claimed. StarRocks is documented as an external-catalog read path only. Databend has a live harness limited to an S3-stage read probe of table data files, with no claimed Iceberg REST integration.

At the endpoint level, /iceberg/v1 is the canonical prefix and /_iceberg/v1 is a MinIO AIStor-style compatibility alias. RustFS supports both.

My own run puts one caveat on the matrix's confidence levels. With PyIceberg 0.12.0 against RustFS 1.0.1, the catalog read path works over SigV4: namespace listing and table load return clean responses. The append commit does not. PyIceberg signs catalog requests with botocore's generic SigV4 signer, which sets the S3-style x-amz-content-sha256 header on empty-body requests but not on body-carrying ones; RustFS requires that header on its catalog endpoints, so the commit is rejected with missing header: x-amz-content-sha256. The matrix lists PyIceberg as automated-tested, so treat this as a version-sensitive area rather than a permanent gap: pin your PyIceberg version and run one append before you build on it. Signing is where client compatibility actually breaks today. A separate, still-open issue (#8103, filed September 2026) reports Spark failing metadata-table lookups for a different signing reason: Iceberg's signer double-encodes a URI path that RustFS canonicalizes the S3 way, turning what should be a 404 into a 403.

What's in scope, and what isn't

Before you build on it, here is the scope RustFS documents. Iceberg format v1 and v2 are supported with v2 as the default. Staged table creation, purge-on-drop, and Iceberg v3 are not supported. There is no SQL execution engine, no multi-table atomic transactions, and no independent active-active writes across regions. RustFS implements the open Iceberg REST Catalog protocol and does not claim full AWS S3 Tables control-plane compatibility, so "open-source S3 Tables" means a free, open-protocol Iceberg catalog rather than a drop-in AWS S3 Tables clone. Metadata deletion and background maintenance are off by default, with no built-in periodic scheduler. Dropping a table removes the catalog entry but keeps the underlying objects, so recursive deletion of paths still referenced by snapshots would lose data.

Where the catalog state lives, and how it fails closed

The default catalog backing is object-backed: RustFS persists catalog state as ordinary S3 objects — Iceberg metadata JSON, manifests, and the current pointer file — under a reserved internal prefix on the same bucket, not in a separate embedded database. That is why the backup story for your data is also the backup story for your catalog. The backing is configurable with RUSTFS_TABLE_CATALOG_BACKING; a durable-strong mode exists for deployments that want stronger durability than the default object-backed state, at the cost of a more involved cutover.

Because the catalog is objects plus a CAS-driven pointer, it fails closed by design. The support matrix tracks concrete cases: a stale commit token or stale base metadata returns a conflict without advancing the table pointer; missing metadata, manifest, data, or delete objects fail closed before a commit or maintenance step can advance state; concurrent writers resolve to a single winning CAS with retryable conflicts for the losers; catalog and ordinary S3 permission denials both block a data-plane bypass; and a stale maintenance plan fails closed before any object deletion. Across those cases the direction is the same: the system refuses to advance rather than risk corruption, and each guarantee is tied to a specific RustFS build and client version, not something to assume without testing your exact stack.

Upgrade risk concentrates in the durability-strong cutover. Moving to durable-strong backing requires a preflight reporting SNAPSHOT_MATERIALIZED with no blockers, an exclusive migration fence that drains in-flight writes, and a restart before admitting writers. Snapshot format uses a two-gate switch (RUSTFS_TABLE_CATALOG_STRONG_SNAPSHOT_V2 plus a fleet-confirmed gate); once a version-2 snapshot is persisted, rolling catalog writers back to a binary that only reads version 1 fails closed rather than replacing in-process state. The rollback red line is explicit: after v2 is on disk, do not revert to a v1-only binary. Keep the object-backed catalog backup in your restore set until durable-strong has cleared your retention window, and rehearse the disaster-recovery path on an isolated bucket before it carries data you cannot lose.

RustFS S3 Tables next to the alternatives

MinIO AIStor Tables. MinIO offers a comparable embedded-catalog capability, but only inside AIStor, its commercially licensed product line. AIStor is a separate binary from the AGPL-3.0 MinIO Object Store, and MinIO prices AIStor by capacity at roughly $0.02 per GB per month. The MinIO Object Store entered maintenance mode in December 2025 and no longer takes new features, so the open-source line does not carry this capability. RustFS ships the same class of capability in its Apache-2.0 core, with no community-versus-enterprise split. Licensing and deployment model are where they differ, not feature scope: if you want to drop the separate catalog without taking on a commercial license, RustFS is the lower-friction option, while AIStor brings a more structured commercial-support and compliance story.

Standalone open-source catalogs (Apache Polaris, Project Nessie). Polaris and Nessie are independent, Apache-2.0 REST catalogs you deploy and operate yourself, and both sit in front of any S3-compatible store. They are the genuine alternative if you want catalog/object separation rather than embedding: one catalog can serve several storage backends, and you control its versioning and access policy directly. The cost is exactly the moving part RustFS removes — you run, secure, back up, and upgrade a second service, and you own the wire between catalog and storage. RustFS trades that separation for a single process with one upgrade surface and one backup domain.

On maturity the trade is real: Polaris and Nessie are established projects with broad production adoption and active communities, while RustFS S3 Tables shipped with the 1.0.0 GA and has a thinner third-party deployment track record so far. If an existing operational playbook and catalog maturity weigh more than avoiding a second service, the standalone catalogs are the lower-risk starting point today; RustFS is the bet on a smaller operational footprint that is still closing that gap.

How to choose. The line is "embed vs separate," not "RustFS vs everyone else." If your priority is not running an extra service, RustFS or AIStor fit. If you already standardize on Polaris or Nessie across several backends, RustFS can sit underneath as just the S3 layer. If you need a managed, vendor-run catalog, the comparison shifts to AWS Glue or a cloud warehouse's native catalog rather than self-hosted Polaris/Nessie or RustFS.

Before you take this to production

Pin the version. Use RustFS 1.0.0 or later with a fixed tag rather than latest, and test in an isolated bucket. Give the table bucket its own lifecycle thinking, because table buckets are excluded from ordinary lifecycle expiration and enabling the mode on an existing bucket changes how those rules apply.

Configure both credential paths. Use least-privilege IAM for the catalog actions (admin:SetTableBucket, admin:GetTableBucket, and the namespace and table actions) plus separate S3 permissions for table file reads and writes. Leave metadata deletion and background maintenance off until you have reviewed the retained references, and never recursively delete S3 paths that snapshots may still point to. Start with DuckDB or PyIceberg on a single node to prove create, write, read, and drop before you scale out.

Namespaces group tables, but most of the isolation that matters lands on S3 prefixes, not on the namespace itself. The bucket-level enable response reports credential-scope: warehouse-prefix, and each table's config narrows it to a per-table UUID prefix (s3://my-bucket/tables/<uuid> in my run), which is the scope the catalog expects client credentials to cover. Two consequences follow. You cannot pre-provision a data-access policy for a table's prefix before that table exists, so namespace-level separation gives you organization rather than enforcement. And the realistic granularity options are coarser than the object model suggests: one credential set per producing application, strict prefix conventions, or a bucket per tenant when boundaries must be hard. Keep the catalog admin actions on a separate identity from data-plane writers either way.

Plan metadata maintenance up front. Metadata deletion and background maintenance are off by default and there is no built-in periodic scheduler, so dropping a table removes only the catalog entry and leaves the underlying Parquet files in place — live snapshots keep referencing them and keep occupying storage. Before a table carries real volume, set up a periodic cleanup job (your engine's expire_snapshots / remove_orphan_files, or a scheduled sweep of unreferenced paths) and never recursively delete S3 paths that live snapshots still reference.

The cleanup jobs themselves have teeth. remove_orphan_files deletes any data file current metadata does not reference, so a writer that crashes or pauses mid-commit leaves uncommitted files behind, and a cleanup pass with an aggressive older_than window can delete files a writer is about to commit — the Iceberg docs default that window to days, not minutes, for exactly this reason. expire_snapshots shrinks history on purpose, which is precisely what breaks point-in-time recovery: expire snapshots before your backup has copied them, and a restore lands on a newer state than the one you meant to keep. And every commit writes a fresh metadata JSON plus manifest list, so a table fed by many small commits accumulates metadata objects even while its data barely grows; until snapshots expire, all of it stays in the prefix your catalog reads on every table load.

Treat the catalog's reserved prefix as part of your backup set, since there is no independent metadata store to restore from.

FAQ

Is RustFS S3 Tables free and open source?

Yes. RustFS is Apache 2.0, and S3 Tables ships as a built-in feature of the storage server with no separate commercial edition.

Do I still need Hive Metastore or AWS Glue?

No. The Iceberg REST Catalog runs inside RustFS, so you do not deploy or operate a separate catalog service.

Which query engines work with RustFS S3 Tables?

Any client that speaks the Iceberg REST Catalog protocol. PyIceberg and DuckDB are automated in the RustFS test matrix; Spark and Trino have live harnesses; StarRocks and Databend are narrower.

Is RustFS S3 Tables production-ready?

It is a built-in feature of the Apache-2.0 storage server with no separate commercial edition. As with any feature, validate it against your own workload and keep a rollback path before it carries critical data.

How does RustFS S3 Tables differ from AWS S3 Tables?

AWS S3 Tables is a managed commercial service. RustFS implements the open Iceberg REST Catalog protocol on a self-hosted, Apache-2.0 stack and does not claim full AWS S3 Tables control-plane compatibility.

A note on how this was verified

Article type: hands-on control-plane walkthrough plus a systematic read of the official RustFS documentation, GA announcement, and published support matrix.

The walkthrough above was executed first-hand on a single-node RustFS 1.0.1 (Windows x86_64 binary, release build dated 2026-10-03) on 2026-10-11: bucket creation, table-bucket enablement, status check, namespace creation, table creation, and table load. A PyIceberg 0.12.0 client was also connected over SigV4: catalog reads (namespace listing, table load) succeeded, and the first append commit was rejected with the missing header: x-amz-content-sha256 error quoted in this article. The responses quoted here are taken from those runs, including the Sha256Mismatch rejection. Not exercised: successful engine-side data writes or scans (the commit path blocked them), DuckDB, concurrency benchmarking of metadata commits, large-volume snapshot or metadata-growth stress testing, failure-injection recovery drills, and the durable-strong cutover with its v2 snapshot gate. The operational cautions reflect documented behavior and known limits, not first-hand war stories. Validate against your own workload and keep a rollback path before it carries data you cannot lose.

Sources

RustFS GA announcement, "RustFS 1.0.0 GA" (2026-09-16): https://rustfs.com/blog/announcing-rustfs-1-0-0-ga/
RustFS S3 Tables documentation (EN): https://docs.rustfs.com/en/administration/data/s3-tables
RustFS S3 Tables support matrix (main): https://github.com/rustfs/rustfs/blob/main/docs/architecture/s3-tables-support-matrix.md
RustFS blog, "RustFS Launches S3 Tables" (2026-09-10): https://rustfs.com/blog/rustfs-s3-tables-iceberg-rest-catalog-quickstart
MinIO AIStor feature set and pricing: https://www.min.io/blog/aistor-overview
RustFS release 1.0.1 binaries (Windows x86_64, used for the hands-on run): https://github.com/rustfs/rustfs/releases/tag/1.0.1
rustfs/rustfs issue #8103, Spark SigV4 metadata-table failure (open as of 2026-10-11): https://github.com/rustfs/rustfs/issues/8103
Apache Iceberg maintenance procedures (remove_orphan_files / expire_snapshots semantics and defaults): https://iceberg.apache.org/docs/latest/spark-procedures/

S3 vs NFS for Kubernetes: Where Each One Actually Wins

Ethan Carter — Fri, 02 Oct 2026 13:15:00 +0000

S3 and NFS solve different Kubernetes storage problems. Use NFS or a CSI file driver when pods need shared, POSIX-style read-write-many file access. Use S3-compatible object storage when apps call the S3 API for immutable blobs, backups, and data lakes. RustFS gives you that S3 layer self-hosted under Apache 2.0.

Key Stats

Dimension	NFS (file)	S3 object storage (RustFS)
Access protocol	NFS v3 / v4 over TCP	HTTP REST (S3 API)
Concurrent pods	ReadWriteMany, native	One writer per object; concurrency via versioning / ETag
Kubernetes mount	csi-driver-nfs (`nfs.csi.k8s.io`, GA, K8s 1.21+)	No native volume type; needs s3fs-fuse or the app SDK
POSIX semantics	Yes (close-to-open consistency)	No — object store; a FUSE layer is only partial
RustFS status	—	Apache 2.0, S3-compatible, single binary; S3 Tables (Iceberg) & MinIO on-disk compat are Preview

What is NFS storage in Kubernetes?

NFS is a network filesystem that lets multiple pods read and write the same files across the network. In Kubernetes you usually consume it through the CSI NFS driver, whose plugin name is nfs.csi.k8s.io and which sig-storage ships as GA for Kubernetes 1.21 and later. The driver needs an existing NFSv3 or NFSv4 server and provisions Persistent Volumes by creating one subdirectory per PVC. NFS is the canonical ReadWriteMany filesystem: many pods on many nodes open the same file at once, which a block volume such as EBS or a local PV cannot do. We reach for it with shared model checkpoints, Jupyter home directories, and CMS uploads. The catch is that throughput tracks your filer, so a single NetApp or TrueNAS appliance becomes both the ceiling and the single point of failure. NFS itself has no built-in replication or tiering, so that lives on the backing storage.

What is S3-compatible object storage in Kubernetes?

S3-compatible object storage presents data as immutable blobs addressed by a key over an HTTP API, not as a POSIX filesystem. In Kubernetes there is no native "s3" volume type, so pods reach it through the application's AWS SDK or an S3 client such as aws s3 or mc, with credentials normally injected from a Secret. RustFS is an Apache 2.0, S3-compatible server built in Rust; its Feature & Status table lists S3 Core, Versioning, Object Lock (WORM), Lifecycle, and Distributed Mode all as Available. The mental model matters more than any benchmark: objects are addressed by key, versioned, and immutable until overwritten. There is no file handle, no in-place append, and no directory tree the kernel understands. For backups, artifacts, data lakes, and any app that already speaks the S3 API, this is the natural fit. We treat NFS and S3 as two separate tiers, not competitors.

The POSIX gap: why S3 doesn't mount like NFS

Object storage is not a filesystem, and that gap is the whole story. If a pod needs a path such as /data/shared/models/ckpt.bin with real POSIX semantics, NFS hands it over natively. S3 gives you a bucket and a key; to make it look like a directory tree you mount it through a FUSE layer such as s3fs-fuse. We have run s3fs-fuse in production, and its own README is blunt about the limits: random writes or appends rewrite the entire object, directory listings suffer network latency, there are no hard links and no atomic renames, and multiple clients mounting the same bucket get no coordination. s3fs-fuse preserves only a "large subset of POSIX" — mode, uid/gid, symlinks — not the full contract. So if your app does in-place edits, file locking, or expects rename() to be instant, a FUSE mount will bite you. Own it up front: S3 wins as an API, loses as a filesystem.

ReadWriteMany: where NFS still wins

ReadWriteMany, meaning many pods writing the same files at once, is NFS territory. Kubernetes access modes are ReadWriteOnce, ReadOnlyMany, and ReadWriteMany; a block volume or local PV is ReadWriteOnce by default, so a second node cannot attach it. NFS was built for concurrent multi-client access over the network, so the csi-driver-nfs driver happily serves 50 replicas of a web frontend from one share. We use NFS for Jupyter home directories and shared training caches where every pod must see the same file tree. The operational cost is real: you run and monitor an NFS server, and a file lock held by a crashed pod can stall others until the lock expires. If your requirement is "shared mutable files across nodes," deploy csi-driver-nfs and stop debating. S3 cannot do this without a FUSE shim, and that shim gives you no POSIX locking.

Immutable blobs, backups, and data lakes: where S3 wins

S3 pulls ahead the moment your data is immutable, append-only, or key-addressed. Backups are the obvious case: our nightly PostgreSQL dumps and CI artifacts land in RustFS as versioned objects, so a bad overwrite is one versionId away from recovery, not a restore-from-tape drama. Object storage also scales horizontally without a single filer ceiling; RustFS Distributed Mode is marked Available, so you grow capacity by adding nodes rather than buying a bigger appliance. Data lakes and Iceberg tables are S3-native: you write Parquet through the S3 API and let the catalog track partitions. RustFS lists S3 Tables (Iceberg REST) as Preview, so the Iceberg control plane is still maturing — we flag that honestly rather than pretend it is GA. For ML datasets, log archives, and any pipeline that streams objects, S3 is the lingua franca, and NFS would only add latency and inode headaches.

How do you choose between them on a real cluster?

Start from the access pattern, not the technology. Ask four questions. First, do pods need POSIX file APIs and in-place edits? If yes, NFS. Second, do many pods write the same files at once? If yes, NFS or a CSI file driver. Third, is the data immutable blobs, backups, or a data lake? If yes, S3. Fourth, does the app already call the S3 API? If yes, S3, and skip the filesystem debate. We draw the line at mutability: mutable-shared-files maps to NFS, immutable-keyed-objects maps to S3. RustFS slots into the S3 side as a self-hosted, Apache 2.0 drop-in you control, with no egress fees and no vendor lock. The anti-pattern is mounting object storage as a filesystem just because the URL is shorter; you pay the FUSE tax for zero benefit. Pick the primitive that matches the workload and stop there.

Running RustFS as your S3 tier in Kubernetes

You can stand up the S3 tier in minutes. RustFS ships Kubernetes Helm charts, marked Available in its Feature & Status, or you can run the container directly. The documented start command, verbatim from the project README, is:

docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest

Default credentials are rustfsadmin / rustfsadmin — rotate these before production, since they ship as defaults. In Kubernetes we expose the S3 endpoint (port 9000) and the Web Console (9001) through a Service, inject the access and secret key from a Secret, and point pods at the endpoint with the AWS SDK or mc. RustFS is Apache 2.0 and built in Rust; a 3-node Distributed Mode deployment fits commodity hardware. Because it is S3-compatible, existing tooling — aws s3, rclone, the MinIO client — works unchanged. NFS stays a separate concern; RustFS complements it rather than replacing it.

FAQ

Can I mount S3 as a filesystem in Kubernetes?

Yes, via a FUSE layer such as s3fs-fuse (GPLv2), which mounts a bucket as a local path. But s3fs-fuse's own README warns of real limits: random writes rewrite the whole object, there are no hard links, no atomic renames, and no coordination between multiple clients mounting the same bucket. Treat it as a read-mostly or write-once mount, not a POSIX replacement.

Is the in-tree NFS volume deprecated in Kubernetes?

The nfs volume type still appears in the Kubernetes docs, but the maintained, recommended path is the CSI NFS driver (nfs.csi.k8s.io), which sig-storage ships as GA for Kubernetes 1.21 and later. It requires an existing NFSv3 or NFSv4 server and provisions PVs by creating a subdirectory per PVC. For new clusters, install csi-driver-nfs with Helm.

Which should I use for AI/ML training data?

Immutable, large datasets (Parquet, image shards, model checkpoints stored as objects) belong in S3-compatible object storage; RustFS serves them over the S3 API at scale. Shared, frequently-mutated small files or Jupyter home directories belong on NFS with ReadWriteMany. Mix both: object store for the corpus, NFS for scratch.

Can RustFS replace NFS?

No. RustFS is an S3-compatible object store and has no built-in POSIX or FUSE filesystem. Its Feature & Status table lists S3 Tables (Iceberg) and MinIO on-disk compatibility as Preview and does not list FUSE or POSIX. Use RustFS for the object tier and keep a CSI file driver such as csi-driver-nfs for shared files.

How do pods access RustFS?

Through the S3 API: inject the access and secret key from a Kubernetes Secret and point the AWS SDK or mc at the RustFS Service endpoint (port 9000). There is no native Kubernetes volume type, so apps must speak S3. If you must have a filesystem view, run s3fs-fuse as a sidecar or init container that mounts the bucket.

S3 Storage Tiering: Pushing Cold Data Down the Ladder

Ethan Carter — Thu, 01 Oct 2026 13:14:00 +0000

S3 storage tiering is the practice of moving objects from hot, expensive storage classes to colder, cheaper ones as access frequency drops, using lifecycle rules. It cuts your recurring storage bill without deleting a single byte. Standard sits at the top of the ladder, Glacier Deep Archive at the bottom.

Class	Min storage duration	Retrieval charge	First-byte latency	Notes
S3 Standard	None	None	Milliseconds	General-purpose hot data
S3 Intelligent-Tiering	None	None (no retrieval fee)	Milliseconds	Auto-tiers; saves up to 40% / 68% / 95%
S3 Standard-IA	30 days	Per GB	Milliseconds	128 KB minimum object size
S3 Glacier Instant Retrieval	90 days	Per GB	Milliseconds	Archive with instant access
S3 Glacier Flexible Retrieval	90 days	Per GB (restore)	Minutes to hours	Formerly S3 Glacier
S3 Glacier Deep Archive	180 days	Per GB (restore)	Hours	Cheapest class on the ladder

What is S3 storage tiering?

S3 storage tiering is the practice of placing each object in the cheapest storage class that still meets its access-latency needs, then moving it downward as it cools. AWS S3 exposes six classes on a single ladder: Standard, Intelligent-Tiering, Standard-IA, Glacier Instant Retrieval, Glacier Flexible Retrieval, and Glacier Deep Archive. You do not pick the class by hand for every object. Instead, you set lifecycle rules that transition objects after a given number of days of age or inactivity. The win is recurring: Standard carries the highest per-GB rate on the ladder, while Glacier Deep Archive is the lowest. Tiering captures that spread automatically. The catch is that colder classes add a per-GB retrieval charge and a minimum storage duration (30, 90, or 180 days), so the move only pays off for data you rarely read. Done right, it is one of the most effective recurring cost controls on an S3 bill. It changes nothing about your application, because the object key and API stay the same.

Why does cold data quietly drain your budget?

Most S3 bills are dominated by storage, not requests. A dataset that lands hot on launch day is often untouched three months later, yet it keeps sitting in Standard, accruing the highest per-GB rate on the ladder every single day. Teams rarely delete it (it might be needed) and rarely move it (the move is not automatic). That gap is avoidable waste. The structural fix is a lifecycle rule, not a cleanup sprint. Consider a 1 TB log archive read once a quarter: in Standard it costs the full hot rate 90 days out of 90; in a Glacier class the per-GB storage is substantially lower and you pay a per-GB retrieval only on the one day you actually read it. S3 Intelligent-Tiering quantifies the upside publicly: its Infrequent Access tier saves up to 40% on storage versus Standard, the Archive Instant Access tier up to 68%, and the opt-in Deep Archive Access tier up to 95%. Those are AWS-published figures, not marketing ranges.

How do lifecycle rules push objects down the ladder?

A lifecycle rule is just XML attached to a bucket. The Transition action names a storage class and a day count; S3 moves objects automatically once they cross that age. Here is AWS's own example that tiers the logs/ prefix over a lifetime (verbatim from the S3 User Guide):

<LifecycleConfiguration>
  <Rule>
    <ID>example-id</ID>
    <Filter><Prefix>logs/</Prefix></Filter>
    <Status>Enabled</Status>
    <Transition><Days>30</Days><StorageClass>STANDARD_IA</StorageClass></Transition>
    <Transition><Days>90</Days><StorageClass>GLACIER</StorageClass></Transition>
    <Expiration><Days>365</Days></Expiration>
  </Rule>
</LifecycleConfiguration>

Apply it with the AWS CLI (verbatim command form from the AWS CLI reference):

aws s3api put-bucket-lifecycle-configuration --bucket <bucket> --lifecycle-configuration file://lifecycle.json

Because RustFS is S3-compatible and lists Lifecycle Management (ILM) as Generally Available, the same S3 API call works against a RustFS endpoint. One rule, three actions: tier to IA at day 30, to Glacier at day 90, expire at day 365.

S3 Standard vs Standard-IA: when does the first step pay off?

Standard-IA trades a lower per-GB storage rate for two costs: a per-GB retrieval fee on every GET, and a 30-day minimum storage duration. The break-even is purely about read frequency. If you read an object more than about once a month, the retrieval fees erase the storage savings; keep it in Standard. If you read it quarterly or less, IA wins. AWS enforces a 128 KB minimum billable object size for IA, so transitioning a bucket of tiny files can cost more in overhead than it saves. Filter those out with <ObjectSizeGreaterThan>. The transition itself is billed as a request, so tiering millions of tiny objects has its own price. Practical rule: IA is the right first rung for objects larger than 128 KB that you access roughly once a quarter or less. Everything hotter stays in Standard; everything colder keeps falling.

Glacier tiers: at what point do milliseconds stop mattering?

Glacier is three rungs, and the only difference is retrieval time and minimum duration. Glacier Instant Retrieval gives you millisecond access at a per-GB retrieval fee and a 90-day minimum. Use it for archives you might need right now. Glacier Flexible Retrieval (the former S3 Glacier) takes minutes to hours to restore and also carries a 90-day minimum; it is the cheapest of the three for true cold storage you read a few times a year. Glacier Deep Archive is the bottom: hours-long retrieval and a 180-day minimum, the lowest per-GB rate on the entire ladder.

Tier	Minimum storage duration	Retrieval time
Glacier Instant Retrieval	90 days	Milliseconds
Glacier Flexible Retrieval	90 days	Minutes to hours
Glacier Deep Archive	180 days	Hours

The trap is the minimums. Delete or re-tier an object before its window and AWS charges you for the remaining days as a prorated fee. So Glacier only pays off for data with a real retention horizon: backups and compliance archives.

Intelligent-Tiering: automatic savings without the ops burden

S3 Intelligent-Tiering is the only class that moves data for you. You opt in, pay a small monthly monitoring and automation charge per 1,000 objects, and AWS watches access patterns. Objects not touched for 30 days drop to the Infrequent Access tier (up to 40% storage savings), and after 90 days of silence to Archive Instant Access (up to 68%). There are no retrieval charges and no minimum storage duration, which removes the biggest risk of manual tiering: getting the break-even wrong and paying penalties. Objects smaller than 128 KB are not auto-tiered; they stay at the Frequent Access rate and incur no monitoring fee. The trade-off is the monitoring charge itself. At very high object counts it can exceed what you would save, so it suits data lakes and user-generated content where access is genuinely unpredictable. For known-access patterns, explicit lifecycle rules are cheaper.

The traps that turn tiering into a loss

Tiering is not free, and four costs bite. Minimum storage durations apply: Standard-IA holds objects for 30 days, Glacier Instant and Flexible for 90, Deep Archive for 180. Delete or transition out early and AWS bills the remaining days as a prorated charge. The 128 KB minimum billable size on IA and Glacier classes means tiny objects cost more to tier than to leave alone, so filter them with <ObjectSizeGreaterThan>. Every transition is billed as a request, so moving millions of objects spikes a one-time request cost. And lifecycle only goes down the ladder: to bring an object back to Standard you must copy it explicitly with aws s3 cp or aws s3 sync, never a transition. Miss any of these and the savings go negative.

Self-hosting tiering with RustFS

If you run storage yourself, the same playbook applies. RustFS is Apache-2.0, S3-compatible object storage and lists Lifecycle Management (ILM) plus ILM Tiering (Remote S3) as Generally Available in its current Feature & Status table. A self-hosted RustFS cluster can apply S3 lifecycle rules that push cold objects to a remote S3 tier (another RustFS pool or an external S3 endpoint) once they cool, instead of holding everything on expensive local disk. Be clear on the model: RustFS does not replicate AWS's six in-place local storage classes. Its tiering is ILM-driven movement to a remote tier, not a drop-in Glacier class switch. Spin up a node to try it:

docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest

Default credentials are rustfsadmin/rustfsadmin; set real keys before production.

Can I move objects back to S3 Standard with a lifecycle rule?

No. Lifecycle transitions only go down the ladder. To bring an object back to Standard you must copy it explicitly with aws s3 cp or aws s3 sync, never a transition action.

What is the cheapest S3 storage class?

S3 Glacier Deep Archive: the lowest per-GB storage rate on the ladder, with a 180-day minimum storage duration and hours-long retrieval. AWS publishes the exact per-GB rates on aws.amazon.com/s3/pricing.

Does S3 Intelligent-Tiering have retrieval fees?

No retrieval charges and no minimum storage duration. You pay a small monthly monitoring and automation fee per 1,000 objects, plus standard storage. Objects under 128 KB are not auto-tiered and incur no monitoring fee.

What happens if I delete an object before the minimum storage duration?

You are charged a prorated amount equal to the storage cost for the remaining days. Standard-IA enforces 30 days, Glacier Instant and Flexible enforce 90, and Deep Archive enforces 180. Match your rules to real retention horizons.

Can RustFS do S3 storage tiering?

Yes. RustFS lists Lifecycle Management (ILM) and ILM Tiering (Remote S3) as Generally Available. It moves cold objects to a remote S3 tier through S3-compatible lifecycle rules. It does not implement AWS's local Glacier class ladder in place; tiering is movement to a remote tier.

Sources

AWS S3 storage classes: https://aws.amazon.com/s3/storage-classes/ (checked 2026-09-28)
AWS S3 Intelligent-Tiering: https://aws.amazon.com/s3/storage-classes/intelligent-tiering/ (checked 2026-09-28)
AWS S3 lifecycle configuration examples: https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecycle-configuration-examples.html (checked 2026-09-28)
AWS S3 pricing (per-GB list rates): https://aws.amazon.com/s3/pricing/ (checked 2026-09-28; rendered dynamically, verify current rates there)
AWS CLI put-bucket-lifecycle-configuration: https://docs.aws.amazon.com/cli/latest/reference/s3api/put-bucket-lifecycle-configuration.html (checked 2026-09-28)
RustFS README Feature & Status: https://github.com/rustfs/rustfs (checked 2026-09-28)

RustFS is Apache-2.0, S3-compatible object storage. Run it yourself: github.com/rustfs/rustfs

A 5-Minute RustFS Stack with Docker Compose

Ethan Carter — Sun, 27 Sep 2026 13:52:00 +0000

A 5-Minute RustFS Stack with Docker Compose

RustFS is an Apache 2.0, S3-compatible object store written in Rust that you can run as a single Docker Compose stack and reach in about five minutes. This guide walks through the official docker-compose-simple.yml. One command brings up a RustFS node on ports 9000 (S3 API) and 9001 (web console) with default credentials rustfsadmin / rustfsadmin. You get a MinIO-compatible S3 endpoint on your own hardware, with versioning, Object Lock, IAM, and server-side encryption included, and no cloud account required. One caveat before we start: the simple file is a single node writing across four local volumes, and S3 Tables (Iceberg) is still in preview.

Key stats

	RustFS (docker-compose-simple)
License	Apache 2.0
Image	`rustfs/rustfs:latest`
S3 API / console ports	9000 / 9001
Default credentials	`rustfsadmin` / `rustfsadmin` (change before exposure)
Storage layout	4 local volumes: `/data/rustfs{0...3}`
Preview features	S3 Tables (Iceberg REST), MinIO On-Disk Compatibility

All rows are from the RustFS README and the docker-compose-simple.yml in the repo root (links in Sources). No benchmarks invented.

What is a RustFS Docker Compose stack?

A RustFS Docker Compose stack is a single declarative file that launches the RustFS server as a container plus a tiny helper that fixes volume ownership, so you get a working S3-compatible endpoint without hand-writing docker run flags. It is the easiest path to self-hosting. Instead of remembering -p 9000:9000 -p 9001:9001 -v ..., you describe the desired state once and let Compose reconcile it. The official repo ships two files. docker-compose-simple.yml runs a bare node. docker-compose.yml adds Prometheus, Grafana, Tempo, Jaeger, and an optional Nginx. For a five-minute start you want the simple one. It is Apache 2.0 end to end, and the only thing you install is Docker itself.

Why Docker Compose instead of a bare docker run?

A bare docker run works, but it is one long command you have to retype and remember. The simple Compose file captures the same setup as a reproducible artifact. Ports, volumes, environment, restart policy, and a health check all live in one place you can commit to git. The README's own quick start offers both paths. The fast path is docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest. The maintained path is docker compose -f docker-compose-simple.yml up -d. Compose also handles the ownership gotcha for you. RustFS runs as UID/GID 10001:10001, and the volume-permission-helper service chowns the named volumes before the server starts. With a raw docker run and bind mounts, that permission step is on you.

What does docker-compose-simple.yml actually do?

The file defines two services. The rustfs service pulls rustfs/rustfs:latest, maps ports 9000 and 9001, and sets the environment that turns a generic image into your node. RUSTFS_VOLUMES=/data/rustfs{0...3} lays out four storage volumes. RUSTFS_ADDRESS=0.0.0.0:9000 and RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001 bind the API and console. RUSTFS_CONSOLE_ENABLE=true switches the web UI on. Credentials come from RUSTFS_ACCESS_KEY and RUSTFS_SECRET_KEY, both set to rustfsadmin in the file and marked # CHANGEME. A healthcheck hits /health on 9000 and /rustfs/console/health on 9001 every 30 seconds. The second service, volume-permission-helper, is an Alpine container that chowns the volumes to 10001:10001 and exits. That is the whole stack.

Here is the file, verbatim from the repo root:

# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

services:
  # RustFS main service
  rustfs:
    image: rustfs/rustfs:latest
    container_name: rustfs-server
    depends_on:
      - volume-permission-helper
    security_opt:
      - "no-new-privileges:true"
    ports:
      - "9000:9000" # S3 API port
      - "9001:9001" # Console port
    environment:
      - RUSTFS_VOLUMES=/data/rustfs{0...3}
      - RUSTFS_ADDRESS=0.0.0.0:9000
      - RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001
      - RUSTFS_CONSOLE_ENABLE=true
      - RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS=*
      - RUSTFS_ACCESS_KEY=rustfsadmin # CHANGEME
      - RUSTFS_SECRET_KEY=rustfsadmin # CHANGEME
      - RUSTFS_OBS_LOGGER_LEVEL=info
      - RUSTFS_OBS_LOG_DIRECTORY=/app/logs
      # Optional TLS:
      # - mount your cert directory to /opt/tls
      # - set RUSTFS_TLS_PATH=/opt/tls
      # - when TLS is enabled, health checks automatically switch to HTTPS
      # Keep strict disk topology checks enabled by default.
      # For local testing only, set `RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true` explicitly.
      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-false}

    volumes:
      - rustfs_data_0:/data/rustfs0
      - rustfs_data_1:/data/rustfs1
      - rustfs_data_2:/data/rustfs2
      - rustfs_data_3:/data/rustfs3
      - logs:/app/logs
    networks:
      - rustfs-network
    restart: unless-stopped
    healthcheck:
      # Default behavior:
      # - HTTP when RUSTFS_TLS_PATH is unset
      # - HTTPS when RUSTFS_TLS_PATH is set
      # - loopback TLS uses `-k` unless strict host+CA values are provided
      #
      # Strict TLS example:
      # - set RUSTFS_HEALTHCHECK_HOST to a SAN-covered hostname
      # - optionally set RUSTFS_HEALTHCHECK_CA (defaults to /opt/tls/ca.crt when present)
      test:
        [
          "CMD",
          "sh", "-ec",
          "host=\"$${RUSTFS_HEALTHCHECK_HOST:-127.0.0.1}\"; scheme=\"http\"; set -- -fsS; \
          if [ -n \"$${RUSTFS_TLS_PATH:-}\" ]; then \
            scheme=\"https\"; \
            ca_path=\"$${RUSTFS_HEALTHCHECK_CA:-}\"; \
            if [ -z \"$${ca_path}\" ] && [ -f /opt/tls/ca.crt ]; then ca_path=/opt/tls/ca.crt; fi; \
            case \"$${host}\" in 127.0.0.1|localhost) strict_host=false ;; *) strict_host=true ;; esac; \
            if [ \"$${strict_host}\" = true ] && [ -n \"$${ca_path}\" ]; then \
              set -- \"$${@}\" --cacert \"$${ca_path}\" --resolve \"$${host}:9000:127.0.0.1\" --resolve \"$${host}:9001:127.0.0.1\"; \
            else \
              set -- \"$${@}\" -k; \
            fi; \
          fi; \
          curl \"$${@}\" \"$${scheme}://$${host}:9000/health\" && \
          curl \"$${@}\" \"$${scheme}://$${host}:9001/rustfs/console/health\""
        ]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 40s

  # RustFS volume permissions fixer service
  volume-permission-helper:
    image: alpine
    volumes:
      - rustfs_data_0:/data0
      - rustfs_data_1:/data1
      - rustfs_data_2:/data2
      - rustfs_data_3:/data3
      - logs:/logs
    command: >
      sh -c "
        chown -R 10001:10001 /data0 /data1 /data2 /data3 /logs &&
        echo 'Volume Permissions fixed' &&
        exit 0
      "
    # Permission baseline:
    # - default RustFS runtime user is 10001:10001
    # - alternatively, run rustfs service with host-matched `user: \"<uid>:<gid>\"`
    restart: "no"

networks:
  rustfs-network:

volumes:
  rustfs_data_0:
  rustfs_data_1:
  rustfs_data_2:
  rustfs_data_3:
  logs:

How do you start the stack in under five minutes?

Three steps. First, install Docker Engine and the Compose plugin, or Podman plus podman compose. Second, fetch the official file with curl -O https://raw.githubusercontent.com/rustfs/rustfs/main/docker-compose-simple.yml, or copy it from the repo root. Third, launch it: docker compose -f docker-compose-simple.yml up -d. Compose pulls the image, starts the helper, then the server. The health check flips to healthy after roughly 40 seconds, the start_period. Verify the containers are up with docker compose -f docker-compose-simple.yml ps. You now have an S3 endpoint on localhost:9000 and a console on localhost:9001. If you prefer Podman, the README notes podman compose -f docker-compose-simple.yml up -d is a drop-in replacement.

How do you verify the S3 endpoint is alive?

The health check already proves the server answers, but a real S3 call is the better proof. Hit the API health endpoint directly: curl -fsS http://localhost:9000/health should return 200. For an actual object operation, point any S3 CLI at it. mc (MinIO Client) speaks RustFS's S3 dialect. Run mc alias set rustfs http://localhost:9000 rustfsadmin rustfsadmin, then mc mb rustfs/demo, then mc ls rustfs. A bucket listing means the endpoint, credentials, and console path all work. Open http://localhost:9001 in a browser and log in with rustfsadmin / rustfsadmin to see buckets, users, and IAM policies in the UI. Five minutes, done.

What should you change before exposing it to the network?

Two non-negotiables. First, change the credentials. The file ships rustfsadmin / rustfsadmin, which the inline # CHANGEME comment flags as public knowledge. Set RUSTFS_ACCESS_KEY and RUSTFS_SECRET_KEY to real secrets, or pass them via your env file, before any port faces the internet. Second, decide on TLS. The file shows the opt-in path: mount certs to /opt/tls and set RUSTFS_TLS_PATH=/opt/tls. The health check switches to HTTPS automatically when TLS is on. Leave RUSTFS_UNSAFE_BYPASS_DISK_CHECK at its default false in real deployments. Setting it true is for local throwaway testing only. Bind to localhost behind a reverse proxy unless you have a reason not to.

Where does RustFS fit next to MinIO and the rest?

RustFS positions itself as a MinIO-compatible, Apache 2.0 S3 store built in Rust, so the migration story is the draw. Your existing mc scripts, SDKs, and bucket policies mostly keep working. The README Feature & Status table marks S3 Core, Versioning, Object Lock (WORM), Lifecycle Management, Bucket and Site Replication, IAM/Policies, Server-Side Encryption, and Distributed Mode as available. The same surface MinIO users expect. The gap to name plainly: RustFS is a younger project, and two features are still preview, S3 Tables (Iceberg REST) and MinIO On-Disk Compatibility. If you need Iceberg tables in production today, that row is not your answer yet. For general S3 put/get, versioning, immutability, and IAM, the simple Compose stack is a credible replacement for MinIO.

What is still in preview, and what is ready?

Ready in the simple stack: single-node S3 on four local volumes, versioning, Object Lock, IAM, SSE, bucket replication, and the web console. All marked available in the README. In preview: S3 Tables (Iceberg REST) and MinIO On-Disk Compatibility, both shipped behind an opt-in flag or a bounded compatibility claim. There is no POSIX/FUSE mount either, so do not expect to mount RustFS like a filesystem. The simple file is one node. Distributed Mode exists and is marked available, but standing up erasure coding across hosts is a separate, larger topology, not the five-minute job. Read the Feature & Status table before you assume a capability.

FAQ

Can I run the RustFS Compose stack without Docker?

Yes. The README documents a drop-in Podman path: podman compose -f docker-compose-simple.yml up -d. Podman Compose reads the same file, so the stack is identical. You still need a container runtime; there is no native binary-only Compose equivalent.

Do I have to use the four-volume layout?

The simple file sets RUSTFS_VOLUMES=/data/rustfs{0...3}, four named volumes on the host. You can change the count or point at bind mounts, but keep every path writable by UID/GID 10001:10001 or run the rustfs service with a host-matched user:. The volume-permission-helper only fixes the named volumes it mounts.

Why does the console show rustfsadmin / rustfsadmin?

Those are the built-in defaults, carried in the Compose file as RUSTFS_ACCESS_KEY / RUSTFS_SECRET_KEY with a # CHANGEME note. They are public and well-known, so change them before exposing the listener. The README confirms the same default credentials.

Is RustFS free to self-host?

The server is Apache 2.0, so self-hosting, modifying, and embedding it carries no license fee and no network copyleft. You pay for the hardware and bandwidth, not the software. The observability add-ons in docker-compose.yml (Prometheus, Grafana, and so on) are their own licenses.

How do I upgrade the stack?

Change the image tag, rustfs/rustfs:latest or pin rustfs/rustfs:1.0.1, then docker compose -f docker-compose-simple.yml up -d again. Compose recreates the container; the named volumes persist your data across the recreate.

Sources

RustFS README: docker run command, default credentials, Feature & Status table, Compose commands: https://github.com/rustfs/rustfs
docker-compose-simple.yml (verbatim, repo root): https://raw.githubusercontent.com/rustfs/rustfs/main/docker-compose-simple.yml
docker-compose.yml (full observability stack): https://raw.githubusercontent.com/rustfs/rustfs/main/docker-compose.yml

S3 Presigned URLs: Security Pitfalls You'll Regret

Ethan Carter — Sat, 26 Sep 2026 13:24:00 +0000

S3 Presigned URLs: Security Pitfalls You'll Regret

A presigned URL is a time-limited, signed link that grants temporary S3 access without sharing your credentials. The regrets come from treating it like a password: over-long expirations, plaintext HTTP, no network lock, and signing with permanent keys. The hardening below is about the URL model itself, so it applies to Amazon S3 and to any S3-compatible endpoint that speaks SigV4.

Key stats

Limit / control	Value (Amazon S3)	Why it matters
CLI / SDK max expiration	604,800 s (7 days)	`--expires-in` caps here; the console caps at 12 hours
Default expiration	3,600 s (1 hour)	Set it explicitly; never leave the default for sensitive data
Temporary-credential URLs	Expire with the session	STS AssumeRole defaults to 1 hour; EC2 profile ~6 hours
Enforce HTTPS	`aws:SecureTransport = false` → Deny	Blocks plaintext HTTP requests
Max signature age	`s3:signatureAge > 600000` ms → Deny	Kills URLs older than 10 minutes

Every number above is from AWS's own presigned-URL and security-best-practices docs (links in Sources). We cite the docs, not blog lore.

What is an S3 presigned URL, really?

A presigned URL is a signed Amazon S3 link that wraps your security credentials and grants one specific operation — usually a GET to download or a PUT to upload — for a fixed window. Anyone holding the link can perform that operation, with no AWS account of their own. The signature is computed with your credentials under Signature Version 4 (SigV4), and AWS now uses SigV4 for every presigned URL, so the region must be set explicitly. You generate one with the AWS CLI:

aws s3 presign s3://amzn-s3-demo-bucket/mydoc.txt --expires-in 604800

That command returns a URL carrying X-Amz-Algorithm, X-Amz-Credential, X-Amz-Expires, and X-Amz-Signature. The operation is limited to the permissions of whoever signed it. In other words, a presigned URL is a short-lived, single-purpose capability delegation — not an account, and not a password you can rotate after the fact.

How long can a presigned URL stay valid?

The ceiling depends on how you create it. From the AWS CLI or SDK, --expires-in accepts up to 604,800 seconds — exactly seven days — and defaults to 3,600 seconds (one hour) if you omit it. The S3 console is stricter: it lets you set between 1 minute and 12 hours, with no path to a 7-day link through the UI. For programmatic generation, 7 days is the hard stop for URLs signed with long-term IAM user credentials under SigV4.

A longer window is rarely the right answer. A presigned URL is a bearer token: the moment it leaks, whoever holds it has the access it grants until it expires. If you set 7 days "just in case," you have given a stolen link a week of life. Favor minutes for download links and single-use uploads. The only reason to push toward the maximum is a genuinely long, unattended transfer where the recipient polls over days — and even then, prefer a refresh loop over one giant timeout.

Why do presigned URLs expire earlier than you set?

Because the URL is only as alive as the credential that signed it. AWS states this plainly: a presigned URL expires when its underlying credential is revoked, deleted, or deactivated, even if you asked for a later expiration. This bites hardest with temporary credentials. A URL signed by an IAM role session dies when that session ends, regardless of the --expires-in you specified. The same is true for STS AssumeRole (default session of 1 hour) and EC2 instance-profile credentials (which rotate with a maximum validity of about 6 hours).

The practical consequence: if you generate a "7-day" URL inside a Lambda using a role, it may stop working in under an hour. For long-lived links, sign with a dedicated IAM user whose key you can rotate, or build a service that re-issues the URL on demand. Treat the configured expiration as an upper bound, never a guarantee, and log which principal signed each URL so you can revoke the source fast.

The biggest pitfall: presigned URLs are bearer tokens

AWS calls them out directly: presigned URLs are bearer tokens that grant access to whoever possesses them, so they must be protected appropriately. The trap is forgetting that. Teams paste them into Slack, commit them to repos, log them with the request, or stuff them into client-side JavaScript where anyone can read them. Once a URL is out, you cannot rotate it like a password — you can only wait for it to expire or revoke the signing credential entirely, which kills every URL that credential ever made.

Three habits cut most of the risk. Never log the full URL; log a hash or just the object key. Generate the URL at request time on the server, not once at build time. Scope it to the smallest action and shortest time the caller needs. If a URL must travel to a browser, hand it over through a short-lived endpoint and expire it aggressively. The bearer-token model means the link itself is the secret — handle it like one.

How do you keep presigned URLs short-lived by default?

You can enforce a maximum signature age at the bucket level, so even a URL generated with a long --expires-in gets rejected once its signature is too old. AWS documents this with the s3:signatureAge condition key. The example below denies any presigned request on a bucket if the signature is more than 10 minutes old (600,000 milliseconds):

{
    "Version":"2012-10-17",
    "Statement": [{
        "Sid": "Deny a presigned URL request if the signature is more than 10 min old",
        "Effect": "Deny",
        "Principal": {"AWS": "*"},
        "Action": "s3:*",
        "Resource": "arn:aws:s3:::amzn-s3-demo-bucket/*",
        "Condition": {"NumericGreaterThan": {"s3:signatureAge": "600000"}}
    }]
}

This is defense in depth: the URL's own expiration might be an hour, but the bucket refuses anything signed more than 10 minutes ago. Tune the millisecond value to your workflow. Pair it with short expirations at generation time, and a leaked link becomes useless fast even if someone sits on it. Note this works for SigV4-signed requests and complements — not replaces — a small --expires-in.

How do you force HTTPS on every presigned request?

Presigned URLs carry credentials in the query string, so a plaintext HTTP handoff is a credential leak waiting to happen. Amazon S3's security guidance is to allow only encrypted connections and deny the rest with the aws:SecureTransport condition. The documented example bucket policy denies every request that is not TLS:

{
    "Version":"2012-10-17",
    "Statement": [{
        "Sid": "RestrictToTLSRequestsOnly",
        "Action": "s3:*",
        "Effect": "Deny",
        "Resource": [
            "arn:aws:s3:::amzn-s3-demo-bucket",
            "arn:aws:s3:::amzn-s3-demo-bucket/*"
        ],
        "Condition": {"Bool": {"aws:SecureTransport": "false"}},
        "Principal": "*"
    }]
}

With that in place, a presigned URL opened over HTTP returns Access Denied instead of serving the object. This applies to all requests to the bucket, presigned or not. One caveat: aws:SecureTransport is an AWS condition key; if you run an S3-compatible endpoint such as RustFS behind your own proxy, enforce TLS at the proxy or load balancer instead. The goal is the same — never let a signed URL traverse the wire in clear text.

How do you lock presigned URLs to a network?

By default a presigned URL works from anywhere on earth. If your recipients are internal, that is more reach than you want. AWS lets you constrain the network path with IAM condition keys. For requests hitting the public S3 endpoint, use aws:SourceIp; for traffic through a VPC endpoint, use aws:SourceVpc or aws:SourceVpce. The documented IAM statement denies all access unless it originates from a specific IP range:

{
    "Sid": "NetworkRestrictionForIAMPrincipal",
    "Effect": "Deny",
    "Action": "*",
    "Resource": "*",
    "Condition": {
        "NotIpAddressIfExists": {"aws:SourceIp": "IP-address-range"},
        "BoolIfExists": {"aws:ViaAWSService": "false"}
    }
}

This restricts the credential itself, so any URL it signs is also bound to that network. Apply it at the IAM principal, the bucket policy, or both. For a VPC-only architecture, the aws:SourceVpc / aws:SourceVpce pair keeps URLs from ever leaving your private path. Combine with the short signature-age and HTTPS rules above and a leaked link is both geofenced and short-lived.

Which credential should sign the URL?

The signing credential decides how much damage a leak causes and how fast you can contain it. Long-term IAM user access keys are the worst choice for high-churn URL generation: they do not rotate, and revoking one means editing every system that depends on it. AWS's own best-practice guidance is to use IAM roles for applications and services rather than embedding long-term credentials, precisely because roles hand out temporary, scoped permissions.

For presigned URLs, that means generating them inside a role session — a Lambda execution role, an EC2 instance profile, or an STS AssumeRole call — and accepting that the URL lives only as long as that session. If you genuinely need a multi-day link, isolate a single IAM user with a tight policy (only s3:GetObject on the specific prefix) and rotate its key on a schedule. Never sign URLs with an admin or root credential. The principle: the credential behind the URL should be the easiest thing in the chain to revoke.

RustFS and other S3-compatible endpoints: same rules, your perimeter

The hardening above is about the URL model, not about Amazon S3 specifically. Any S3-compatible endpoint that speaks SigV4 — RustFS among them — accepts the same presigned URLs your AWS SDK produces, and the same bearer-token risks apply. The difference is the perimeter: AWS gives you aws:SecureTransport, aws:SourceIp, and s3:signatureAge as managed condition keys; a self-hosted store puts that enforcement on you, at the proxy, the network policy, or the bucket policy if the server implements it.

RustFS ships under Apache 2.0 with S3 Core, Versioning, Object Lock, Server-Side Encryption, and IAM/Policies marked available, and it starts with one command:

docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest

Default credentials are rustfsadmin / rustfsadmin — rotate them before exposure. If you run your own S3 surface, you own the HTTPS enforcement and network lock the cloud gives you for free, so bake them into the ingress. The security pitfalls are identical; only the place you close them changes.

FAQ

Are presigned URLs safe to send over email?

Yes, if they are short-lived and HTTPS-only. A presigned URL in email is a bearer token, so keep --expires-in to minutes for downloads and never send one that grants PUT to a writable prefix. Enforce aws:SecureTransport so the link fails on plaintext HTTP.

Can a presigned URL be revoked before it expires?

Not the URL itself. You revoke the signing credential — delete or deactivate the IAM user key, or end the role session — and every URL it created stops working immediately, even with time left. There is no per-URL kill switch.

Why is my presigned URL returning AccessDenied?

Most often the signing principal lacks the underlying permission (for example s3:GetObject), the bucket policy denies it, the region is wrong, or the request headers (such as Content-Type on a PUT) differ from generation. Clock drift also breaks SigV4 signatures.

What is the maximum expiration for a presigned URL?

Seven days (604,800 seconds) from the AWS CLI or SDK with IAM user credentials under SigV4. The console caps at 12 hours. URLs signed with temporary credentials expire with the session — STS AssumeRole defaults to 1 hour.

Do presigned URLs work with S3-compatible stores like RustFS?

Yes. Any endpoint that implements SigV4-signed S3 requests accepts the same presigned URLs. The security model is identical; you just supply the HTTPS and network restrictions yourself instead of relying on AWS condition keys.

Sources

AWS — Share an object with a presigned URL (CLI command, 604800 s max): https://docs.aws.amazon.com/AmazonS3/latest/userguide/ShareObjectPreSignedURL.html
AWS — aws s3 presign CLI reference (--expires-in default 3600, max 604800): https://docs.aws.amazon.com/cli/latest/reference/s3/presign.html
AWS — Download and upload with presigned URLs (expiration by credential type, bearer-token warning, s3:signatureAge, aws:SourceIp): https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-presigned-url.html
AWS — Upload objects with presigned URLs (curl PUT example, SDK patterns): https://docs.aws.amazon.com/AmazonS3/latest/userguide/PresignedUrlUploadObject.html
AWS — Example bucket policies (RestrictToTLSRequestsOnly / aws:SecureTransport): https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html
AWS — Security best practices for Amazon S3 (IAM roles, least privilege, HTTPS): https://docs.aws.amazon.com/AmazonS3/latest/userguide/security-best-practices.html
RustFS README (Apache 2.0, Feature & Status, docker run, default credentials): https://raw.githubusercontent.com/rustfs/rustfs/main/README.md

Re-checked 2026-09-23.

DEV Community: Ethan Carter

AI Data Storage: Building the Pipeline Behind the Model

What counts as AI data in storage terms

Why did object storage win the ML dataset layer

How does a training-data pipeline actually flow

How do models read data during training

Checkpoints and artifacts are the write-heavy half nobody plans for

Where does RustFS fit as a self-hosted S3 layer

The lakehouse edge: Iceberg and the catalog question

What we would tell a team starting fresh

FAQ

What is the difference between a data lake and object storage for AI?

Do I need a POSIX filesystem mount for training data?

How big are model checkpoints, really?

Can RustFS replace cloud object storage for training?

What is S3 Tables and do I need it for AI?

A note on how this was verified

Sources

Spin Up a Self-Hosted S3 Bucket in 5 Minutes with RustFS

TL;DR

Why a self-hosted S3 bucket

Prerequisites

Step 1: Start RustFS

Step 2: Open the console and make a bucket

Step 3: Actually use it

Why RustFS specifically

Bonus: give back to RustFS (open source runs year-round)

FAQ

Do I need an AWS account to follow this?

Is rustfsadmin / rustfsadmin safe to leave?

Can I use this as my CI artifact store?

What if I need a filesystem mount, not an API?

A note on how this was verified

Sources

Feeding GPUs from Object Storage: A Self-Hosted S3 Pipeline for Kubernetes MLOps

Key Stats

What does feeding GPUs from object storage mean?

Why not just use NFS or local NVMe for training data?

Where RustFS fits: a self-hosted S3 tier

How do you deploy RustFS on Kubernetes?

How do training pods actually read from RustFS?

Checkpoints, artifacts, and the model registry bucket

What about a filesystem view? Built-in gateways first, s3fs second

A reference pipeline you can copy: kind + RustFS + a shard-pull Job

What actually changes on OpenShift

FAQ

Is RustFS a drop-in for S3 in my training code?

Can I run this on OpenShift specifically?

Do I need a CSI driver to mount RustFS?

How do I avoid egress and lock-in?

What is still maturing on RustFS?

A note on how this was verified

Sources

Object Lock Isn't Absolute: What WORM Really Blocks on S3 and Self-Hosted RustFS

Data preservation is keeping data exact on purpose

Why immutability beats a backup against ransomware

WORM and Object Lock: what governance and compliance modes actually do

Versioning: the safety net Object Lock depends on

Retention periods vs legal holds: fixed dates or indefinite?

How AWS S3 handles immutability

Running WORM on self-hosted RustFS

Hands-on: lock a bucket and try to delete a protected object

Lifecycle management: where retention meets automation

The self-hosting trade-off you can't skip

When should you self-host data preservation?

FAQ

What is the difference between Object Lock and bucket versioning?

Can a compliance-mode lock be removed early?

Does a legal hold expire?

What is variable retention / an event hold?

What happens if I try to expire a locked object with a lifecycle rule?

Is RustFS Object Lock production-ready?

A note on how this was verified

Sources

Apache Iceberg & Lakehouse: RustFS S3 Tables in Practice

What a lakehouse actually is

Why the table format became the battleground

Where object storage fits in the stack

What RustFS S3 Tables provides

Which engines connect — and the honest status

Is `rustfsadmin` / `rustfsadmin` safe to leave?