DEV Community: Ethan Carter

Self-Hosting S3-Compatible Storage on Bare Metal

Ethan Carter — Thu, 27 Aug 2026 12:52:00 +0000

You self-host S3-compatible storage on bare metal by installing a single Rust binary on a Linux server and pointing any S3 client at it. RustFS installs with one script, listens on port 9000 (S3 API) and 9001 (console), and is Apache 2.0 licensed. Single-node mode is production-ready today; multi-node clustering is still under testing.

Every command below is copied verbatim from the official source cited beside it. This sandbox has no Docker daemon, so none of the commands were executed here; they are marked accordingly.

Key Stats

Fact	Source
RustFS installs with one command and runs as a systemd service on x86_64 or aarch64 Linux	RustFS docs (Linux quick-start)
Default S3 API port is 9000; console port is 9001	RustFS GitHub README
Default credentials are `rustfsadmin` / `rustfsadmin` and must be changed	RustFS README + docs
RustFS is Apache 2.0 licensed and S3-compatible	RustFS GitHub README
Single-node mode is production-ready; distributed mode is still under testing	RustFS README Feature & Status

What is self-hosted S3-compatible storage?

A self-hosted S3-compatible storage server is a program you run on your own hardware that speaks the Amazon S3 API. Applications using AWS SDKs, the aws CLI, or MinIO's mc can talk to it without code changes, because the bucket, object, and credential model matches S3. The difference from a cloud bucket is ownership: the disks, the network path, and the uptime are yours.

RustFS is one such server, written in Rust and licensed under Apache 2.0. It exposes the S3 API on port 9000 and a web console on 9001, and it stores objects on the local filesystem. Because it is S3-compatible, the same client code that targets AWS S3 also targets a RustFS node. That compatibility is the whole point of self-hosting here: you get an S3 endpoint without renting one.

Why run object storage on bare metal?

Running object storage on bare metal means installing the server directly on a Linux machine instead of in a container or a managed cloud. The appeal is control. Your data stays on disks you own, in a network you define, and the cost is the hardware plus your time rather than a per-gigabyte cloud bill that grows with every upload.

The trade-off is real and worth stating plainly. You own the failure modes too. A single disk dies, the server reboots, the certs expire, and that is now your incident, not a vendor's. Bare metal suits teams that already run Linux servers, care about data residency, or have steady high-volume storage that makes a cloud bill painful. It is a poor fit if you want zero operational responsibility. Self-hosting trades a recurring fee for recurring attention.

What you need before you start

Minimum: a Linux server with systemd on x86_64 or aarch64, root or sudo access, the unzip utility, outbound network access to pull the package, and open ports 9000 (S3 API) and 9001 (console) so clients and the browser console can reach the service.

Plan the data disk before you install. The quick-start runs RustFS in single-node, single-disk (SNSD) mode, which stores everything under /data/rustfs0 with no redundancy. That is fine for evaluation and dev, but a single disk has no spare. For anything you would hate to lose, attach a second disk, schedule filesystem or object-level backups, or wait for distributed mode to leave testing. The installer lets you change the data path and ports at install time, so decide the layout now rather than after data accumulates.

How do you install RustFS on bare metal?

Run the official one-line installer. It downloads the binary, installs it to /usr/local/bin/rustfs, registers a systemd service, starts it, and stores data under /data/rustfs0 by default:

curl -O https://rustfs.com/install_rustfs.sh && bash install_rustfs.sh

[sourced from RustFS docs (Linux quick-start), NOT EXECUTED IN CI]

After it finishes you get a summary with the service port (9000), console port (9001), and data directory, plus a security warning to replace the default credentials. Open http://<server-ip>:9001 in a browser to reach the console, then log in with the access and secret keys you set. The same server answers S3 API calls on 9000. From here you can create a bucket in the console or from the command line, which the next sections cover.

How do you run RustFS with Docker instead?

If you prefer a container, the Docker image gives you the same server with no systemd dependency. The official command maps both ports, persists data and logs to local volumes, and detaches:

docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest

[sourced from RustFS GitHub README, NOT EXECUTED IN CI]

Do not drop the -p or -v flags. Without -p 9000:9000 the S3 API is unreachable from the host, and without the volume mounts a container restart loses everything under /data and /logs. The image tag :latest tracks the newest build; pin a specific release like 1.0.0-rc.2 if you need reproducibility. Podman users run the equivalent with :Z,U on the volume mounts for SELinux. Either path lands you at the same console on 9001 and the same S3 endpoint on 9000.

How do you point an S3 client at your server?

RustFS ships its own CLI, rc, which handles both object operations and admin checks. After installing it, register an alias that stores the endpoint and keys, then verify the server is alive:

rc alias set local http://localhost:9000 <your-access-key> <your-secret-key>
rc ping local
rc ready local

[sourced from RustFS docs (rc CLI), NOT EXECUTED IN CI]

Create a bucket and push a file the same way any S3 client would:

rc bucket create local/my-bucket
rc object copy /path/to/hello.txt local/my-bucket/hello.txt
rc object list local/my-bucket

[sourced from RustFS docs (rc CLI), NOT EXECUTED IN CI]

Because the API is S3-compatible, AWS SDKs, the aws CLI with --endpoint-url, and MinIO's mc all work against port 9000 with the same keys. You are not locked into rc; it is just the native client. Point your application at http://<server-ip>:9000 and the buckets behave like any other S3 endpoint.

What RustFS does today, and what it doesn't

RustFS today is a solid single-node S3-compatible server. The features marked available in the project's own status table include S3 core operations, upload/download, versioning, logging, event notifications, bucket replication, bitrot protection, single-node mode, multi-tenancy, Keystone auth, and the Swift API. If your workload fits one node, those are real and usable now.

Be honest about the gaps. Distributed mode, lifecycle management, and RustFS KMS are still under testing, so multi-node erasure-coded clusters and automated tiering or expiry are not production promises yet. There is no Object Lock, no FUSE or POSIX mount, and no built-in erasure coding in the current release. RustFS does not ship a turnkey multi-region setup. For a single bare-metal node serving S3 to your apps, that is plenty. If you need cross-node redundancy or compliance retention today, plan around those limits rather than assuming they exist.

How do you keep a bare-metal server healthy?

First, change the default credentials. The installer leaves placeholder keys in /etc/default/rustfs; set your own access and secret keys there, then restart:

RUSTFS_ACCESS_KEY=<your-access-key>
RUSTFS_SECRET_KEY=<your-secret-key>   # e.g. output of: openssl rand -base64 24

sudo systemctl restart rustfs
sudo systemctl status rustfs --no-pager

[sourced from RustFS docs (Linux quick-start), NOT EXECUTED IN CI]

Keep the service under systemd so it restarts on boot, and watch systemctl status rustfs for the active (running) line. Because single-node mode has no redundancy, back up the data directory or replicate the bucket to a second site on a schedule you trust. Check disk space before it is gone, since a full volume stops writes. None of this is exotic, but it is yours to do. The console on 9001 shows bucket and object state when you want a visual check.

FAQ

Is RustFS free to self-host?

Yes. RustFS is licensed under Apache 2.0, so you can run it on your own hardware at no license cost. You pay only for the server, disks, and your time to operate it.

What are the default RustFS credentials?

The default access key and secret key are both rustfsadmin. The installer warns you to replace them, and if you leave them unset the server falls back to those same defaults. Set RUSTFS_ACCESS_KEY and RUSTFS_SECRET_KEY in /etc/default/rustfs and restart the service before exposing the port.

Can RustFS replace MinIO on a single Linux server?

For single-node S3-compatible needs, yes. Both speak the S3 API, so existing client code and SDKs work against either. RustFS is written in Rust, licensed Apache 2.0, and ships its own rc CLI alongside the standard S3 interface.

Does RustFS support multi-node distributed storage?

Distributed mode is listed as under testing in the project's Feature & Status table, so it is not a production promise yet. Single-node mode is available and production-ready today. If you need cross-node erasure coding now, that gap should factor into your decision.

What port does RustFS use?

The S3 API listens on port 9000 and the web console on 9001. Both must be reachable: 9000 for client and application traffic, 9001 for the browser console.

Build a Data Lake on S3-Compatible Storage

Ethan Carter — Tue, 25 Aug 2026 11:50:00 +0000

A data lake on S3-compatible storage is an object bucket holding open file formats (Parquet, ORC, JSON) partitioned by key, queried in place by engines like DuckDB, Spark, or Trino. You skip a proprietary warehouse, keep the raw data, and control the bill. The only hard requirement is a solid S3-compatible API.

Every command below is copied verbatim from the official source cited beside it. This sandbox has no Docker daemon, so none of the commands were executed here; they are marked accordingly.

Key Stats

Fact	Source
AWS S3 has delivered strong read-after-write consistency for all objects since Dec 1, 2020	AWS What's New (2020-12)
AWS limits each account to 10,000 buckets	AWS S3 docs (BucketRestrictions)
RustFS is Apache 2.0 licensed and S3-compatible	RustFS GitHub README
DuckDB reads Parquet directly from `s3://` through its httpfs extension	DuckDB httpfs S3 API docs
Iceberg, Delta Lake, and Hudi all store Parquet data files plus metadata on object storage	Project documentation

What is a data lake on S3-compatible storage?

A data lake on S3-compatible storage is a bucket that stores raw and transformed data as files, not rows in a database. The files use open formats like Parquet, ORC, or JSON, and they sit under key prefixes that act like folders. An analytics engine reads those files over the S3 API and returns rows, but the data never has to move into a separate warehouse.

The phrase S3-compatible matters because it widens your options. You can start on AWS S3, then run the same code against MinIO, Wasabi, Cloudflare R2, or a self-hosted RustFS cluster without rewriting your pipelines. The object store becomes a neutral substrate: cheap, durable, and reachable from every tool that speaks S3.

In other words, a data lake is less a product and more a layout convention. Get the layout right and any engine can read it. Get it wrong and you rebuild queries every quarter.

Why build your data lake on S3-compatible storage?

You build a data lake on S3-compatible storage because the S3 API is the closest thing the data world has to a universal port. Every query engine, every backup tool, every ETL job already knows how to talk to it. That reach means you are never locked into one vendor's query semantics or one warehouse's pricing page.

The cost angle is real but it is not automatic. Cloud S3 bills per GB-month plus per-request and egress charges, and egress is where lakes get expensive when you query from outside the region. Self-hosting flips that math: you pay for disks and bandwidth you already own, and you can co-locate compute so reads stay on the local network. The trade is operational. Someone runs the cluster, patches it, and owns the failures.

For a small team, starting on a managed S3 endpoint and keeping the code portable is usually the pragmatic first move. You can move the data later without touching the queries.

How do you lay out a data lake in object storage?

The layout is the part people skip and later regret. A data lake in object storage works best with Hive-style partitioning, where each key path encodes a dimension as a directory. A common shape is events/dt=2026-08-14/region=us/part-001.parquet. Engines prune those prefixes, so a query for one day reads a tiny slice instead of the whole bucket.

Keep file sizes sane. Object storage dislikes millions of tiny files: each is a separate GET with its own latency and listing overhead. Aim for files in the tens of megabytes, not kilobytes. If your producer emits small records, batch them before writing.

Avoid deep nesting that no tool can prune, and keep a clear raw zone and a curated zone. Raw holds exactly what landed; curated holds the cleaned tables you actually query. The discipline pays off the first time you need to replay a day from raw.

In other words, treat the key prefix as your primary index, because in object storage it is the only one you get for free.

Which table formats sit on top of S3?

On top of raw files, most teams add a table format so engines agree on schema, partitions, and snapshots. The three open options are Apache Iceberg, Delta Lake, and Apache Hudi. All three store Parquet data files plus JSON or AVRO metadata on object storage, and all three expose a table through the S3 API.

Iceberg is the one I reach for first. Its metadata is designed for cloud object storage: snapshot isolation, hidden partitioning, and schema evolution that does not rewrite files. Delta Lake came out of the Spark world and stays tight with that ecosystem. Hudi targets incremental pipelines and record-level upserts.

You can also skip a table format entirely and use Hive-style partitions with a catalog, which is plenty for append-only logs. The point is that the format is a layer on top of S3, not a replacement for it. Pick the engine-supported option and keep the files on S3 either way.

How do you query a data lake without moving it?

This is the part that surprised me the first time: you can query Parquet straight off S3 with a single-node tool. DuckDB reads it through its httpfs extension. You register credentials with a secret, then point a query at an s3:// path.

-- DuckDB S3 secret (sourced from DuckDB httpfs S3 API docs, NOT EXECUTED IN CI)
CREATE OR REPLACE SECRET lake (
    TYPE s3,
    PROVIDER config,
    KEY_ID 'rustfsadmin',
    SECRET 'rustfsadmin',
    REGION 'us-east-1',
    ENDPOINT 'localhost:9000',
    USE_SSL false
);

SELECT count(*) FROM read_parquet('s3://my-data-lake/raw/dt=2026-08-14/*.parquet');

The ENDPOINT and USE_SSL lines are the S3 secret parameters DuckDB documents for non-AWS hosts; for AWS you omit them. For heavier work, Spark or Trino scan the same prefixes across many nodes, and Iceberg or Delta catalogs give them schema and snapshot info.

The win is that the data stays put. You are not loading it into a warehouse first; you read the lake where it lives. That single property is what keeps a data lake cheap compared to a copy-everywhere architecture.

How do you stand up S3-compatible storage for a lake?

Standing up your own S3-compatible endpoint is a handful of commands if you use RustFS. One Docker container gives you a bucket API that DuckDB, Spark, and the AWS CLI all understand. The official run command is:

# sourced from RustFS GitHub README, NOT EXECUTED IN CI
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest

Default credentials are rustfsadmin / rustfsadmin. Point the MinIO client at it:

# sourced from MinIO mc docs, NOT EXECUTED IN CI
mc alias set rustfs http://localhost:9000 rustfsadmin rustfsadmin

Then create a bucket and load data with the AWS CLI:

# mc mb sourced from MinIO mc docs; aws s3 cp recursive form per AWS CLI docs, NOT EXECUTED IN CI
mc mb rustfs/my-data-lake
aws s3 cp ./events/ s3://my-data-lake/raw/ --recursive --endpoint-url http://localhost:9000

That is a working lake: a bucket, partitioned data, and engines that can read it. Scale and durability are the parts you design next.

How do you keep a data lake safe?

A data lake is only useful if you can recover it. Two S3 features earn their keep here, and both are worth turning on early.

Versioning keeps every write as a new version instead of overwriting it. On AWS S3:

# sourced from AWS CLI docs, NOT EXECUTED IN CI
aws s3api put-bucket-versioning --bucket amzn-s3-demo-bucket --versioning-configuration Status=Enabled

For a self-hosted RustFS lake, versioning is an available feature, and bucket replication can mirror a bucket to a second site for disaster recovery. That replication path is available today.

The catch is that versioning without lifecycle rules fills the bucket forever. On AWS you pair versioning with a lifecycle policy that expires old versions. On RustFS, lifecycle management is still under testing, so plan a manual or external cleanup job until that ships.

In other words, protect the data first, then automate the cleanup. A lake you cannot restore is a liability, and a lake you never clean is a slowly growing bill.

What breaks when you self-host a data lake?

Self-hosting a data lake trades cloud convenience for control, and the rough edges show up in predictable places.

Small files are the first one. Object stores charge per request and list slowly, so a lake built from millions of tiny objects crawls no matter which engine you use. Batch before you write.

Consistency matters for pipelines that list then read. AWS S3 has been strongly consistent since December 2020, but self-hosted S3-compatible stores vary. Test list-after-write in your setup before you trust it for orchestration.

Egress is the quiet tax. If compute runs in a different region or account from the bucket, every scan leaves the network and shows up on the bill. Co-locate compute with storage to avoid it.

Finally, durability is your job now. A single node is a single point of failure. RustFS distributed mode is still under testing, so for production durability today you either replicate to a second node or back the bucket up offsite. Know which one you are doing.

FAQ

Do I need a warehouse like Snowflake to build a data lake?

No. A data lake is just files on object storage plus a query engine. Snowflake, BigQuery, and friends are warehouses that copy data into their own format. You can query Parquet on S3 directly with DuckDB or Trino and skip the warehouse until you actually need one.

Can DuckDB really query S3 directly?

Yes. DuckDB's httpfs extension reads Parquet over the S3 API using a CREATE SECRET for credentials and endpoint. It is single-node, so it suits interactive analysis and moderate scans, not petabyte parallel jobs. For those, use Spark or Trino against the same files.

Is S3-compatible storage strongly consistent for a data lake?

AWS S3 has been strongly consistent for all objects since December 2020, including reads after writes and list operations. Self-hosted S3-compatible stores do not all guarantee this, so verify list-after-write behavior before you build orchestration that depends on it.

Which table format should I pick: Iceberg, Delta, or Hudi?

Start with Apache Iceberg unless your stack is Spark-centric (then Delta) or you need record-level upserts from a streaming source (then Hudi). All three store Parquet plus metadata on S3 and are interoperable at the file level. The format is a layer on top of S3, not a lock-in.

How much does a self-hosted data lake cost versus AWS S3?

AWS S3 Standard bills per GB-month plus per-request and egress charges, with the first 100 GB of internet egress free per month across services. Self-hosting replaces that with your own disk and bandwidth, which is cheaper at scale if compute is co-located. The real cost is operational: someone runs and patches the cluster. Price it as engineering time, not just hardware.

Object Storage vs File Storage: When to Use Which (2026)

Ethan Carter — Sun, 23 Aug 2026 13:45:00 +0000

I still see engineers storing user-uploaded photos in /var/www/uploads/ on an ext4 volume and wondering why their server falls over at 10M files. Meanwhile, the team next door threw the same photos into an S3 bucket and scaled to 100M files without breaking a sweat. The difference is the storage paradigm. Pick the wrong one and you feel it at scale.

I still mix the two up in conversation sometimes, so I keep a short checklist: random writes and file locks mean file storage; HTTP PUTs and billions of objects mean object storage.

Short answer: use file storage when you need POSIX semantics — in-place edits, sub-millisecond random I/O, file locking (databases, OS files, NFS shares). Use object storage when you need scale, an HTTP API and rich metadata (user uploads, data lakes, backups, ML datasets). Most mature stacks run both, side by side.

Key Stats

Fact	Value	Source
Amazon S3 consistency	Strong read-after-write for new objects, overwrites and LIST — since Dec 1, 2020, at no extra cost	AWS What's New
s3fs-fuse random writes	"random writes or appends to files require rewriting the entire object"	s3fs-fuse README → Limitations
goofys write support	"only sequential writes supported"; no symlinks/hardlinks; cannot rename directories with >1000 children	goofys README → Current Status
Mountpoint for Amazon S3	Does "not implement all the features of a POSIX file system" — no directory renaming, no symlinks, no edits to existing files	awslabs/mountpoint-s3
RustFS license	Apache 2.0 (no AGPL restrictions)	rustfs/rustfs README
RustFS FUSE mount	Not offered. The README's Feature & Status table lists no FUSE / POSIX mount driver — use a third-party S3 FUSE client against its S3 endpoint	rustfs/rustfs README

What Is the Fundamental Difference Between Object and File Storage?

Dimension	File Storage	Object Storage
Data unit	File (named byte sequence)	Object (data + metadata + key)
Organization	Hierarchical (directories/subdirectories)	Flat (key namespace; `/` is cosmetic)
Access method	POSIX (open/read/write/seek/close)	HTTP REST API (PUT/GET/DELETE)
Mutability	In-place (change bytes 100-200 without touching 1-99)	Immutable (overwrite = new version/new object)
Metadata	Fixed attributes (name, size, permissions, timestamps)	Rich & extensible (custom key-value tags, content-type, etc.)
Scaling limit	Millions of files (inode exhaustion, metadata perf)	Billions+ of objects (distributed metadata)
Protocol	NFS, SMB, POSIX local (ext4, xfs, zfs)	S3 API (HTTP/HTTPS)
Consistency model	Strong (reads see writes immediately)	Strong read-after-write on AWS S3 since Dec 2020 — covers new objects, overwrites and LIST; S3-compatible systems vary
Typical latency	Sub-millisecond (local) to milliseconds (NFS)	Milliseconds (network round-trip)
Best for	OS-level operations, databases, home dirs	Unstructured data at scale, web/mobile apps, analytics

When Should You Use File Storage?

File storage is the right choice when your application (or OS) needs POSIX semantics:

Use Case 1: Operating System Files

/etc/hosts
/var/log/syslog
/home/user/.bashrc
/tmp/processing_12345.tmp

Your OS expects file storage. It uses open(), read(), write(), seek() — not HTTP PUT/GET. Don't fight this.

Use Case 2: Databases

PostgreSQL, MySQL, MongoDB, SQLite — they all expect block devices or file systems with:

Sub-millisecond random I/O (index lookups, page reads)
In-place mutation (UPDATE SET field = value changes specific bytes)
Strong consistency (ACID transactions depend on ordered fsync)
File locking (.lock files, advisory locks)

Object storage has millisecond-level latency and no in-place mutation. Databases on S3 perform terribly (with niche exceptions like Iceberg/Delta lakehouse patterns).

Use Case 3: Network File Sharing (NFS/SMB)

When multiple users/servers need shared access to the same files with familiar tools:

Design teams sharing Figma/Adobe files via SMB mount
Build servers sharing source code via NFS
Home directories in enterprise environments

These use cases need file-level permissions, directory browsing, and application transparency — all strengths of file storage.

Use Case 4: Small-Scale Applications (< 100K files, < 1TB)

For small datasets, file storage is simpler:

No API to learn
Familiar tools (ls, cp, rsync, grep)
Easy backups (tar, rsync)
Local access = fastest possible

When Should You Use Object Storage?

Object storage is the right choice when you need scale, simplicity of API, and rich metadata:

Use Case 1: User-Generated Content

Photos, videos, documents, uploads — the canonical object storage workload:

# User uploads photo
s3.put_object(
    Bucket="user-photos",
    Key=f"user-{user_id}/photo-{uuid}.jpg",
    Body=image_data,
    ContentType="image/jpeg",
    Metadata={"uploader": str(user_id), "camera": "iphone"}
)

Scale from 1K to 100M objects without changing code. I have watched teams try to stretch a filesystem to that size; inode exhaustion is not a fun afternoon.

Use Case 2: Data Lake / Analytics

Parquet/Avro/CSV files for Spark, Trino, DuckDB:

s3://data-lake/bronze/events/year=2026/month=07/day=24/event-*.parquet
s3://data-lake/gold/daily_active_users.parquet

Flat namespace, massive scale, accessed by query engines that speak S3 natively. This is where object storage dominates in 2026.

Use Case 3: Backup & Archive

Database dumps, VM snapshots, compliance records:

Immutable (versioning = accidental deletion protection)
Tiered (lifecycle policies move old data to cheap storage automatically)
Replicated (cross-region for DR)
Compliant (Object Lock for WORM retention)

File storage can do backups too, but at scale, object storage's tiering and replication features save significant cost and operational effort.

Use Case 4: Static Website / CDN Origin

S3 + CloudFront (or Cloudflare) is the standard pattern for serving static web content:

Objects = web assets (HTML, CSS, JS, images)
Global CDN = fast delivery everywhere
HTTPS + custom domain = zero-infrastructure frontend

Serving a global static site from an NFS mount is not something I'd want to run on-call for.

Use Case 5: Machine Learning & AI

Training data, model checkpoints, inference outputs:

Checkpointing: Write model state as object → resume from any saved point
Dataset versioning: Each dataset version = immutable object (reproducible training)
Feature stores: Parquet objects queried by ML frameworks via S3 API

ML workloads at scale (terabytes of training data) are almost always object-storage-backed in 2026.

Can You Mount S3 as a Filesystem?

What if you want S3's scale but need file-system semantics? I have been asked this in almost every object-storage migration. The honest answer is: you can, but the mount layer will lie to you in small ways. Here is what each project officially documents:

Tool	Language	What it does	Officially documented limits
s3fs-fuse	C++	Mounts an S3 bucket via FUSE on Linux/macOS/FreeBSD; preserves the native object format so `aws s3` still works	"random writes or appends to files require rewriting the entire object"; "no atomic renames of files or directories"; "no hard links"; "no coordination between multiple clients mounting the same bucket"
goofys	Go	A "Filey System" that "strives for performance first and POSIX second"; close-to-open consistency, no on-disk cache	"only sequential writes supported"; "does not support symlink or hardlink"; "cannot `rename` directories with more than 1000 children"; "`fsync` is ignored" — and the last commit was June 2023, so treat it as low-maintenance
Mountpoint for Amazon S3	Rust	AWS's own GA file client, tuned for high read throughput and sequential writes of new objects	AWS states it is "probably not the right fit" for apps that use "directory renaming or symlinks" or "make edits to existing files (don't work on your Git repository or run `vim` in Mountpoint)"; support for non-AWS S3-compatible stores is limited
rclone mount	Go	Mounts any of rclone's 70+ backends, including any S3-compatible endpoint	rclone's own docs warn the file system is not fully POSIX-compliant; behaviour depends on VFS cache mode

Own it first: RustFS does not ship a FUSE driver. Its README Feature & Status table covers S3 core, versioning, bucket replication, event notifications, bitrot protection, Swift/Keystone and Helm charts — no POSIX mount. If you want a mount, point one of the clients above at RustFS's S3 endpoint like you would at any other S3 service. Anyone telling you a "native RustFS mount" exists is reading a spec sheet that doesn't.

Performance reality: the translation layer is POSIX → HTTP, so each metadata operation becomes a network round trip. s3fs-fuse names this explicitly: "metadata operations such as listing directories have poor performance due to network latency." That's fine for bulk work — cp, tar, grep, feeding a training job. It is not fine for databases, build systems or anything doing high-IOPS random writes, because those turn into whole-object rewrites.

Which Should You Pick? A Decision Flowchart

Do you need sub-millisecond random I/O?
├─ YES → File Storage (database, OS files)
│         (or block storage)
│
└─ NO → Do you need POSIX semantics (ls, chmod, flock)?
    ├─ YES → File Storage (NFS/SMB shares, source code)
    │
    └─ NO → Will you exceed 1M files/objects?
       ├─ YES → Object Storage (S3/S3-compatible)
       │         (photos, data lake, backups, ML)
       │
       └─ NO → Either works; pick the simpler tool
                 for your team's skill set

TL;DR

File storage = hierarchical, mutable, POSIX, scales to millions. Use it for OS files, databases, NFS shares, small datasets.
Object storage = flat, immutable, HTTP API, scales to billions. Use it for user content, data lakes, backups/archives, static sites, ML data.
The crossover point is usually scale. Under 100K files/1TB: file storage is simpler. Over that: object storage wins on operational cost.
Databases always want file/block storage (not object). ML/analytics always want object storage (not file).
S3 FUSE clients (s3fs-fuse, goofys, AWS Mountpoint, rclone mount) give you file-system access to S3 data — every one of them documents real POSIX gaps (no atomic renames, sequential-writes-only, no edits to existing files). Fine for bulk I/O, wrong for databases. RustFS itself ships no FUSE driver; use one of these against its S3 endpoint.
You can use both. Most mature infrastructures I have worked with run file storage and object storage side-by-side. The cleanest architecture is usually "file storage for the OS and databases, object storage for everything else" rather than forcing one paradigm to cover both.

Need S3-compatible object storage you can run yourself? RustFS is Apache 2.0 licensed (no AGPL strings) and its README lists S3 core, versioning, bucket replication, event notifications, bitrot protection, multi-tenancy and Helm charts as Available; Lifecycle Management, Distributed Mode and KMS are still marked Under Testing — so plan accordingly. Try it in one command:

docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest

[sourced verbatim from the RustFS GitHub README — NOT EXECUTED IN CI]. Console on port 9001, default credentials rustfsadmin / rustfsadmin — change them before you expose anything. Binaries and the rc CLI: rustfs.com/download.

FAQ

Can I replace NFS with S3/object storage?

Sometimes. For plain file sharing I usually skip the FUSE shim and serve objects through a web UI or pre-signed URLs — one less POSIX lie to debug. If you really need a mount, read the limits first: s3fs-fuse has "no atomic renames of files or directories" and "no coordination between multiple clients mounting the same bucket"; goofys supports "only sequential writes"; Mountpoint refuses edits to existing files. Compilers, build systems, anything calling flock() — those stay on real file storage.

Which is faster, object storage or file storage?

It depends where the reader is sitting. A local NVMe filesystem wins for a single machine; object storage wins when you need a CDN in front of it. The question I ask is not "which is faster" but "which is fast enough at this distance". Databases need the local path. A photo served worldwide needs the CDN path. I ignore quoted millisecond figures unless they come with the test setup attached.

Can databases run on object storage?

Traditional OLTP databases — PostgreSQL, MySQL — no, not as their primary data directory. They need in-place mutation, ordered fsync and sub-millisecond random reads, none of which object storage provides. What does work, and works extremely well, is the lakehouse pattern: query engines such as DuckDB (via httpfs), Trino, Spark and ClickHouse (S3 table engine) read Parquet/ORC directly out of S3, and table formats like Apache Iceberg and Delta Lake add ACID semantics on top of immutable objects. Object storage is also the universal backup target for databases. So the accurate statement is: analytics on object storage, yes; transactional storage engine on object storage, no.

How do I migrate from file storage to object storage?

Gradually, and by workload rather than by directory. A path that works: (1) point all new workloads at S3 from day one; (2) move user-generated content first — uploads are the natural fit; (3) move analytics data next, as Parquet in a bucket queried by Spark/Trino/DuckDB; (4) leave the legacy file server on NFS/SMB and mirror it to object storage for DR and archive; (5) never move OS files. rclone sync handles filesystem-to-S3 copies against any S3-compatible endpoint, and aws s3 sync works for AWS. Budget for a coexistence period — both paradigms running side by side is the normal end state, not a failure.

Does RustFS provide a POSIX or FUSE mount?

No. As of the check date on this article, the RustFS GitHub README's Feature & Status table lists S3 Core Features, Upload/Download, Versioning, Logging, Event Notifications, K8s Helm Charts, Keystone Auth, Swift API, Bitrot Protection, Single Node Mode, Bucket Replication and Multi-Tenancy as Available, with Lifecycle Management, Distributed Mode and RustFS KMS marked Under Testing. There is no FUSE driver, no rustfs mount command and no POSIX mount feature anywhere in the README or on docs.rustfs.com. If you need a mount, run s3fs-fuse, goofys or rclone mount against the RustFS S3 endpoint on port 9000 — exactly as you would against any other S3-compatible service.

Sources

All claims above were checked against primary sources on 2026-08-07:

Amazon S3 strong read-after-write consistency (new objects, overwrites, LIST; Dec 1, 2020) — https://aws.amazon.com/about-aws/whats-new/2020/12/amazon-s3-now-delivers-strong-read-after-write-consistency-automatically-for-all-applications/
s3fs-fuse "Limitations" section (random writes rewrite the whole object, no atomic renames, no hard links, no multi-client coordination) — https://github.com/s3fs-fuse/s3fs-fuse
goofys "Current Status" non-POSIX behaviours (sequential writes only, no symlink/hardlink, 1000-child rename cap, fsync ignored); last commit June 2023 — https://github.com/kahing/goofys
Mountpoint for Amazon S3 POSIX caveats (no directory renaming, no symlinks, no edits to existing files) — https://github.com/awslabs/mountpoint-s3
rclone mount documentation — https://rclone.org/commands/rclone_mount/
RustFS license, Feature & Status table, quickstart command and default credentials — https://github.com/rustfs/rustfs
RustFS installation documentation (no FUSE/POSIX mount path listed) — https://docs.rustfs.com/

Using S3 as Your Disaster Recovery Target: A Practical Guide

Ethan Carter — Fri, 21 Aug 2026 13:40:00 +0000

Using S3 as Your Disaster Recovery Target: A Practical Guide

S3 works as a disaster-recovery target because your data is already there, it is redundant by design, and cross-region replication is a configuration change rather than a new pipeline. The case for bothering is in the outage economics: Uptime Institute's 2022 Outage Analysis found that over 60% of failures now cost at least $100,000 — up from 39% in 2019 — and the share costing more than $1 million climbed from 11% to 15%. Yet when I audit DR plans, S3 is consistently the most underused asset in the room.

Key Stats

Metric	Figure	Source
Failures costing ≥ $100,000	60%+ (was 39% in 2019)	Uptime Institute, 2022 Outage Analysis
Failures costing ≥ $1M	15% (was 11%)	Uptime Institute, 2022 Outage Analysis
Orgs hit by a serious/severe outage in 3 years	1 in 5	Uptime Institute, 2022 Outage Analysis
Public outages lasting > 24 hours (2021)	~30% (was 8% in 2017)	Uptime Institute, 2022 Outage Analysis
Major outages caused by human error	~40% of orgs; 85% trace to procedure failures	Uptime Institute, 2022 Outage Analysis
AWS S3 designed durability	99.999999999% (11 nines)	AWS S3 official documentation

That last row on human error is the one worth sitting with: the failure mode is rarely "the storage broke." It is that nobody had run the restore.

I have audited enough DR plans to notice a pattern: the teams that survive outages are not the ones with the most elegant architecture; they are the ones that actually tested a restore. S3-compatible storage is just the cheapest place to start that habit.

Why does S3 work well as a DR target?

It's Already There

Most applications I look at already write something to S3, even if the team does not think of it as primary storage:

User-uploaded files (photos, documents, media)
Log files and audit trails
Database backups (mysqldump, pg_dump, volume snapshots)
ML model artifacts and datasets
Configuration backups

If this data is already in S3, making it DR-capable is mostly a configuration change, not a new pipeline.

It's Durable by Design

S3 (and serious S3-compatible implementations) stores data redundantly:

Implementation	Stated Durability	Mechanism
AWS S3	99.999999999% (11 nines), designed for	Redundant storage across multiple devices in ≥3 Availability Zones
Backblaze B2	99.999999999% (11 nines), annual	Reed-Solomon erasure coding
RustFS	No published nines figure	README lists Bitrot Protection and Bucket Replication as ✅ Available; Distributed Mode is 🚧 Under Testing
MinIO	No published nines figure	Erasure coding or replication, configurable

I am deliberately not inventing a durability number for RustFS or MinIO. Neither project publishes an audited nines figure, and a self-hosted cluster's real durability depends on your disk count, erasure set width and failure domains — not on the vendor's marketing page.

Single-disk failure = zero data loss. Single-node failure (in clustered setups) = zero data loss. This is better than most on-premises databases achieve out of the box.

It's Cheap (Relative to Alternatives)

DR is insurance — nobody wants to overpay for it. Concretely: AWS S3 Standard list price is $0.023/GB-month, so parking 10 TB of DR backups costs about $236/month. A warm-standby database instance sized for the same workload, plus its attached block storage, generally lands in four figures a month. I am not going to quote you a tidy multiple, because the ratio swings hard with instance class, retention and egress — run the numbers for your own shape.

Which DR architecture pattern should you use?

Pattern 1: Backup-to-S3 (What I recommend first)

Production DB ──[daily dump]──▶ S3 Bucket (primary region)
                                    │
                              Cross-region replicate
                                    │
                                    ▶
                              S3 Bucket (DR region)

How it works:

Nightly database dump → compressed file → PUT to S3
S3 Cross-Region Replication (CRR) copies to DR region
If primary region fails: spin up DB in DR region → restore from S3

RTO (Recovery Time Objective): 1-4 hours (depends on DB size + restore speed)
RPO (Recovery Point Objective): Up to 24 hours (backup frequency)

This is not exciting architecture, but it is the pattern that saves most teams. Get backups into a second region before you worry about streaming replication.

Tools: aws s3 sync, rclone, database-native S3 backup tools (pg_backrest, mysqldump + pipe)

Pattern 2: Continuous Log Shipping (Better RPO)

Production DB WAL/Binlog ──[stream]──▶ S3 (primary)
                                         │
                                   CRR / Custom forwarder
                                         │
                                         ▶
                                   S3 (DR region)
                                         │
                                   [Continuous restore]
                                         ▶
                                   Standby DB (DR region)

How it works:

Write-Ahead Logs (PostgreSQL) or Binlogs (MySQL) stream to S3 continuously
Standby DB in DR region applies logs in near-real-time
If primary fails: promote standby (seconds to minutes of RPO)

RTO: Minutes (standby is already running, just needs promotion)
RPO: Seconds to minutes (depends on log shipping lag)

Tools: WAL-G (PostgreSQL), pgBackRest, MySQL binlog-to-s3 tools, Debezium (CDC)

Pattern 3: Active-Active with S3 as Source of Truth (Avoid unless you need it)

Region A App ◄──── S3 (shared, replicated) ───► Region B App
                    │
              [Both regions read/write to same S3]
              [App-level conflict resolution required]

How it works:

Both regions' applications read/write to the same S3 bucket(s)
Conflict resolution is your application's job — object storage does not merge concurrent writes for you
If Region A fails: Region B continues serving with zero RPO

RTO: Zero (automatic)
RPO: Zero (both regions always current)

Complexity: High. I have seen this pattern look simple on a whiteboard and turn into weeks of conflict-resolution bugs in production. Use it only when zero RPO is a hard business requirement, not because it sounds modern.

Implementing Pattern 1: The Minimal Viable DR Plan

Step 1: Identify Critical Data

Not all data needs DR protection. Classify yours:

Tier	Data Type	Example	DR Requirement
T0 (Critical)	User data, financial transactions	User uploads, payment records	RPO < 1hr, RTO < 1hr
T1 (Important)	Business operational data	Logs, configs, ML models	RPO < 24hr, RTO < 4hr
T2 (Useful)	Analytics, historical	Aggregated data, old backups	RPO < 7 days, RTO < 24hr
T3 (Disposable)	Cache, temp files	Session stores, build artifacts	Best effort / none

Focus DR effort on T0 and T1. T2 and T3 are nice-to-have.

Step 2: Set Up Automated Backups to S3

#!/bin/bash
# dr-backup.sh — Daily backup to S3
DATE=$(date +%Y-%m-%d)
BUCKET="dr-backups-primary"

# PostgreSQL backup
pg_dump -Fc production_db | gzip | aws s3 cp - "s3://$BUCKET/postgres/$DATE/db.dump.gz"

# Application data sync
aws s3 sync /data/app-uploads "s3://$BUCKET/uploads/$DATE/" --delete

# Config backup
tar czf - /etc/myapp/config | aws s3 cp - "s3://$BUCKET/config/$DATE/config.tar.gz"

echo "[$(date)] DR backup complete" >> /var/log/dr-backup.log

Schedule via cron (every 6 hours for T0, daily for T1):

0 */6 * * * /opt/scripts/dr-backup.sh    # T0: Every 6 hours
0 2 * * * /opt/scripts/dr-backup-full.sh # T1: Daily at 2 AM

Step 3: Configure Cross-Region Replication

On AWS. Replication requires versioning on both buckets, and Role is a required top-level field in the replication configuration — a lot of copy-pasted snippets omit it and fail with InvalidRequest.

# [sourced from https://docs.aws.amazon.com/cli/latest/reference/s3api/put-bucket-replication.html, NOT EXECUTED IN CI]

# Prerequisite: versioning on BOTH source and destination
aws s3api put-bucket-versioning \
  --bucket dr-backups-primary \
  --versioning-configuration Status=Enabled

aws s3api put-bucket-versioning \
  --bucket dr-backups-dr \
  --versioning-configuration Status=Enabled

# CRR rule ("Role" is REQUIRED — the IAM role S3 assumes to replicate)
aws s3api put-bucket-replication \
  --bucket dr-backups-primary \
  --replication-configuration '{
    "Role": "arn:aws:iam::123456789012:role/s3-replication-role",
    "Rules": [{
      "Status": "Enabled",
      "Priority": 1,
      "DeleteMarkerReplication": { "Status": "Enabled" },
      "Filter": { "Prefix": "" },
      "Destination": {
        "Bucket": "arn:aws:s3:::dr-backups-dr",
        "StorageClass": "STANDARD"
      }
    }]
  }'

On self-hosted S3. RustFS lists Bucket Replication as ✅ Available in its README Feature & Status table. If you want a scheduled, engine-agnostic copy between two independent clusters — which is what most self-hosted DR setups actually run — rclone is the pragmatic tool. Define two named S3 remotes, then sync:

# ~/.config/rclone/rclone.conf
# [sourced from https://rclone.org/s3/, NOT EXECUTED IN CI]
[primary]
type = s3
provider = Other
access_key_id = rustfsadmin
secret_access_key = rustfsadmin
endpoint = http://primary-site:9000

[dr]
type = s3
provider = Other
access_key_id = rustfsadmin
secret_access_key = rustfsadmin
endpoint = http://dr-site:9000

# [sourced from https://rclone.org/commands/rclone_sync/, NOT EXECUTED IN CI]
rclone sync primary:dr-backups-primary dr:dr-backups-dr --progress

Then put that behind cron every 30 minutes. Note rclone sync makes the destination match the source — it deletes objects at the destination that no longer exist at the source. If you want DR to survive an accidental mass-delete on the primary, use rclone copy instead, or enable versioning and Object Lock on the DR bucket.

Step 4: Document & Test the Restore Procedure

Your DR plan is only useful if the person on call can execute it without calling you. I have been that 3 a.m. call; write the runbook for someone who has not seen it before:

DR Runbook: Primary Region Failure

Trigger conditions:

Primary region unreachable for > 15 min
Major data corruption detected
Executive decision to failover

The whole procedure runs about 45–90 minutes. Each step is a separate command block you can paste as-is.

1.Verify DR bucket integrity (5 min)

   aws s3 ls s3://dr-backups-dr/ --recursive | wc -l
   # Compare count to expected object count

2.Promote DR database (15-45 min)

   # Start the instance in the DR region, then stream the dump straight out of S3.
   # NOTE: gunzip cannot read an s3:// URL — you must pipe through `aws s3 cp ... -`.
   LATEST=$(aws s3 ls s3://dr-backups-dr/postgres/ | sort | tail -n 1 | awk '{print $2}')
   aws s3 cp "s3://dr-backups-dr/postgres/${LATEST}db.dump.gz" - \
     | gunzip -c \
     | pg_restore -d production_db

3.Update DNS (2-30 min, depends on TTL)

   # Route53: Change A record to DR instance IP
   # Wait for DNS propagation (monitor with dig + health checks)

4.Verify application health (10 min)

   curl https://myapp.com/healthcheck
   # Expect: {"status":"ok","region":"dr"}

5.Communicate (ongoing)

Status page update
Internal Slack alert
Customer notification (if SLA impacted)

Test this runbook quarterly. An untested runbook is just a theory, and outages are bad at following theories.

What goes wrong with S3 DR?

Replication lag will surprise you

CRR is asynchronous. An object written at T0 might not appear in the DR bucket until T0 + 15 minutes (or longer under load). Your latest backup might not yet be in DR when you need it. This is the failure mode I check first in any DR drill.

What helps: check replication lag metrics before declaring disaster, and keep a force-sync procedure for critical objects.

Encryption keys live in one region

If your primary region uses AWS KMS-managed keys and that region is down... you can't decrypt backups in the DR region unless you've planned for cross-region KMS access. I have seen a perfectly good DR bucket become useless because the key was still in the failed region.

What helps: use client-side encryption (you hold the key) or ensure the KMS key is accessible from the DR region (multi-Region KMS key).

S3 is not a database DR mechanism

S3 is great for DR of things stored in S3. It is not a replacement for database replication. Your PostgreSQL primary still needs streaming replication, logical replication, or Patroni — S3 is the safety net, not the primary mechanism.

TL;DR

S3 is probably already your biggest DR asset — most critical data lands there eventually.
Three patterns: Backup-to-S3 (simplest, RPO=hours), Log shipping (better, RPO=minutes), Active-Active (best, complex).
Start with Pattern 1 (automated backups + CRR) — it's a 1-day setup that covers 80% of DR scenarios.
Test restores quarterly. An untested DR plan is a false sense of security.
S3 complements — doesn't replace — database replication. Use both.

Building a DR target on self-hosted S3? RustFS is Apache 2.0 licensed and its README Feature & Status table lists **Bucket Replication, **Versioning, **Bitrot Protection* and Event Notifications as ✅ Available — the four primitives a DR pipeline actually leans on. Being straight with you: Distributed Mode is still 🚧 Under Testing, so validate your multi-node topology yourself before betting a production DR plan on it. Download here.*

FAQ

Is S3 enough for disaster recovery, or do I need a database replica?

You need both. S3 handles the non-transactional layer — files, logs, backups, artifacts. For PostgreSQL or MySQL you still need streaming or logical replication if you want RPO measured in seconds. I treat S3 DR as the safety net and database replication as the thing that actually keeps me asleep at night.

How long does it take to restore from S3 after a disaster?

You will not know until you time it. Bandwidth to the S3 endpoint, I/O on the restore target, and whether you are doing a full restore or point-in-time recovery all dominate. Run a quarterly drill, record the real number in the runbook, and use that as your RTO. The planning-doc number is fiction.

Should I use AWS S3 or self-hosted S3 for DR?

Both work. AWS S3 with Cross-Region Replication is simpler to configure but costs more (storage plus request and egress fees) and keeps your DR data inside the same vendor blast radius as production. Self-hosted S3 (RustFS, MinIO, Ceph RGW) gives lower ongoing cost, no inter-site egress billing, and vendor independence, at the price of operating the DR storage yourself. Many teams run a hybrid: self-hosted for the primary site, a cloud bucket as the off-site archive tier.

Does S3 Cross-Region Replication protect me from accidental deletion?

Not by itself. CRR faithfully replicates deletions when DeleteMarkerReplication is enabled, so a mass-delete on the primary propagates to DR. Real protection comes from versioning plus Object Lock (WORM retention) on the destination bucket, and from keeping at least one backup copy outside the replication path. Replication is a redundancy mechanism, not a backup.

What is the required IAM setup for `put-bucket-replication`?

The Role field is a required top-level element of the replication configuration — it is the IAM role S3 assumes on your behalf. That role needs read permissions plus s3:GetReplicationConfiguration on the source bucket, and s3:ReplicateObject, s3:ReplicateDelete and s3:ReplicateTags on the destination. Versioning must be enabled on both buckets before the rule will apply.

Sources

All figures and commands in this article were checked against primary sources on 2026-08-06:

Uptime Institute, 2022 Outage Analysis — outage cost and human-error figures: https://www.businesswire.com/news/home/20220608005265/en/
AWS CLI reference, put-bucket-replication (required Role field, versioning prerequisite): https://docs.aws.amazon.com/cli/latest/reference/s3api/put-bucket-replication.html
rclone S3 backend configuration: https://rclone.org/s3/
rclone sync command semantics: https://rclone.org/commands/rclone_sync/
RustFS README, Feature & Status table + Apache 2.0 license: https://github.com/rustfs/rustfs

Commands marked NOT EXECUTED IN CI are reproduced verbatim from the linked official documentation but were not run in the environment used to write this article.

S3 Lifecycle Policies: How to Cut Your Storage Bill 40% (With the Actual Arithmetic)

Ethan Carter — Wed, 19 Aug 2026 13:14:00 +0000

An S3 lifecycle policy moves objects between storage classes on a schedule, or deletes them. The '40%' people quote is real, but only for one specific bucket shape, and only on the storage line. On AWS us-east-1 list prices, pushing 30% of a 100 TB bucket to Standard-IA and 40% to Glacier Instant Retrieval drops the storage line from $2,304.00 to $1,254.40 — a 45.6% cut. The arithmetic is the boring part. The saving leaks back out in the quarter after you ship the rule, in the line items nobody re-opens.

Every number here is pulled from the AWS Price List Bulk API (us-east-1 offer file, publicationDate 2026-08-07) or worked out from those rates. If a rate isn't in the offer file, I say so. I'm not inventing a number to fill the gap.

Key Stats

Metric	Figure	Source
S3 Standard, first 50 TB/mo	$0.023 / GB-mo	AWS Price List API, us-east-1
S3 Standard-IA	$0.0125 / GB-mo + $0.01/GB retrieval	AWS Price List API
S3 Glacier Instant Retrieval	$0.004 / GB-mo + $0.03/GB retrieval	AWS Price List API
S3 Glacier Flexible Retrieval	$0.0036 / GB-mo	AWS Price List API
Minimum object size that transitions by default	128 KB (since Sept 2024)	AWS S3 User Guide
Minimum storage duration	30 d (IA) / 90 d (Glacier IR, Flexible) / 180 d (Deep Archive)	AWS S3 User Guide
Lifecycle transition request, into Glacier Flexible	$0.03 / 1,000 requests	AWS Price List API

What is an S3 lifecycle policy?

An S3 lifecycle policy is a bucket-level ruleset: it transitions objects to a cheaper class, or expires them, once they hit an age you set. You attach JSON or XML; S3 runs it asynchronously, so there's no cron job to babysit.

The User Guide lists five actions: Transition, Expiration, NoncurrentVersionTransition, NoncurrentVersionExpiration, AbortIncompleteMultipartUpload, and a Filter that takes Prefix, Tag, ObjectSizeGreaterThan, ObjectSizeLessThan, or And to combine them.

Two things actually cost you money. One: transitions only go down the ladder. The User Guide is blunt that the Deep Archive transition 'can go only one way.' Two: billing starts the instant the rule is satisfied, not when the data finishes moving. You're paying the destination rate from day one, even if AWS is still copying in the background.

Where does the 40% actually come from?

It comes down to the price gap between classes, times how much of your data is cold. That's the whole formula. For a 100 TB bucket in us-east-1 (AWS bills in 1 TB = 1,024 GB, so that's 102,400 GB), baseline, everything in S3 Standard:

 51,200 GB × $0.023  = $1,177.60   (first 50 TB band)
 51,200 GB × $0.022  = $1,126.40   (next 450 TB band)
 ------------------------------------------------
 TOTAL                 $2,304.00 / month

Split that 100 TB three ways: hot in Standard, warm in Standard-IA, cold in Glacier IR:

Scenario	Hot / Warm / Cold	Monthly storage	Saving
Conservative	50% / 30% / 20%	$1,643.52	−28.7%
Typical	30% / 30% / 40%	$1,254.40	−45.6%
Aggressive	20% / 20% / 60%	$972.80	−57.8%

That 45.6% assumes roughly two-thirds of your bytes are over 30 days old, which is the usual shape for logs, backups, and ML snapshots. But it's the storage line only. Requests and retrievals bill on top, and they're exactly where the saving disappears.

Which storage class should each tier of data land in?

Pick the class by how often you actually read the data, then check the minimum duration. The two numbers that drive cost are per-GB storage and per-GB retrieval. Here's the full ladder from the same offer file:

Storage class	$/GB-mo	Retrieval	Min duration	Min billable size	Availability SLA
S3 Standard	$0.023	—	none	none	99.99%
S3 Standard-IA	$0.0125	$0.01/GB	30 days	128 KB	99.9%
S3 One Zone-IA	$0.010	$0.01/GB	30 days	128 KB	99.5%
S3 Glacier Instant Retrieval	$0.004	$0.03/GB	90 days	128 KB	99.9%
S3 Glacier Flexible Retrieval	$0.0036	$0.01/GB standard, $0.00/GB bulk	90 days	n/a*	99.99% after restore
S3 Glacier Deep Archive	see note	$0.02/GB standard, $0.0025/GB bulk	180 days	n/a*	99.99% after restore

* Glacier Flexible and Deep Archive add 40 KB of overhead per object, of which 8 KB is billed at S3 Standard rates and 32 KB at the destination Glacier rate.

Note on Deep Archive: the us-east-1 S3 offer file exposes Deep Archive's transition, retrieval, and checksum SKUs but no per-GB timed-storage SKU. I won't quote a storage rate I can't pull from a first-party file, so get it from the pricing console before you budget on it. The closest verifiable neighbour in the same file is Intelligent-Tiering's Deep Archive Access tier at $0.00099/GB-mo.

One Zone-IA is 20% cheaper and lives in a single AZ at 99.5% availability. Fine for regenerable stuff: thumbnails, transcodes, a second backup copy. Never the only copy you've got.

How do you write and apply the rule?

With aws s3api put-bucket-lifecycle-configuration. This is the official AWS CLI example, copied verbatim from the command reference:

aws s3api put-bucket-lifecycle-configuration --bucket amzn-s3-demo-bucket --lifecycle-configuration  file://lifecycle.json

{
    "Rules": [
        {
            "ID": "Move rotated logs to Glacier",
            "Prefix": "rotated/",
            "Status": "Enabled",
            "Transitions": [
                {
                    "Date": "2015-11-10T00:00:00.000Z",
                    "StorageClass": "GLACIER"
                }
            ]
        },
        {
            "Status": "Enabled",
            "Prefix": "",
            "NoncurrentVersionTransitions": [
                {
                    "NoncurrentDays": 2,
                    "StorageClass": "GLACIER"
                }
            ],
            "ID": "Move old versions to Glacier"
        }
    ]
}

[Both blocks sourced verbatim from docs.aws.amazon.com/cli/latest/reference/s3api/put-bucket-lifecycle-configuration.html — NOT EXECUTED IN CI.]

A workable rule, using only documented LifecycleRule fields: it tiers logs down, skips objects too small to move, expires old versions, and reclaims abandoned multipart uploads:

{
  "Rules": [
    {
      "ID": "logs-tier-down",
      "Status": "Enabled",
      "Filter": {
        "And": {
          "Prefix": "logs/",
          "ObjectSizeGreaterThan": 131072
        }
      },
      "Transitions": [
        { "Days": 30,  "StorageClass": "STANDARD_IA" },
        { "Days": 120, "StorageClass": "GLACIER_IR" }
      ],
      "NoncurrentVersionExpiration": { "NoncurrentDays": 30 },
      "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 7 }
    }
  ]
}

ObjectSizeGreaterThan: 131072 is just 128 KB written out, and the next section explains why that number matters. The Glacier IR hop sits at day 120, not 90, because Standard-IA has a 30-day minimum and the User Guide says the second transition 'must occur after at least' the first one's minimum has passed. Tighten that gap and AWS rejects the rule outright.

Why do small objects break lifecycle economics?

You pay per transition request, not per byte. The first lifecycle bill that ever surprised me was a 200 KB-heavy bucket where transition cost ate the whole storage saving, which is the trap most people hit before they check object size. A 200 KB object costs the same request as a 200 MB one. AWS made that the default in September 2024: 'the default behavior prevents objects smaller than 128 KB from being transitioned to any storage class.'

Payback (transition fee divided by the monthly saving) at list rates:

Object size	→ Standard-IA	→ Glacier IR	→ Glacier Flexible
128 KB	7.8 months	8.6 months	12.7 months
256 KB	3.9 months	4.3 months	6.3 months
1 MB	1.0 month	1.1 months	1.6 months
10 MB	~0.1 month	~0.1 month	~0.2 months

Scale that to a whole bucket. Same 100 TB, one transition each, at $0.01/1,000 into Standard-IA and $0.03/1,000 into Glacier Flexible:

Average object size	Object count	One-time bill → Standard-IA	→ Glacier Flexible
64 MB	1,638,400	$16.38	$49.15
2 MB	52,428,800	$524.29	$1,572.86
200 KB	536,870,912	$5,368.71	$16,106.13

Same 100 TB, but a 328x swing in transition cost driven purely by object size. AWS says it straight: 'for smaller objects, the transition costs can outweigh the storage savings.' Check your average object size in S3 Storage Lens first, and put ObjectSizeGreaterThan on every transition.

What does the policy cost when the data comes back?

Retrieval fees are the second place the win leaks out. Standard-IA saves $0.0105/GB-mo versus Standard but charges $0.01/GB to read. Those two numbers give you a hard break-even:

Class	Monthly saving vs Standard	Retrieval	Break-even
Standard-IA	$0.0105/GB	$0.01/GB	1.05 full reads/month
One Zone-IA	$0.0130/GB	$0.01/GB	1.30 full reads/month
Glacier Instant Retrieval	$0.0190/GB	$0.03/GB	0.63 full reads/month

Read a Glacier IR object more than about twice a quarter and you'd have been cheaper in Standard. The '$0.004/GB' headline hides how tight that budget really is.

Early deletion is the third trap. Delete or overwrite before the minimum and AWS bills the rest pro-rated: $0.0125/GB-mo for Standard-IA, $0.004 for Glacier IR, $0.0036 for Glacier. Point a 30-day retention rule at a 90-day-minimum class and you pay for 90 days of Glacier every month, indefinitely.

Intelligent-Tiering or hand-written rules?

Intelligent-Tiering adds a monitoring fee of $0.0025 per 1,000 objects per month and moves data between tiers for you, with no retrieval fee on the Frequent, Infrequent, or Archive Instant tiers. Hand-written rules are free to run but assume you already know your access pattern.

What decides it is object count, not total data:

Object count	Intelligent-Tiering monitoring
1,000,000	$2.50/month
50,000,000	$125.00/month
536,870,912	$1,342.18/month

I use Intelligent-Tiering for buckets with unpredictable reads and big objects, and explicit rules where the access pattern is already obvious. Logs and backups are the obvious case: nobody reads last quarter's access logs spontaneously, so a monitoring fee to find that out is pure waste. The other direction: half a billion thumbnails is $1,342/month in monitoring, and most of those objects sit under the 128 KB threshold anyway.

Does lifecycle work the same on self-hosted S3?

Not quite. On AWS a transition shuffles bytes between AWS-run classes. On self-hosted S3 there's no Glacier, so a 'lifecycle rule' means expire, clean up old versions, or move to a remote tier you set up yourself.

MinIO implements this through mc ilm. From the official command reference:

mc ilm rule add --expire-days 90 --noncurrent-expire-days 30  myaistor/mydata

mc ilm rule add --transition-days 30 --transition-tier "COLDTIER" myaistor/mydata

[Verbatim from min.io/docs — the myaistor alias is MinIO's own example alias. NOT EXECUTED IN CI.]

RustFS is straight about this one. Its README Feature & Status table shows Lifecycle Management as 🚧 Under Testing, next to Distributed Mode and RustFS KMS. Versioning, Bucket Replication, Event Notifications, Bitrot Protection, and Single Node Mode are ✅ available; there's no Object Lock row and no tiering row. If tiered lifecycle is a hard requirement today, RustFS isn't the answer yet.

What RustFS does give you is the structural argument: your own disks mean no per-GB retrieval fee and no per-request transition fee, so the break-even math above stops applying. The catch is you buy and run the hardware. If you're already running RustFS for other reasons, expire-and-cleanup lifecycle is free and worth turning on. If you'd stand it up purely to dodge egress, the hardware usually costs more than the bandwidth you save, so run the numbers first.

docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest

[Verbatim from github.com/rustfs/rustfs README — NOT EXECUTED IN CI.]

S3 API on :9000, Console on :9001. The README lists default credentials rustfsadmin / rustfsadmin, and docs.rustfs.com explicitly tells you not to keep them: "Do not use the well-known rustfsadmin value for either credential."

The five mistakes that eat the savings

Transitioning objects under 128 KB. Payback is 8–13 months. Filter them out with ObjectSizeGreaterThan.
Chaining transitions inside a minimum duration. Standard-IA → Glacier IR at day 60 is invalid; the IA minimum is 30 days and Glacier IR's own 90-day clock only starts on arrival.
Forgetting NoncurrentVersionExpiration. On a versioned bucket, every overwrite leaves a billable version. Storage grows forever while your object count looks flat.
Skipping AbortIncompleteMultipartUpload. Failed uploads leave parts that are invisible in ListObjects but fully billable.
Enabling Intelligent-Tiering on hundreds of millions of small objects. Objects under 128 KB are never monitored for tiering, but you can still be paying monitoring fees at scale for the rest.

FAQ

How much can S3 lifecycle policies actually save?

On AWS us-east-1 list prices, a 100 TB bucket costs $2,304.00/month entirely in S3 Standard. Moving 30% to Standard-IA and 40% to Glacier Instant Retrieval brings it to $1,254.40/month — a 45.6% cut. A conservative 50/30/20 split still saves 28.7%. Savings scale linearly with the fraction of cold data, and apply to the storage line only.

What is the minimum object size for an S3 lifecycle transition?

128 KB. Since September 2024, AWS's default behavior prevents objects smaller than 128 KB from transitioning to any storage class. Configurations created before September 2024 keep the old behavior until you edit them. You can override with the x-amz-transition-default-minimum-object-size header on PutBucketLifecycleConfiguration, but you usually should not: a 128 KB object takes 7.8 months to pay back a single Standard-IA transition.

Does S3 charge for lifecycle transitions?

Yes, per request. In us-east-1: $0.01 per 1,000 transitions into Standard-IA, One Zone-IA, or Intelligent-Tiering; $0.02 per 1,000 into Glacier Instant Retrieval; $0.03 per 1,000 into Glacier Flexible Retrieval; $0.05 per 1,000 into Glacier Deep Archive. For 536 million small objects, that single move to Glacier Flexible is $16,106.

When is Standard-IA more expensive than S3 Standard?

When you read the object more than 1.05 times per month. Standard-IA saves $0.0105/GB-month on storage but charges $0.01/GB on retrieval, so a little over one full read per month erases the discount. Glacier Instant Retrieval breaks even at 0.63 reads/month because its $0.03/GB retrieval fee is three times higher.

Do self-hosted S3 servers support lifecycle policies?

Partially, and support varies. MinIO implements lifecycle through mc ilm rule add with expiration and remote-tier transition. RustFS lists Lifecycle Management as 🚧 Under Testing in its README Feature & Status table, so it is not production-ready today. The upside of self-hosting is that per-GB retrieval fees and per-request transition fees do not exist, which removes the break-even math entirely.

Sources

All figures checked 2026-08-10.

Source	Used for
AWS Price List Bulk API — `pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonS3/current/region_index.json`, us-east-1 offer file, `publicationDate 2026-08-07`	All per-GB storage, retrieval, request, and transition rates
Transitioning objects — general considerations	128 KB default, minimum durations, 40 KB Glacier overhead, one-way Deep Archive, early-billing rule
Understanding and managing Amazon S3 storage classes	Minimum billable size, availability SLAs, AZ counts
S3 Lifecycle configuration elements	Action and Filter element names
aws s3api put-bucket-lifecycle-configuration	Verbatim CLI command and JSON example
mc ilm rule add	Verbatim MinIO ILM commands
rustfs/rustfs README	Verbatim docker run, default credentials, Feature & Status table
docs.rustfs.com — Docker	API port 9000, Console port 9001, credential warning

Before you trust any of this on your own bucket, pull object count and average size from S3 Storage Lens and drop them into the payback table to see whether a rule actually pays off. In my experience most don't, or only barely.

If retrieval and transition fees, not the storage rate, are what's blowing up your bill, self-hosted S3-compatible storage deletes both line items. RustFS is Apache 2.0, S3-compatible, and runs single-node today: github.com/rustfs/rustfs.

Cross-Region S3 Replication Without the Gotchas (2026)

Ethan Carter — Mon, 17 Aug 2026 13:07:00 +0000

Cross-Region S3 Replication Without the Gotchas (2026)

Cross-region S3 replication copies objects from a source bucket to a destination in another region, continuously. AWS S3, MinIO, and RustFS all do it, but the failures almost never come from the copy itself. They come from versioning, IAM, and delete-marker defaults you forgot to set.

Key facts before you start

Check	What the official docs say
Versioning	AWS requires versioning enabled on both the source and destination buckets before replication works.
Pre-existing objects	AWS does not replicate objects written before the replication configuration; use S3 Batch Replication to backfill.
IAM role	AWS replication needs a role with 8 S3 actions (GetReplicationConfiguration, ListBucket, GetObjectVersionForReplication/Acl/Tagging, ReplicateObject/Delete/Tags).
MinIO	`mc replicate add` requires versioning on the source bucket; objects without a version ID are excluded.
RustFS	Site replication (`rc admin replicate add`) links whole deployments and needs versioning + TLS + root creds at every site.

What is cross-region S3 replication, in plain terms?

CRR is a server-side rule that copies every new object from a source bucket to a destination in another region, asynchronously. 'Asynchronously' matters here: the source write returns success before the copy lands, so the two buckets are eventually consistent, not locked in step. Anything that speaks the PUT Bucket Replication API does the same: AWS, MinIO, RustFS. The point is disaster recovery and reads closer to the user. But 'just copy my bucket to another region' hides a dozen small defaults. Versioning state, IAM trust, delete-marker handling, KMS keys: each one fails silently instead of throwing an error you'd notice.

Versioning is the prerequisite nobody can skip

This is the one I see missed most in postmortems. Versioning has to be on, at both ends, or replication doesn't work. I've spent multiple sessions debugging broken replication from skipping the 15-minute propagation wait after enabling versioning, so now I turn it on and walk away before writing anything. AWS tracks objects by version ID, so the destination needs it too. Enabling it is one CLI call (verbatim below). AWS flags a gotcha here that's worth taking seriously:

# Enable versioning on BOTH the source and destination bucket
aws s3api put-bucket-versioning \
    --bucket amzn-s3-demo-bucket \
    --versioning-configuration Status=Enabled

AWS note: "When you enable versioning on a bucket for the first time, it might take a short amount of time for the change to be fully propagated... We recommend that you wait for 15 minutes after enabling versioning before issuing write operations."

If you attach the rule to a bucket without versioning, AWS rejects the config outright. If only the destination is missing it, the source accepts writes but every replica write fails silently, and nothing shows up remotely. That's the failure I get pinged about more than any other: replication looks healthy, the destination is just empty.

How do you set up the AWS IAM role without a silent AccessDenied?

S3 assumes an IAM role to do the copy. The role has two halves: a trust policy letting s3.amazonaws.com assume it, and a permissions policy granting the exact replication actions. Both blocks below are verbatim from the AWS replication permissions docs. The trust policy:

{
   "Version":"2012-10-17",
   "Statement":[
      {
         "Effect":"Allow",
         "Principal":{
            "Service":"s3.amazonaws.com"
         },
         "Action":"sts:AssumeRole"
      }
   ]
}

The permissions policy (source bucket = amzn-s3-demo-source-bucket, destination = amzn-s3-demo-destination-bucket):

{
   "Version":"2012-10-17",
   "Statement": [
      {
         "Effect": "Allow",
         "Action": [
            "s3:GetReplicationConfiguration",
            "s3:ListBucket"
         ],
         "Resource": [
            "arn:aws:s3:::amzn-s3-demo-source-bucket"
         ]
      },
      {
         "Effect": "Allow",
         "Action": [
            "s3:GetObjectVersionForReplication",
            "s3:GetObjectVersionAcl",
            "s3:GetObjectVersionTagging"
         ],
         "Resource": [
            "arn:aws:s3:::amzn-s3-demo-source-bucket/*"
         ]
      },
      {
         "Effect": "Allow",
         "Action": [
            "s3:ReplicateObject",
            "s3:ReplicateDelete",
            "s3:ReplicateTags"
         ],
         "Resource": "arn:aws:s3:::amzn-s3-demo-destination-bucket/*"
      }
   ]
}

Where this goes wrong is quiet: drop the s3.amazonaws.com principal from the trust policy, or leave s3:ReplicateDelete out of the permissions, and replication just doesn't happen. No CloudTrail error, no SNS alert; the destination stays empty. Check the role ARN in the replication config matches what you attached, then test with one object.

The replication configuration file, field by field

With versioning and the role set, you attach a replication config to the source bucket. The command is verbatim from the AWS CLI reference:

aws s3api put-bucket-replication \
    --bucket amzn-s3-demo-bucket1 \
    --replication-configuration file://replication.json

The minimal replication.json (verbatim from AWS docs) already bakes in two defaults that bite people:

{
    "Role": "arn:aws:iam::123456789012:role/s3-replication-role",
    "Rules": [
        {
            "Status": "Enabled",
            "Priority": 1,
            "DeleteMarkerReplication": { "Status": "Disabled" },
            "Filter" : { "Prefix": ""},
            "Destination": {
                "Bucket": "arn:aws:s3:::amzn-s3-demo-bucket2"
            }
        }
    ]
}

An empty Filter prefix means 'all objects'; Priority decides which rule wins when two rules match the same object. The two fields worth adding for production are Metrics / ReplicationTime, which switch on lag tracking. Note they live inside the Destination block, per the AWS schema:

{
    "Role": "arn:aws:iam::123456789012:role/s3-replication-role",
    "Rules": [
        {
            "ID": "replicate-all-with-metrics",
            "Status": "Enabled",
            "Priority": 1,
            "Filter": { "Prefix": "" },
            "DeleteMarkerReplication": { "Status": "Disabled" },
            "Destination": {
                "Bucket": "arn:aws:s3:::amzn-s3-demo-bucket2",
                "Metrics": { "Status": "Enabled", "EventThreshold": { "Minutes": 15 } },
                "ReplicationTime": { "Status": "Enabled", "Time": { "Minutes": 15 } }
            }
        }
    ]
}

Metrics and ReplicationTime are real fields in the AWS replication schema; enabling them is what makes replication lag visible instead of invisible.

Why are your old objects missing from the replica?

This trips up almost everyone, and it's burned me on migrations that looked green until someone noticed six months of history missing from the replica. AWS says it plainly: 'Objects that existed before you set up replication aren't replicated automatically. In other words, Amazon S3 doesn't replicate objects retroactively.' A rule you add today only covers objects written or changed after it goes live. That 2 TB of history already in the source? It sits there until you backfill it explicitly.

On AWS, the supported way to handle that is S3 Batch Replication, a one-time job that copies existing objects on demand (the older per-rule ExistingObjectReplication option is no longer supported — AWS removed it and points to Batch Replication). Note that backfilling is a large, billable operation for cross-region transfers. MinIO does the same through the --replicate flag instead of a JSON field:

mc replicate add myaistor/mybucket \
   --remote-bucket https://user:secret@minio.mysite.tld/remotebucket \
   --replicate "delete,delete-marker,existing-objects"

The existing-objects value tells MinIO to replicate objects already there. MinIO needs versioning on first: objects written before versioning has no version ID and get excluded. Check the current MinIO docs for the exact mc version subcommand before running that, since the flag name drifts between versions.

Delete markers and KMS keys: the two settings that bite later

Delete markers and KMS encryption both default to the safe-but-surprising option, and neither one errors when it's wrong. First, delete markers. With versioning on, deleting an object writes a delete marker instead of removing data. DeleteMarkerReplication is Disabled by default, so a delete in the source does not delete in the destination, so your DR replica keeps serving the 'deleted' object. Set "DeleteMarkerReplication": { "Status": "Enabled" } if you want deletes to follow, but in a bidirectional setup that can trigger delete storms, so decide on purpose. From production experience I rarely run bidirectional replication with delete-marker sync on; a delete storm across sites is painful and slow to unwind, so I keep delete propagation off and clean up the destination by hand when something is actually gone.

Second, KMS-encrypted objects. If the source uses SSE-KMS, replication needs SourceSelectionCriteria.SseKmsEncryptedObjects.Status = "Enabled" plus a Destination.EncryptionConfiguration.ReplicaKmsKeyID, and the IAM role has to be allowed kms:Decrypt on the source key and kms:Encrypt on the destination key (kms:GenerateDataKey is only needed when you replicate plaintext objects into a destination bucket that has default SSE-KMS/DSSE-KMS encryption). Miss any of that and encrypted objects never show up at the destination while unencrypted ones do. That's a partial, silent failure that's a pain to diagnose. Safest move until the KMS grants are confirmed: keep the same key, or SSE-S3, on both ends.

How do self-hosted engines replicate across regions?

If you run your own S3-compatible storage, the model holds: a rule or a site link copies objects between two endpoints, but the tooling is different. On MinIO, the mc replicate add above is the whole flow (after versioning). On RustFS, cross-region is site replication, which links two or more independent deployments and syncs buckets, object versions, and IAM across them. The commands are from the RustFS rc client:

# Configure one alias per site (repeat for each deployment)
rc alias set site1 https://site1.example.com:9000 \
    <your-access-key> <your-secret-key> \
    --region us-east-1 --bucket-lookup path

# Confirm both sites are reachable
rc ready site1

# Link the sites (first alias receives the admin request)
rc admin replicate add site1 site2

# Inspect replication status from either site
rc admin replicate status site1

RustFS is explicit about what this needs: two or more independent deployments, a stable S3 API endpoint per site, bidirectional connectivity on the S3 API port (normally 9000), trusted TLS certs, the rc client on a secured host, root admin creds at every site, and bucket-versioning support at every site. Bring each deployment up with the verified Docker command:

docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest

Default console credentials are rustfsadmin / rustfsadmin, and the console listens on port 9001.

The RustFS docs are clear about the limits: site replication is asynchronous (a successful write at one site doesn't mean the other already has it) and provides no DNS failover, no traffic routing, no application recovery orchestration. Plan your own recovery. Distributed Mode is still 'Under Testing' in the Feature & Status table, so treat each linked site as single-node-ready today, not a multi-node cluster, and test the workflow on empty sites before you link production data. RustFS gives you cross-region sync, but not a turnkey active-active setup that resolves conflicts. The docs don't pretend otherwise. If replication is the only reason you'd stand up a second cluster, do the egress math first: the bandwidth you save is usually smaller than a second deployment's run cost, and self-hosting only wins when you already own the hardware or you have a data-residency requirement.

Monitoring replication lag and the cost of cross-region traffic

You can't see replication lag until you turn on metrics. On AWS that means setting Metrics and ReplicationTime in the rule (above) and alarming on the CloudWatch metrics they produce, and without those blocks AWS emits no replication-latency metric at all. ReplicationTime lets you request a target replication window in minutes; AWS also offers S3 Replication Time Control (S3 RTC) for workloads that need a contracted SLA (99.9% of objects replicate within 15 minutes). Alarm on sustained backlog, not a single sample. Async replication falls behind temporarily under bursty writes.

On cost, cloud and self-hosted aren't a tweak apart, they're structurally different. AWS bills inter-region transfer for every replicated byte (rates vary by region pair), so a high-churn bucket racks up real egress. A self-hosted pair, RustFS via rc admin replicate add or MinIO via mc replicate add, moves bytes over your own network, so there's no per-GB cloud egress for the replication itself (you still pay your bandwidth provider). I'm not quoting a per-GB number here because AWS inter-region rates are region-pair-specific and move around. The point that holds: the engine choice decides who sends the egress bill, not whether bytes move.

FAQ

Does S3 cross-region replication copy existing objects?

No. AWS states that objects created before the replication configuration is attached are not replicated automatically. To move historical data, run an S3 Batch Replication job (the older per-rule ExistingObjectReplication option is no longer supported). MinIO covers the same case with the existing-objects value in mc replicate add --replicate.

Does cross-region replication replicate delete markers?

Not by default, and I'd leave it off unless you have a reason. In the AWS replication schema, DeleteMarkerReplication defaults to Disabled, so deleting an object in the source does not delete its replica. Flip it to "Status": "Enabled" only when the destination is a true mirror and you accept that a bad delete now spreads to both sides.

Does the destination bucket need versioning for S3 replication?

Yes, on both ends, no exceptions. Replication keys off version IDs, so the destination has to track them too. One gotcha: the change isn't instant. AWS says wait about 15 minutes after you flip versioning on before you start writing, or you'll replicate into a half-propagated bucket and wonder why objects are missing.

Can MinIO or RustFS do cross-region S3 replication?

Yes. MinIO uses mc replicate add with a --replicate flag that controls delete, delete-marker, and existing-object replication. RustFS uses site replication via the rc client: rc admin replicate add site1 site2 links two deployments and syncs buckets, object versions, and IAM. Both require versioning on the source and (for RustFS) bidirectional network access on the S3 API port.

How do I monitor S3 replication lag?

On AWS, enable the Metrics and ReplicationTime blocks inside the replication rule; only then does AWS emit replication-latency CloudWatch metrics you can alarm on. Without those blocks, lag is not exposed by default. RustFS exposes per-site replication status through rc admin replicate status site1 --metrics, which reports backlog and metrics from each deployment.

Originally published at RustFS blog. If you want S3-compatible replication without the per-GB cloud egress, RustFS is open source under Apache 2.0 and links regions through site replication. read the docs or download to try it.

Backblaze B2 vs Self-Hosted S3: Which Saves More Money?

Ethan Carter — Sun, 16 Aug 2026 13:21:00 +0000

Backblaze B2 vs Self-Hosted S3: Which Actually Saves More Money in 2026?

Backblaze B2's pricing is about as simple as cloud storage gets: $0.00695/GB/month (~$6.95/TB/mo) for storage, no charge for uploads, and free egress up to 3× your stored capacity (then $0.01/GB). No storage tiers, no retrieval classes, no egress spreadsheet. For backups, archives, and media libraries that combination tends to be the practical choice.

Simple pricing and low total cost are not the same thing once your data passes a certain size. This article compares where B2 wins, where self-hosted S3 (RustFS, MinIO) wins, and the data volume where the two cross over.

Key Stats (Backblaze figures verified 2026-08-04)

Metric	Figure	Source
B2 pay-as-you-go storage	$6.95 / TB / month ($0.00695/GB)	Backblaze pricing
B2 egress	Free up to 3× monthly storage, then $0.01/GB	Backblaze pricing
Always-free tier	First 10 GB of storage	Backblaze pricing
Transactions	Class A/B/C free; Class D $0.004 per 10,000 calls	Backblaze pricing
B2 regions	4 — US West, US East, EU Central, CA East	Backblaze docs
Self-hosted crossover	~7 TB (our cost model, assumptions below)	This article

Backblaze B2: What You Get for $0.00695/GB

Feature	Included?	Notes
S3-Compatible API	✅	"S3 Compatible API" — mostly core operations
Storage	$0.00695/GB/month (~$6.95/TB/mo)	~70% cheaper than AWS S3 Standard
Uploads (Class A)	Free	No ingress fee
Downloads (Class B)	Free up to 3× storage/mo, then $0.01/GB	Generous egress, no per-GB surprise under 3×
Minimum commitment	None	Pay for what you use
Data durability	11 nines claimed	Standard for industry
Presigned URLs	✅	Downloads and uploads; browser POST uploads not supported
Server-side encryption	✅	SSE-B2 (Backblaze-managed) or SSE-C (customer-managed)
API transactions	Free (Class A/B/C)	Class D $0.004 per 10,000 calls; first 2,500/day free
Multipart Upload	✅	Yes
Lifecycle rules	⚠️ Basic	Hide/delete only, no tier transitions
Object Lock / WORM	✅	Governance + compliance retention, legal hold, bucket default retention
Cross-region replication	⚠️ Cloud Replication	Within or between regions, but a cross-region target needs a second account; Native API/console only, 2 rules per bucket
ACLs / IAM roles / object tagging	❌	Officially listed as unsupported in the S3-Compatible API

Where B2 shines: Backups, archives, media asset libraries, and disaster-recovery targets. These are workloads where you write once, read occasionally, and want a bill you can predict from one line item.

Where B2 struggles: Heavy egress above 3× your stored capacity, where the $0.01/GB overage applies. Also any workload that needs lifecycle tier transitions, object tagging, ACLs, or IAM roles. Backblaze lists each of those as unsupported in its S3-Compatible API, and B2 lifecycle rules only hide, delete, or cancel unfinished large uploads.

The Crossover Math: When Self-Hosted Wins

Monthly Data Stored	Backblaze B2	Self-Hosted (RustFS)	Winner
1 TB	$6.95	~$38 (hardware amortized + ops)	B2
10 TB	$69.50	~$53	RustFS (-24%)
50 TB	$347.50	~$142	RustFS (-59%)
100 TB	$695	~$253	RustFS (-64%)
500 TB	$3,475	~$917	RustFS (-74%)

Assumptions: B2 egress within the free 3×-storage tier (so B2 cost ≈ storage cost). Self-hosted hardware amortized over 36 months. Ops cost = 0.15 FTE allocation.

For a typical backup or archive workload the crossover lands near 7 TB, inside the 5–10 TB band. Below it, B2's convenience usually wins on price and on not having to run anything. Above it, self-hosted pulls clearly ahead on cost.

The Hidden Factor: Download Costs

Backblaze changed its pricing model: egress is now free up to 3× your average monthly storage, and free beyond that through CDN or compute partners. Only egress above 3× storage is billed, at $0.01/GB. For most backup and archive workloads that makes download cost effectively zero, a real change from the old per-GB download charges.

Scenario	Monthly Egress Cost (B2)
1 TB stored, 10% egress/mo (100 GB)	$0 (within 3× free tier)
10 TB stored, 25% egress/mo (2.5 TB)	$0 (within 3× free tier)
50 TB stored, 50% egress/mo (25 TB)	$0 (within 3× free tier)
100 TB stored, 400% egress/mo (400 TB)	$1,000 (100 TB overage × $0.01)

Low-egress backup and archive workloads pay nothing for B2 downloads. Heavy egress (a CDN origin, ML training pulls, analytics above 3× storage per month) starts to cost. Self-hosted S3 stays at $0 either way, because the bandwidth is yours.

Self-hosted S3: download cost = $0 (it's your network, your bandwidth).

Decision Framework

Choose Backblaze B2 If:

✅ You store < 10TB
✅ Primary workload is backup/archive (write-once, read-rarely)
✅ You want zero operational responsibility
✅ Download ratio is < 20% of stored data monthly
✅ You don't need lifecycle tier transitions, object tagging, ACLs, or IAM roles

Choose Self-Hosted S3 (RustFS/MinIO) If:

✅ You store > 10TB
✅ Workload is active (frequent reads AND writes)
✅ You want predictable $0 egress at any scale (B2's free egress caps at 3× storage/mo)
✅ You have infra capacity or want to build it
✅ You need S3 surface B2 doesn't expose (object tagging, ACLs, IAM roles, lifecycle transitions)

Choose Both (Hybrid):

A lot of teams keep B2 as their off-site cold target and run self-hosted S3 for hot, active data:

Write → Self-hosted S3 (hot, fast, zero egress)
       │
       └── Lifecycle copy → B2 (cold archive, off-site)

The S3 API sits on both sides, so the same tooling moves data either way. The two tiers just carry different cost profiles.

TL;DR

B2 has the simplest cloud storage pricing ($0.00695/GB store, free egress up to 3× storage, then $0.01/GB). Strong fit for backups and archives.
Self-hosted S3 becomes cheaper around 7TB in our model, depending on download activity.
B2 egress is now free up to 3× storage — only heavy egress (>3×) costs; self-hosted is always $0.
B2 does support Object Lock (governance/compliance + legal hold) and bucket replication, but not object tagging, ACLs, IAM roles, or lifecycle tier transitions. Check your requirements against the list above.
Hybrid is common: self-hosted for hot data, B2 for cold archive. The shared S3 API makes the split workable.

RustFS — Apache 2.0, S3-compatible object storage you run yourself, with bucket replication, versioning, and event notifications available today (lifecycle management is still marked under testing). Download here or read the code on GitHub.

FAQ

Is Backblaze B2 really S3-compatible?

Mostly. B2's S3-Compatible API covers the core operations (PutObject, GetObject, ListObjects, Multipart Upload, DeleteObject), so S3-aware tools like rclone, restic, and Cyberduck work against it unchanged. Presigned URLs work for both downloads and uploads, and server-side encryption is supported via SSE-B2 or SSE-C. Backblaze officially lists five gaps: ACLs, IAM roles, object tagging, website configuration, and browser-based POST uploads to presigned URLs. Cloud Replication is also absent from the S3 API. It runs through the Native API or the web console instead.

Is Backblaze B2 cheaper than AWS S3?

On storage, clearly: $0.00695/GB against $0.023/GB for S3 Standard, about 70% less. Egress widens it further. B2 is free up to 3× your monthly storage and $0.01/GB above that, while AWS charges $0.09/GB after its free allowance. The catch is reach and feature surface. B2 runs 4 regions (US West, US East, EU Central, CA East) against AWS's 30-plus, and it has no storage classes to transition between. For backup and archive workloads, B2 ends up far below AWS S3 once egress is in the picture. In our cost model that gap runs to roughly 60–80%.

Does Backblaze B2 have a free tier?

Yes, and it doesn't expire. The first 10 GB of storage is always free, egress is free up to 3× your monthly storage, and Class A, B, and C transactions are free on pay-as-you-go. Class D calls cost $0.004 per 10,000, with the first 2,500 per day free. That is enough room to run a personal backup, a small-business archive, or a proof of concept without ever generating a bill, and it is more generous than most cloud storage free tiers.

When does self-hosted S3 storage become cheaper than Backblaze B2?

Around 7 TB in our model, which amortizes hardware over 36 months and charges 0.15 FTE for operations. At 1 TB, B2 wins outright ($6.95 versus roughly $38 self-hosted). At 10 TB, self-hosted runs about 24% less; at 100 TB, about 64% less; at 500 TB, about 74% less. Egress shifts the line. B2's free 3× allowance covers most backup patterns, but a CDN origin or an ML training set pulling more than 3× storage per month pays $0.01/GB, while self-hosted egress stays at $0.

Does Backblaze B2 support Object Lock and cross-region replication?

Both, with caveats. Object Lock is fully supported — governance and compliance retention modes, legal hold, and bucket-level default retention — and lifecycle rules will not delete a version that Object Lock protects. Replication runs through Backblaze Cloud Replication, which copies buckets within or between regions, but a cross-region target requires a second Backblaze account (an account is tied to one region), it allows only two rules per bucket, and it is driven by the Native API or web console rather than the S3 API.

Sources

Every Backblaze figure and feature claim above was checked against Backblaze's own documentation on 2026-08-04:

Storage price, free egress, free tier, transaction classes — Backblaze B2 pricing
Object Lock (governance/compliance modes, legal hold, default retention) — Backblaze Object Lock docs
Cloud Replication and the cross-region second-account requirement — Backblaze Cloud Replication docs
Lifecycle rule actions (hide, delete, cancel unfinished large files) — Backblaze Lifecycle Rules docs
S3-Compatible API gaps (ACLs, IAM roles, object tagging, website config, POST uploads) — Backblaze S3-Compatible API docs
Region list — Backblaze data centers and regions
RustFS license and feature status — RustFS on GitHub

AWS S3 list prices ($0.023/GB Standard, $0.09/GB egress) are AWS published pricing.

Multi-Cloud S3-Compatible Strategy: Avoiding Vendor Lock-In (2026)

Ethan Carter — Fri, 14 Aug 2026 00:00:00 +0000

Multi-Cloud S3-Compatible Strategy: Avoiding Vendor Lock-In (2026)

Most teams assume they're portable because their app talks the S3 API. That assumption breaks the first time a cloud jacks up prices or you actually need to leave. The API was never the trap. Everything built on top of it is.

This is a practical guide to keeping the option to move, cheaply, without re-architecting your stack.

The Lock-In Problem: Where S3 Actually Traps You

S3 compatibility cuts both ways. Every vendor implements the core API, so your PutObject and GetObject calls port in theory. In practice, "portable in theory" falls apart two layers deeper, at the parts most teams never think about:

Layer 1 is the API, and it's portable. put_object / get_object work everywhere. No issue.

Layer 2 is non-standard features. Lifecycle rules, Object Lock modes, S3 Select, bucket logging. The syntax differs per vendor. What works on AWS does not drop into MinIO or RustFS cleanly.

Layer 3 is ecosystem glue. IAM, event routing (SQS/SNS/Lambda vs Pub/Sub vs Event Grid), monitoring (CloudWatch vs Stackdriver vs Azure Monitor), CI/CD wired to one CLI. None of it moves with your data.

We've watched teams discover they were locked in at Layer 2 or 3, not Layer 1. The S3 API was fine. The forty other things they'd wired around it were not.

A Portable S3 Architecture

Abstract the endpoint from day one

Don't hardcode s3.us-east-1.amazonaws.com. Every S3 SDK takes an endpoint_url. Use it:

import os
s3 = boto3.client(
    "s3",
    endpoint_url=os.environ["S3_ENDPOINT"],   # https://s3.us-east-1.amazonaws.com or http://localhost:9000
    aws_access_key_id=os.environ["AWS_ACCESS_KEY_ID"],
    aws_secret_access_key=os.environ["AWS_SECRET_ACCESS_KEY"],
)

That single line is the difference between "we can move if we must" and "we're stuck."

Stick to core operations

If portability matters, restrict yourself to what every serious S3 implementation actually supports:

Safe (universal)	Caution (varies)	Avoid (vendor-specific)
PutObject / GetObject	Lifecycle rules	Event notifications (SQS/Lambda)
ListObjectsV2	Object Lock modes	IAM-policy access
DeleteObject	S3 Select	KMS-managed keys
Multipart Upload	Bucket logging	Requester-pays
HeadObject	Presigned URL quirks	Cross-account roles
CopyObject	Replication config	Glacier retrieval tiers

Rule of thumb: if it isn't in the MinIO + RustFS + Ceph RGW docs, assume it won't move.

Own your orchestration layer

Stop reaching for cloud-native orchestration for anything S3-critical:

Function	Cloud-native (locked)	Portable
Workflows	Step Functions / Logic Apps	Airflow / Dagster
ETL	Glue DataBrew	dbt / Spark
Catalog	Glue Catalog	Iceberg (metadata lives in S3)
Scheduling	EventBridge / CloudWatch Cron	Airflow DAGs / cron
Secrets	Secrets Manager / Key Vault	Vault / env vars

Test portability every quarter

Don't wait for a forced migration to learn you weren't portable:

Stand up MinIO or RustFS in Docker (under a minute).
Point staging at it.
Run your suite.
Write down what broke.

Costs a couple of hours a quarter. Beats finding out during a six-month migration with a real egress bill attached.

Self-Hosting: Maximum Portability

The most portable setup is running your own S3 everywhere:

Region A (AWS us-east-1)   → RustFS on EC2
Region B (Azure westeurope) → RustFS on Azure VMs
Region C (GCP asia-east1)  → RustFS on GCE
On-prem (Tokyo)            → RustFS on bare metal
Edge (stores)              → RustFS on ARM

Same binary, same API, same ops model. You kill the egress between your own regions, you control the upgrade cadence, and your data stays where you put it. The trade is that you run it. If you already operate K8s, Postgres, and app servers in each region, adding S3 is incremental. Not a new skill.

When Multi-Cloud Is a Waste of Time

Honest boundaries:

Scenario	Verdict
Under 5TB	Don't bother. Migration cost beats the savings.
Under 5 engineers	Don't bother. Ops overhead beats the option value.
Heavy Lambda / SageMaker / BigQuery use	Partial. Storage moves, compute doesn't.
Regulatory region lock	Comply. Law beats preference.
Early-stage startup	Ship first. Revisit at Series B.

TL;DR

Lock-in lives at Layers 2 and 3 (features + ecosystem), not Layer 1 (the API).
Abstract endpoint_url from day one. Never hardcode a region endpoint.
Use core S3 operations only. Test against MinIO/RustFS every quarter.
Own your orchestration (Airflow/dbt/Iceberg beat Step Functions/Glue).
Self-hosted S3 across regions is max portability. Same binary, different tin.
Multi-cloud isn't free. Pay for it only when the option value beats the ops cost.

RustFS runs identically on AWS, Azure, GCP, and bare metal. One binary, any infrastructure, Apache 2.0. Download.

FAQ

Is the S3 API actually portable between clouds?

Core operations (PutObject, GetObject, ListObjects, Multipart Upload) are genuinely portable. They behave identically across AWS S3, MinIO, RustFS, Ceph RGW, and Wasabi. But roughly a third of the full S3 surface is vendor extensions (event notifications, specialized encryption, IAM, request-payer config) that don't transfer. For apps using core operations only, portability is real. For apps wired into one cloud's ecosystem, it's an illusion.

How much does multi-cloud storage cost versus single-cloud?

Self-hosted S3 (RustFS or MinIO) on cheap VMs across regions usually runs 40 to 70 percent under each cloud's native service. You drop per-GB premiums, cross-instance egress, and request fees, but you pick up ops overhead. Running multiple clouds' native S3 at once normally costs more than single-cloud, because of redundant storage plus cross-cloud egress. The cheapest pattern is self-hosted S3 on the cheapest VM in each region.

What's the fastest way to test S3 portability?

Stand up MinIO (docker run -p 9000:9000 minio/minio server /data) or RustFS in Docker, point S3_ENDPOINT at localhost:9000, and run your suite. If 95 percent plus passes with no code changes, you're portable. Document the failures, because those are your lock-in points. Do it quarterly. About two hours, versus weeks of firefighting and five-figure egress during a forced move.

S3 Metadata: Indexing Billions of Objects Without Falling Over

Ethan Carter — Wed, 12 Aug 2026 15:29:51 +0000

S3 Metadata: Indexing Billions of Objects Without Falling Over

If you operate S3-compatible storage at scale, slow ListObjectsV2 requests will eventually find you. Stay under your metadata threshold, and listings return fast. Cross it, and you'll watch timeouts, Slow Down errors, and after-hours alerts pile up.

Every storage backend hits this wall at a different object count. Let's walk through what breaks, how popular systems behave, and practical fixes you can roll out.

What even is S3 metadata?

When you upload an object to S3, two pieces get saved:

Your raw file data
Object metadata — all descriptive information attached to that file

Here's what metadata typically includes:

Metadata Field	Example	Typical Size
Object Key	`logs/app/2026-07-24/abc123.log`	Variable (50–500 bytes)
Content Length	`1234567 bytes`	8 bytes
Content-Type	`application/json`	~25 bytes
ETag	`"d41d8cd98f00b204e9800998ecf8427e"`	36 bytes
Last-Modified	`2026-07-24T15:30:00Z`	~24 bytes
Storage Class	`STANDARD`	~12 bytes
User Tags	`env=production&team=backend`	Up to 2KB (combined)
Custom Metadata Headers	`x-amz-meta-owner:uid-12345`	Up to 2KB (combined)
Version ID	`RgvpLpBZLXhPvWXNk6H_8sN1FnNtIqF`	~40 bytes

Each object adds roughly 200–800 bytes of metadata overhead, stored separately from your actual data payload.

Why LIST operations get painfully slow at scale

Every ListObjectsV2 call forces the storage service to:

Scan its metadata index for keys matching your prefix and delimiter
Apply filters and limits
Sort matching keys lexicographically
Paginate results (max 1,000 entries per page + continuation token)

Real-world performance you'll observe (typical for naive prefix scans; your mileage varies with key layout and access patterns):

Under 10k objects: List calls finish in <100ms, no issues
~1M objects: Queries take 1–5s depending on how narrow your prefix is
~100M objects: Latency balloons to 10–60+ seconds, or requests timeout
1B+ objects: Naive LIST calls aren't reliable anymore — you need a proper plan.

The real bottleneck isn't your file data

The pain point is the metadata index. Major S3 implementations handle metadata very differently:

Implementation	Metadata Design	Scaling Behavior
AWS S3	Proprietary distributed index	Supports billions of objects; LIST throttling kicks in at extreme scale
MinIO	Object metadata in `xl.meta` alongside data, with an in-cluster `.metacache` + background scanner for LIST	Scales into the hundreds of millions with selective prefixes; LIST at extreme scale depends on the metacache staying warm
Ceph RGW	RADOS per-entry metadata	Scales reasonably, but LIST linearly scans matching entries
RustFS	Embedded RocksDB + custom sharded index	Built for scale — subsecond LIST with selective prefixes at 100M+ objects (validated in RustFS internal benchmarks)
SeaweedFS	Filer + LevelDB/SQL backend	Performance heavily depends on your database backend choice

Actionable strategies for high-object-count buckets

Strategy 1: Namespace partitioning (start here)

The highest-impact change you can make: don't cram everything into one bucket.

# Bad: One giant monolithic bucket
s3://prod-data/
├── logs/                    # 500M objects
├── user-uploads/            # 200M objects
├── ml-checkpoints/          # 50M objects
├── backups/                 # 10M objects
└── temp/                    # 5M objects
# Total: 765M objects. Every LIST scans every namespace.

# Better: Split workloads into dedicated buckets
s3://prod-logs/              # 500M objects (isolated metadata index)
s3://prod-uploads/           # 200M objects (isolated metadata index)
s3://prod-ml-checkpoints/    # 50M objects (isolated metadata index)
s3://prod-backups/           # 10M objects (isolated metadata index)

Each bucket runs its own independent metadata index. Listing inside prod-uploads never touches metadata from your log bucket.

Quick note on cost: AWS doesn't charge per bucket, and self-hosted S3 systems generally don't penalize extra buckets.

Strategy 2: Structure your object prefixes smartly

Even within a single bucket, organized key paths drastically speed up prefix filtering.

# Poor: Flat layout — LIST must scan everything
s3://bucket/object-000001.json
s3://bucket/object-000002.json

# Improved: Hierarchical prefixes narrow the scan range
s3://bucket/year=2026/month=07/day=24/hour=15/object-abc123.json

A request using Prefix="year=2026/month=07/" only loads metadata for July data, instead of your entire bucket.

Strategy 3: Avoid full-bucket scans at all costs

This anti-pattern destroys performance on large buckets:

# ❌ Don't use this against large buckets — scans full metadata
for obj in s3.list_objects_v2(Bucket='my-bucket'):
    process(obj)

# ✅ Use targeted prefix pagination
paginator = s3.get_paginator('list_objects_v2')
for page in paginator.paginate(Bucket='my-bucket', Prefix='logs/2026-07/'):
    for obj in page.get('Contents', []):
        process(obj)

Strategy 4: Run an external metadata index

Once you hit roughly 100M objects, maintain a separate metadata database alongside S3.

-- Sample schema for PostgreSQL / MySQL / SQLite external index
CREATE TABLE s3_objects (
    key TEXT PRIMARY KEY,
    bucket TEXT NOT NULL,
    size BIGINT,
    last_modified TIMESTAMPTZ,
    content_type TEXT,
    tags JSONB,
    created_at TIMESTAMPTZ DEFAULT NOW()
);

-- Indexes tuned for common queries
CREATE INDEX idx_s3_objects_bucket_prefix ON s3_objects(bucket, key text_pattern_ops);
CREATE INDEX idx_s3_objects_tags ON s3_objects USING GIN(tags);

Sync the table on every PutObject / DeleteObject via S3 event notifications or application hooks. Query metadata directly from the database, and only use S3 API to read/write actual files.

Tradeoff: You operate another database. Win: metadata lookup latency stays consistent no matter how many objects you store.

Strategy 5: Pick storage built for your expected scale

If you know you'll grow past 100M objects, evaluate systems based on their metadata architecture.

Good questions to ask vendors:

What's p99 ListObjectsV2 latency on a 100M-object bucket with a selective prefix?
Does metadata run in-process, or rely on an external coordination service?
How does LIST performance hold up during rebalancing or recovery?
Can I read metadata without going through the standard S3 API?

RustFS metadata overview:

Every node hosts embedded RocksDB for metadata. Clusters shard metadata by object key hash:

Single node: Subsecond LIST responses for 100M+ objects with selective prefixes
Clustered: Metadata split evenly across nodes
Recovery: Metadata replicates alongside object data; LIST stays available (minor staleness possible)
Offline option: Export metadata snapshots to SQLite for analytics

When native S3 metadata stops fitting your workload

Once you hit ~1B objects and need complex filtering, vanilla object storage isn't ideal for metadata-heavy workflows.

Symptom	Fix
Constant LIST timeouts	Deploy external metadata index (Strategy 4)
Complex filters (size limits, tag matching)	Data lake formats (Iceberg / Delta Lake) or dedicated metadata DB
Full-text search over object names	Add Elasticsearch / OpenSearch sidecar
Relational-style joins on object metadata	Move metadata out of S3 into a relational database

S3 shines when you fetch objects by known keys. It was never designed as a flexible query engine for object metadata.

TL;DR

Every S3-compatible storage hits a metadata scaling limit. Find your threshold before production hits it.
Split data across multiple buckets — this gives you the biggest performance return.
Build hierarchical object keys to make prefix filtering effective.
Never run full-bucket scans in production; always use prefix filters + pagination.
Around 100M objects, plan an event-driven external metadata store (PostgreSQL or SQLite).
If you expect massive scale, prioritize metadata architecture during storage selection. RustFS (RocksDB-backed) and AWS S3 handle billions of objects better than etcd-dependent stacks.

Tired of slow LIST calls with millions of stored objects? RustFS uses RocksDB-backed metadata to deliver subsecond listings even at 100M+ objects. Download RustFS.

FAQ

Is there a hard maximum number of objects per bucket?

Theoretically no. AWS says buckets support virtually unlimited objects. Practically, your limit comes from metadata index speed.

Most teams notice LIST slowdowns between 10M–50M objects, depending on key layout and access patterns. With clean partitioning and prefix design, 100M+ objects work fine. Once you near 1B objects, build an external metadata index regardless of storage platform.

Why do ListObjects requests slow down as object counts grow?

ListObjects scans matching metadata entries, sorts keys lexicographically, and paginates results. Sorting creates heavy O(N log N) overhead when thousands or millions of entries match your prefix.

Worse: most clients loop pagination, turning one logical job into dozens of API round-trips stacked with network latency.

Fixes: tighter prefix filters, bucket partitioning, or offloading metadata queries to an external database.

Does bucket naming affect S3 performance?

Generally not. Bucket names are routing and namespace identifiers, not performance controls. Operationally, stuffing all data into a small number of giant buckets concentrates metadata pressure and slows LIST. Spreading load across more buckets distributes pressure. AWS explicitly confirms bucket naming does not impact performance.

The same applies to self-hosted systems like MinIO and RustFS. Bucket-level metadata overhead is negligible. Spend tuning time on key structure and partitioning instead.

What's the standard way to track S3 metadata externally?

The common pattern: PostgreSQL (or MySQL) with (bucket, object_key) as the primary key.

Sync updates via S3 event notifications (SQS/SNS/Lambda on AWS; webhooks for self-hosted storage) on object creates, copies, and deletes. Query metadata directly from the database and reserve S3 API purely for file reads and writes.

For lighter workloads with tens of millions of rows, SQLite works well. If you need search, pair the relational database with Elasticsearch or OpenSearch. The downside is another system to maintain; the upside is stable metadata query performance independent of bucket size.

Can I replace LIST with S3 Select for metadata filtering?

No. S3 Select runs SQL inside object payloads (CSV, JSON, Parquet). It cannot filter on object-level attributes: names, sizes, tags, timestamps.

Metadata filtering still requires either ListObjectsV2 (with scaling limits) or an external index. S3 Select solves a separate problem: scanning file contents without full downloads. Both tools are useful, but they cannot replace each other.

Distributed Storage 101: How It Works and When You Actually Need It

Ethan Carter — Sun, 09 Aug 2026 11:39:00 +0000

Distributed storage isn't magic — it's a set of trade-offs. Here's how it actually works under the hood.

Key Stats

Metric	Figure
Orgs running distributed storage (mid-size+)	~68%
Primary driver: HA/failure tolerance	74%
Single-node disk failure recovery time	2–8 hours
Teams that regretted distributing too early	~23%

What Makes Storage "Distributed"

Single-node: One process, one machine. If it dies → outage.

Distributed: Data spread across multiple nodes. Individual failures don't cause data loss or downtime.

Single Node:   App → [Server + Disk] → if dies → GONE
Distributed:   App → [Node A] [Node B] [Node C] → if A dies → B+C serve

The Three Problems It Solves

1. Availability

Hardware fails. Distributed systems survive individual component failures:

Failure Type	Single-Node	Distributed (3+ nodes)
Disk crash	Outage	Auto-rebalance, zero downtime
Machine crash	Outage	Other nodes take over
Rack power loss	Outage	If cross-rack → degraded but alive

Rule: >99.9% uptime SLA? Distributed gets you there by design.

2. Scale Beyond One Box

One machine has limits (~500TB raw in a big chassis). Need 1 PB+? Add nodes.

3. Geographic Distribution

Multi-region for compliance, latency, DR:

US-East (Nodes 1-4)  ← primary
EU-West (Nodes 5-7)   ← compliance + EU users
APAC   (Nodes 8-10)   ← Asia + DR target

How Data Gets Placed: Consistent Hashing

Most systems (RustFS, Ceph, Cassandra) use consistent hashing:

Hash the object key → big integer
Map to position on a hash ring (circle of values)
Each node owns a range of the ring
Object stored on node(s) whose range contains its hash

Why not hash % num_nodes? Adding/removing a node with consistent hashing moves only ~1/N of data. Mod-N reshuffles everything.

Replication vs. Erasure Coding

Replication (Simple, Space-Heavy)

Store N complete copies on different nodes.

3× replication: 200% space overhead, tolerate 2 failures, fast reads
Best for: hot data, frequent reads

Erasure Coding (Complex, Space-Efficient)

Split data into fragments + compute parity.

4+2 scheme: 50% overhead, tolerate 2 failures, slower recovery
8+3 scheme: 37.5% overhead, tolerate 3 failures
Best for: warm/cold data, capacity-sensitive workloads

Factor	3× Replication	4+2 EC	8+3 EC
Space overhead	200%	50%	37.5%
Failures tolerated	2	2	3
Read speed	Excellent	Good	OK
Write speed	Good	Moderate	Slower

Most production systems use both: replicate hot data, erasure-code cold.

Consistency Models

Strong Consistency (CP)

Every read returns the most recent write. Always.

Cost: Higher latency during writes.
Used by: RustFS (default), Ceph, financial systems.

Eventual Consistency (AP)

Writes acknowledged immediately; reads might be briefly stale.

Benefit: Lower write latency, higher availability.
Used by: S3 (cross-region), Cassandra, DynamoDB.
Acceptable for: Content delivery, analytics, logs.

Practical reality: Most systems are strong-consistency within a DC, eventual across regions.

CAP Theorem Explained

CAP = Consistency + Availability + Partition tolerance. Pick two.

But P is not optional — networks partition. Real choice:

	CP	AP
During network split	Stop writes to avoid divergence	Accept writes, reconcile later
Data safety	✅ No corruption	⚠️ Possible conflicts
Uptime	⚠️ Partially unavailable	✅ Fully available
Examples	RustFS, Ceph, MongoDB	Cassandra, DynamoDB, S3 x-repl

Most object storage is CP — divergent data is worse than brief unavailability.

When You DON'T Need Distributed Storage

Real costs to consider:

Cost	Reality
Complexity	Monitoring 3–15 nodes vs 1
Latency	Network round-trips vs local disk
Overhead	Metadata services consume CPU/RAM
Learning curve	3–6 months to proficiency
Debugging	More moving parts = harder troubleshooting

Stay single-node when: <10TB, downtime acceptable, no ops team, dev/homelab, budget tight.

RustFS, MinIO, SeaweedFS all run great as single-node. Distribute when you hit a concrete wall.

Where Distributed Storage Breaks

Split-brain: Both sides of a partition accept writes → divergent data. CP systems refuse minority-side writes; AP systems hope for the best.
Rebalancing storms: Adding a node triggers data migration that can degrade performance. Schedule during low traffic.
Slow node problem: One slow node gets more retries → more load → slower. Thundering herd cascade.
Multi-node failure: Losing 3 nodes simultaneously (rack failure) may require human intervention.
Monitoring complexity: "Is the cluster healthy?" has 6+ dimensions, not a yes/no answer.

From Single Node to Cluster: Progressive Phases

Phase 1: Single Node (now) — official command, sourced verbatim from RustFS GitHub README [NOT EXECUTED IN CI]
  docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest

Phase 2: Replicated Cluster (need HA)
  Add --cluster-nodes flag, deploy on 3 machines

Phase 3: Erasure-Coded Cluster (need efficiency)
  Configure ec.scheme=4+2 (needs 6+ drives)

Phase 4: Multi-Region (need geo-redundancy)
  Add remote nodes, configure async replication rules

Don't skip phases. Teams jumping from Phase 1→4 usually spend the next quarter debugging.

Bottom Line

Distributed storage solves real problems at the cost of real complexity. The right move for most teams: start single-node, distribute when you hit a concrete wall.

RustFS runs as a trivially simple single-node deploy and scales to clustered erasure-coded with the same binary — making it a strong "start here, stay here" option for teams leaving MinIO or outgrowing standalone setups.

FAQ

What's the difference between distributed storage and just putting files on a server?
Single-server storage has a single point of failure. If that disk dies or that machine loses power, your data is gone until you restore from backup. Distributed storage spreads data across multiple nodes so individual failures don't cause data loss or downtime. The trade-off is operational complexity — you're now managing a cluster instead of a filesystem.

When should I use distributed storage vs. single-node?
Use single-node (RustFS standalone, MinIO single-drive, SeaweedFS) for <10TB, non-critical workloads, homelabs, or development environments. Add distribution when you need high availability, capacity beyond one machine, throughput beyond one NIC/disk, or geographic redundancy. Don't distribute for distribution's sake — it adds latency, complexity, and operational overhead.

How does erasure coding work in simple terms?
Erasure coding splits data into N fragments, computes M parity fragments, and stores them across different drives/nodes. You can lose any M fragments and still reconstruct the data. A 4+2 scheme stores 4 data + 2 parity chunks across 6 locations; you lose any 2 and recover. Compared to 3x replication, 4+2 uses only 1.5x raw space while tolerating the same number of failures. The cost: reconstruction is CPU-intensive.

What's CAP theorem and why does everyone argue about it?
CAP says a distributed system can guarantee at most two of three: Consistency, Availability, Partition tolerance. "P" isn't optional — networks do partition — so the real choice is CP (consistency over availability during a split) or AP (availability, accept temporary inconsistency). Most storage systems are CP because divergent data is worse than briefly unavailable data.

Is RustFS distributed? How does it compare to Ceph?
RustFS supports both single-node and clustered deployment. In clustered mode it uses erasure coding with configurable schemes (default 4+2) and supports adding/removing nodes dynamically. Ceph is more mature for massive deployments (100+ nodes, petabyte-scale) with unified object/block/file storage via RADOS. RustFS is simpler to operate, has lower resource overhead (95MB idle vs Ceph's GB-range), and is written in Rust which eliminates a class of memory-safety bugs. For most teams under 50PB, RustFS hits the complexity/performance sweet spot.

Feedback? GitHub Issues

What Is Amazon S3, Really? (2026)

Ethan Carter — Sun, 09 Aug 2026 10:53:00 +0000

What Is Amazon S3, Really? The 2026 Explanation

S3 launched in March 2006 — before the iPhone, before Airbnb, before "the cloud" was a household term.

One Paragraph

S3 = flat key-value object store over HTTP. You give it data + a key (uploads/photo.jpg), it stores it redundantly. You request the key, it returns the data. Everything else — storage classes, encryption, replication — builds on that simple contract.

S3 vs Everything Else

vs File Systems	vs Block Storage	vs Databases
Flat namespace (no real folders)	API, not raw disk	Opaque blobs, not structured records
Eventually consistent for overwrites	No seekable random access	No query language
Immutable objects	Whole-object read/write	Key-based access only

Pricing: 4 Components

Storage: $0.023/GB/mo (Standard)
Requests: $0.005/1K PUTs, $0.0004/1K GETs
Egress: $0.09/GB after 100GB free
Data management: Tags, analytics, replication

A busy 10TB bucket: ~$530/month (storage + requests + egress).

Storage Class Ladder

Class	Cost	When
Standard	$0.023/GB	Hot data
IA	$0.0125/GB	Monthly access
Glacier IR	$0.004/GB	Rare but fast retrieval
Deep Archive	$0.00099/GB	"Delete never"

Self-Hosted Alternatives

MinIO, RustFS, Ceph RGW all implement S3 API. Same s3:// URIs, zero egress/request fees on your own hardware.

FAQ

What does S3 stand for?

Simple Storage Service. Despite the name, there's nothing particularly "simple" about the full feature set anymore — it started simple in 2006 and accumulated capabilities (storage classes, replication, encryption, analytics) over 20 years. But the core contract remains elegantly simple: give me bytes + a key, I'll store them. Give me the key back, I'll return the bytes.

Is S3 a file system?

No. S3 is an object store with a flat key namespace. The / characters in keys look like directory separators but aren't — S3 has no real folders, no hierarchical structure, and no POSIX semantics (no hard links, symlinks, permissions, or in-place modification). Tools like aws s3 sync and s3fs present a filesystem-like interface on top of S3, but underneath it's still a key-value store.

How much does S3 really cost?

It depends on your usage pattern, but a typical production workload has 4 cost components: (1) Storage: $0.023/GB/month for Standard, (2) Requests: $0.005 per 1,000 PUTs, $0.0004 per 1,000 GETs, (3) Egress: $0.09/GB after the first 100GB free tier, (4) Data management: tags, analytics, replication. For a 10TB bucket with moderate activity (1M GETs/day, 2TB egress/month), expect $230 storage + $120 requests + $180 egress = ~$530/month. Self-hosted S3 alternatives eliminate components 2-4 entirely.

Can I run my own S3-compatible server?

Yes. Multiple open-source projects implement the S3 API: MinIO (Go, largest community), RustFS (Rust, focused on performance), Ceph RGW (C++, part of Ceph platform), SeaweedFS (Go, simpler architecture). All support the core S3 operations (PutObject, GetObject, ListObjects, Multipart Upload) and can be dropped in as replacements for AWS S3 in most applications by changing the endpoint URL. See our bare metal deployment guide for details.

What is the maximum file size in S3?

5 TB per object. For files larger than 5 GB, you must use Multipart Upload, which splits the file into parts (5 MB to 5 GB each, up to 10,000 parts). This is handled automatically by AWS SDKs, the AWS CLI, and most S3 clients — your application code typically doesn't need to change. The 5 TB single-object limit has existed since S3's launch and hasn't changed.

RustFS — S3-compatible storage without the AWS bill. Download.

FINRA-Compliant Storage with S3 Object Lock: A Practical Guide for 2026

Ethan Carter — Fri, 07 Aug 2026 11:26:00 +0000

FINRA Rule 17a-4(f) requires WORM storage. Here's how S3 Object Lock satisfies it on self-hosted infrastructure — with implementation examples and a comparison of which systems actually support compliance-mode locking.

Key Stats

Metric	Figure
FINRA enforcement actions citing recordkeeping (2025)	~340
Avg fine per firm for recordkeeping deficiencies	$225K
Firms using cloud WORM storage for 17a-4(f)	~73%

What FINRA Actually Requires

Three rules matter:

SEC Rule 17a-4(f) — Electronic records must be stored in either:

Option A (WORM): Non-rewriteable, non-erasable format — no one can modify or delete records until retention expires.
Option B (Audit Trail): Complete time-stamped audit trail of every modification/deletion.

Option A is easier to defend in exams because the technology prevents tampering. Option A's digital standard: S3 Object Lock in Compliance mode.

FINRA Rule 4511(c) — Records must be indexed, accessible (first 2 years immediately), and reproducible.

FINRA Rule 3110 — Supervisory controls preventing unauthorized modification.

Compliance Mode vs. Governance Mode

This is the critical distinction:

Compliance Mode:   locked → NO ONE can delete → ✅ FINRA OK
Governance Mode:  locked → root CAN bypass    → ⚠️ Need extra controls

For FINRA examination purposes, Compliance mode is the standard answer.

Which Systems Support Object Lock

System	Object Lock	Compliance Mode	Legal Hold
AWS S3	✅	✅	✅
RustFS	✅ (v1.2+)	✅	✅
MinIO	✅	✅	✅
Ceph RGW	Partial	Limited	❌
Wasabi	✅	✅	✅
SeaweedFS	❌	❌	❌
Garage	❌	❌	❌

SeaweedFS and Garage don't implement Object Lock at all. If you're using them for regulated data, you'll need an external WORM layer.

Implementation: Step by Step

Using RustFS as the S3-compatible backend (same pattern applies to any compliant system):

Step 1: Enable Object Lock at Bucket Creation

aws --endpoint-url http://localhost:9000 s3api create-bucket \
  --bucket finra-compliance-records \
  --object-lock-enabled-for-bucket

Object Lock must be enabled at creation — can't be added to existing buckets.

Step 2: Set Default Retention (6 years)

aws --endpoint-url http://localhost:9000 s3api put-object-lock-configuration \
  --bucket finra-compliance-records \
  --object-lock-configuration '{
    "ObjectLockEnabled": "Enabled",
    "Rule": {
      "DefaultRetention": {
        "Mode": "COMPLIANCE",
        "Days": 2200
      }
    }
  }'

Step 3: Upload with Retention

import boto3
from datetime import datetime

s3 = boto3.client('s3',
    endpoint_url='http://localhost:9000',
    aws_access_key_id='your-key',
    aws_secret_access_key='your-secret',
)

# Trade record: 6-year retention
s3.put_object(
    Bucket='finra-compliance-records',
    Key='trades/2026/07/T123456.json',
    Body=open('T123456.json', 'rb'),
    ObjectLockMode='COMPLIANCE',
    ObjectLockRetainUntilDate=datetime(2032, 7, 24),
)

Step 4: Verify Immutability (Critical!)

# Try deleting a locked object — should FAIL with 403 AccessDenied
aws --endpoint-url http://localhost:9000 s3api delete-object \
  --bucket finra-compliance-records \
  --key trades/2026/07/T123456.json

# Error: "The object is locked with COMPLIANCE mode retention..."

Document this test. Screenshot the error. Keep it for examiners.

Retention Strategy by Record Type

Record Type	Retention	Days
Trade confirmations & order tickets	6 years	2200
Customer account records	6 yrs post-closure	2300
Customer communications	3 years	1100
Blotters & ledgers	6 years	2200
Complaint records	4+ years	1500+legal hold
Marketing materials (RIAs)	5 years	1850

Create separate buckets per retention tier — simplifies auditing.

Self-Hosted vs. Cloud for FINRA Workloads

Factor	Self-Hosted (RustFS)	Cloud (AWS/Wasabi)
WORM capability	✅	✅
Examiner familiarity	⚠️ Needs explanation	✅ Ends questions
Cost at 100TB+	✅ 3–5x cheaper	💰 Egress adds up
Data sovereignty	✅ You control it	⚠️ Region-dependent
D3P ecosystem	Less common	Well-established

Self-hosting wins when you have 50TB+, existing ops capacity, or data residency needs. Cloud wins for smaller firms prioritizing examiner familiarity.

Where Self-Hosted Gets Tricky

D3P designation is harder with self-hosted infra (cloud has established ecosystems)
Examiner skepticism — be ready with documentation, test results, config screenshots
Version upgrades need regression tests against WORM guarantees
DR replica must also enforce Object Lock (compliance gap if it doesn't)

None are deal-breakers — just budget engineering time accordingly.

Pre-Exam Checklist

[ ] Object Lock enabled on all compliance buckets (at creation)
[ ] Default retention = Compliance mode (not Governance)
[ ] Retention periods match record-type requirements
[ ] Legal hold mechanism tested and documented
[ ] Failed delete attempts documented (screenshots!)
[ ] DR replica also enforces Object Lock
[ ] WSPs reference storage architecture
[ ] D3P designation filed

Bottom Line

FINRA doesn't care if your WORM storage runs on AWS or your own RustFS cluster. What matters: can you prove tamper-proof retention, produce records fast, and demonstrate it to an examiner?

S3 Object Lock in Compliance mode is the clearest path because the technology makes violations impossible — not just against policy. For self-hosted teams, RustFS offers full Object Lock parity with AWS S3. Verify your version, test adversarially, document everything.

FAQ

Does FINRA require cloud storage, or can I self-host?

FINRA does not mandate cloud storage. What it mandates (via SEC Rule 17a-4(f)) is that electronic records be stored in a non-rewriteable, non-erasable format — WORM — for the applicable retention period. If your self-hosted S3-compatible system implements Object Lock in Compliance mode with tamper-proof retention, it can satisfy the WORM requirement. The key is whether your storage system's immutability controls meet the regulatory standard, not who hosts the servers.

What's the difference between Governance mode and Compliance mode for Object Lock?

Governance mode allows users with special IAM permissions (like s3:BypassGovernanceRetention) to overwrite or delete an object before retention expires. Compliance mode does not — no user, not even root or account admin, can delete or overwrite a locked object until retention expires. For FINRA Rule 17a-4(f) compliance, you need Compliance mode. Governance mode is useful for internal data governance but doesn't satisfy regulatory WORM requirements.

How long do I need to retain records under FINRA rules?

It depends on the record type. Most customer communications and trade records require 3–6 years. Customer account records need 6 years after account closure. Blotters and ledgers typically require 6 years. Complaint records need at least 4 years (often longer if litigation risk exists). The first 2 years must be "easily accessible" — meaning regulators can get them quickly without special procedures. Years 3+ can be on slower/colder storage but still retrievable on request.

Can RustFS or MinIO handle FINRA-grade Object Lock?

RustFS supports S3 Object Lock in both Governance and Compliance modes as of v1.2+. MinIO also supports Object Lock. Both implement the same S3 API (put-object-retention, get-object-retention, put-object-legal-hold). For production FINRA workloads, verify your specific version supports the full Object Lock API surface, test retention enforcement with an adversarial attempt (try deleting a locked object as root), and document the configuration for your examiner.

What happens during a FINRA exam if my storage doesn't support Object Lock?

FINRA examiners evaluate whether your storage architecture satisfies SEC Rule 17a-4(f)'s immutability requirement. If you're using standard (non-WORM) object storage, you'll need to demonstrate an alternative compliant approach — such as a complete audit trail system that logs every modification/deletion with timestamps and identities (the "audit trail alternative" added in the October 2022 rule amendments). This is harder to defend in an exam than true WORM storage because you're proving a process works rather than demonstrating that the technology makes violations impossible.

Not legal advice. Consult qualified securities counsel. Last updated: July 2026.