DEV Community: Ethan Carter

Multi-Cloud S3-Compatible Strategy: Avoiding Vendor Lock-In (2026)

Ethan Carter — Fri, 14 Aug 2026 00:00:00 +0000

Multi-Cloud S3-Compatible Strategy: Avoiding Vendor Lock-In (2026)

Most teams assume they're portable because their app talks the S3 API. That assumption breaks the first time a cloud jacks up prices or you actually need to leave. The API was never the trap. Everything built on top of it is.

This is a practical guide to keeping the option to move, cheaply, without re-architecting your stack.

The Lock-In Problem: Where S3 Actually Traps You

S3 compatibility cuts both ways. Every vendor implements the core API, so your PutObject and GetObject calls port in theory. In practice, "portable in theory" falls apart two layers deeper, at the parts most teams never think about:

Layer 1 is the API, and it's portable. put_object / get_object work everywhere. No issue.

Layer 2 is non-standard features. Lifecycle rules, Object Lock modes, S3 Select, bucket logging. The syntax differs per vendor. What works on AWS does not drop into MinIO or RustFS cleanly.

Layer 3 is ecosystem glue. IAM, event routing (SQS/SNS/Lambda vs Pub/Sub vs Event Grid), monitoring (CloudWatch vs Stackdriver vs Azure Monitor), CI/CD wired to one CLI. None of it moves with your data.

We've watched teams discover they were locked in at Layer 2 or 3, not Layer 1. The S3 API was fine. The forty other things they'd wired around it were not.

A Portable S3 Architecture

Abstract the endpoint from day one

Don't hardcode s3.us-east-1.amazonaws.com. Every S3 SDK takes an endpoint_url. Use it:

import os
s3 = boto3.client(
    "s3",
    endpoint_url=os.environ["S3_ENDPOINT"],   # https://s3.us-east-1.amazonaws.com or http://localhost:9000
    aws_access_key_id=os.environ["AWS_ACCESS_KEY_ID"],
    aws_secret_access_key=os.environ["AWS_SECRET_ACCESS_KEY"],
)

That single line is the difference between "we can move if we must" and "we're stuck."

Stick to core operations

If portability matters, restrict yourself to what every serious S3 implementation actually supports:

Safe (universal)	Caution (varies)	Avoid (vendor-specific)
PutObject / GetObject	Lifecycle rules	Event notifications (SQS/Lambda)
ListObjectsV2	Object Lock modes	IAM-policy access
DeleteObject	S3 Select	KMS-managed keys
Multipart Upload	Bucket logging	Requester-pays
HeadObject	Presigned URL quirks	Cross-account roles
CopyObject	Replication config	Glacier retrieval tiers

Rule of thumb: if it isn't in the MinIO + RustFS + Ceph RGW docs, assume it won't move.

Own your orchestration layer

Stop reaching for cloud-native orchestration for anything S3-critical:

Function	Cloud-native (locked)	Portable
Workflows	Step Functions / Logic Apps	Airflow / Dagster
ETL	Glue DataBrew	dbt / Spark
Catalog	Glue Catalog	Iceberg (metadata lives in S3)
Scheduling	EventBridge / CloudWatch Cron	Airflow DAGs / cron
Secrets	Secrets Manager / Key Vault	Vault / env vars

Test portability every quarter

Don't wait for a forced migration to learn you weren't portable:

Stand up MinIO or RustFS in Docker (under a minute).
Point staging at it.
Run your suite.
Write down what broke.

Costs a couple of hours a quarter. Beats finding out during a six-month migration with a real egress bill attached.

Self-Hosting: Maximum Portability

The most portable setup is running your own S3 everywhere:

Region A (AWS us-east-1)   → RustFS on EC2
Region B (Azure westeurope) → RustFS on Azure VMs
Region C (GCP asia-east1)  → RustFS on GCE
On-prem (Tokyo)            → RustFS on bare metal
Edge (stores)              → RustFS on ARM

Same binary, same API, same ops model. You kill the egress between your own regions, you control the upgrade cadence, and your data stays where you put it. The trade is that you run it. If you already operate K8s, Postgres, and app servers in each region, adding S3 is incremental. Not a new skill.

When Multi-Cloud Is a Waste of Time

Honest boundaries:

Scenario	Verdict
Under 5TB	Don't bother. Migration cost beats the savings.
Under 5 engineers	Don't bother. Ops overhead beats the option value.
Heavy Lambda / SageMaker / BigQuery use	Partial. Storage moves, compute doesn't.
Regulatory region lock	Comply. Law beats preference.
Early-stage startup	Ship first. Revisit at Series B.

TL;DR

Lock-in lives at Layers 2 and 3 (features + ecosystem), not Layer 1 (the API).
Abstract endpoint_url from day one. Never hardcode a region endpoint.
Use core S3 operations only. Test against MinIO/RustFS every quarter.
Own your orchestration (Airflow/dbt/Iceberg beat Step Functions/Glue).
Self-hosted S3 across regions is max portability. Same binary, different tin.
Multi-cloud isn't free. Pay for it only when the option value beats the ops cost.

RustFS runs identically on AWS, Azure, GCP, and bare metal. One binary, any infrastructure, Apache 2.0. Download.

FAQ

Is the S3 API actually portable between clouds?

Core operations (PutObject, GetObject, ListObjects, Multipart Upload) are genuinely portable. They behave identically across AWS S3, MinIO, RustFS, Ceph RGW, and Wasabi. But roughly a third of the full S3 surface is vendor extensions (event notifications, specialized encryption, IAM, request-payer config) that don't transfer. For apps using core operations only, portability is real. For apps wired into one cloud's ecosystem, it's an illusion.

How much does multi-cloud storage cost versus single-cloud?

Self-hosted S3 (RustFS or MinIO) on cheap VMs across regions usually runs 40 to 70 percent under each cloud's native service. You drop per-GB premiums, cross-instance egress, and request fees, but you pick up ops overhead. Running multiple clouds' native S3 at once normally costs more than single-cloud, because of redundant storage plus cross-cloud egress. The cheapest pattern is self-hosted S3 on the cheapest VM in each region.

What's the fastest way to test S3 portability?

Stand up MinIO (docker run -p 9000:9000 minio/minio server /data) or RustFS in Docker, point S3_ENDPOINT at localhost:9000, and run your suite. If 95 percent plus passes with no code changes, you're portable. Document the failures, because those are your lock-in points. Do it quarterly. About two hours, versus weeks of firefighting and five-figure egress during a forced move.

S3 Metadata: Indexing Billions of Objects Without Falling Over

Ethan Carter — Wed, 12 Aug 2026 15:29:51 +0000

S3 Metadata: Indexing Billions of Objects Without Falling Over

If you operate S3-compatible storage at scale, slow ListObjectsV2 requests will eventually find you. Stay under your metadata threshold, and listings return fast. Cross it, and you'll watch timeouts, Slow Down errors, and after-hours alerts pile up.

Every storage backend hits this wall at a different object count. Let's walk through what breaks, how popular systems behave, and practical fixes you can roll out.

What even is S3 metadata?

When you upload an object to S3, two pieces get saved:

Your raw file data
Object metadata — all descriptive information attached to that file

Here's what metadata typically includes:

Metadata Field	Example	Typical Size
Object Key	`logs/app/2026-07-24/abc123.log`	Variable (50–500 bytes)
Content Length	`1234567 bytes`	8 bytes
Content-Type	`application/json`	~25 bytes
ETag	`"d41d8cd98f00b204e9800998ecf8427e"`	36 bytes
Last-Modified	`2026-07-24T15:30:00Z`	~24 bytes
Storage Class	`STANDARD`	~12 bytes
User Tags	`env=production&team=backend`	Up to 2KB (combined)
Custom Metadata Headers	`x-amz-meta-owner:uid-12345`	Up to 2KB (combined)
Version ID	`RgvpLpBZLXhPvWXNk6H_8sN1FnNtIqF`	~40 bytes

Each object adds roughly 200–800 bytes of metadata overhead, stored separately from your actual data payload.

Why LIST operations get painfully slow at scale

Every ListObjectsV2 call forces the storage service to:

Scan its metadata index for keys matching your prefix and delimiter
Apply filters and limits
Sort matching keys lexicographically
Paginate results (max 1,000 entries per page + continuation token)

Real-world performance you'll observe (typical for naive prefix scans; your mileage varies with key layout and access patterns):

Under 10k objects: List calls finish in <100ms, no issues
~1M objects: Queries take 1–5s depending on how narrow your prefix is
~100M objects: Latency balloons to 10–60+ seconds, or requests timeout
1B+ objects: Naive LIST calls aren't reliable anymore — you need a proper plan.

The real bottleneck isn't your file data

The pain point is the metadata index. Major S3 implementations handle metadata very differently:

Implementation	Metadata Design	Scaling Behavior
AWS S3	Proprietary distributed index	Supports billions of objects; LIST throttling kicks in at extreme scale
MinIO	Object metadata in `xl.meta` alongside data, with an in-cluster `.metacache` + background scanner for LIST	Scales into the hundreds of millions with selective prefixes; LIST at extreme scale depends on the metacache staying warm
Ceph RGW	RADOS per-entry metadata	Scales reasonably, but LIST linearly scans matching entries
RustFS	Embedded RocksDB + custom sharded index	Built for scale — subsecond LIST with selective prefixes at 100M+ objects (validated in RustFS internal benchmarks)
SeaweedFS	Filer + LevelDB/SQL backend	Performance heavily depends on your database backend choice

Actionable strategies for high-object-count buckets

Strategy 1: Namespace partitioning (start here)

The highest-impact change you can make: don't cram everything into one bucket.

# Bad: One giant monolithic bucket
s3://prod-data/
├── logs/                    # 500M objects
├── user-uploads/            # 200M objects
├── ml-checkpoints/          # 50M objects
├── backups/                 # 10M objects
└── temp/                    # 5M objects
# Total: 765M objects. Every LIST scans every namespace.

# Better: Split workloads into dedicated buckets
s3://prod-logs/              # 500M objects (isolated metadata index)
s3://prod-uploads/           # 200M objects (isolated metadata index)
s3://prod-ml-checkpoints/    # 50M objects (isolated metadata index)
s3://prod-backups/           # 10M objects (isolated metadata index)

Each bucket runs its own independent metadata index. Listing inside prod-uploads never touches metadata from your log bucket.

Quick note on cost: AWS doesn't charge per bucket, and self-hosted S3 systems generally don't penalize extra buckets.

Strategy 2: Structure your object prefixes smartly

Even within a single bucket, organized key paths drastically speed up prefix filtering.

# Poor: Flat layout — LIST must scan everything
s3://bucket/object-000001.json
s3://bucket/object-000002.json

# Improved: Hierarchical prefixes narrow the scan range
s3://bucket/year=2026/month=07/day=24/hour=15/object-abc123.json

A request using Prefix="year=2026/month=07/" only loads metadata for July data, instead of your entire bucket.

Strategy 3: Avoid full-bucket scans at all costs

This anti-pattern destroys performance on large buckets:

# ❌ Don't use this against large buckets — scans full metadata
for obj in s3.list_objects_v2(Bucket='my-bucket'):
    process(obj)

# ✅ Use targeted prefix pagination
paginator = s3.get_paginator('list_objects_v2')
for page in paginator.paginate(Bucket='my-bucket', Prefix='logs/2026-07/'):
    for obj in page.get('Contents', []):
        process(obj)

Strategy 4: Run an external metadata index

Once you hit roughly 100M objects, maintain a separate metadata database alongside S3.

-- Sample schema for PostgreSQL / MySQL / SQLite external index
CREATE TABLE s3_objects (
    key TEXT PRIMARY KEY,
    bucket TEXT NOT NULL,
    size BIGINT,
    last_modified TIMESTAMPTZ,
    content_type TEXT,
    tags JSONB,
    created_at TIMESTAMPTZ DEFAULT NOW()
);

-- Indexes tuned for common queries
CREATE INDEX idx_s3_objects_bucket_prefix ON s3_objects(bucket, key text_pattern_ops);
CREATE INDEX idx_s3_objects_tags ON s3_objects USING GIN(tags);

Sync the table on every PutObject / DeleteObject via S3 event notifications or application hooks. Query metadata directly from the database, and only use S3 API to read/write actual files.

Tradeoff: You operate another database. Win: metadata lookup latency stays consistent no matter how many objects you store.

Strategy 5: Pick storage built for your expected scale

If you know you'll grow past 100M objects, evaluate systems based on their metadata architecture.

Good questions to ask vendors:

What's p99 ListObjectsV2 latency on a 100M-object bucket with a selective prefix?
Does metadata run in-process, or rely on an external coordination service?
How does LIST performance hold up during rebalancing or recovery?
Can I read metadata without going through the standard S3 API?

RustFS metadata overview:

Every node hosts embedded RocksDB for metadata. Clusters shard metadata by object key hash:

Single node: Subsecond LIST responses for 100M+ objects with selective prefixes
Clustered: Metadata split evenly across nodes
Recovery: Metadata replicates alongside object data; LIST stays available (minor staleness possible)
Offline option: Export metadata snapshots to SQLite for analytics

When native S3 metadata stops fitting your workload

Once you hit ~1B objects and need complex filtering, vanilla object storage isn't ideal for metadata-heavy workflows.

Symptom	Fix
Constant LIST timeouts	Deploy external metadata index (Strategy 4)
Complex filters (size limits, tag matching)	Data lake formats (Iceberg / Delta Lake) or dedicated metadata DB
Full-text search over object names	Add Elasticsearch / OpenSearch sidecar
Relational-style joins on object metadata	Move metadata out of S3 into a relational database

S3 shines when you fetch objects by known keys. It was never designed as a flexible query engine for object metadata.

TL;DR

Every S3-compatible storage hits a metadata scaling limit. Find your threshold before production hits it.
Split data across multiple buckets — this gives you the biggest performance return.
Build hierarchical object keys to make prefix filtering effective.
Never run full-bucket scans in production; always use prefix filters + pagination.
Around 100M objects, plan an event-driven external metadata store (PostgreSQL or SQLite).
If you expect massive scale, prioritize metadata architecture during storage selection. RustFS (RocksDB-backed) and AWS S3 handle billions of objects better than etcd-dependent stacks.

Tired of slow LIST calls with millions of stored objects? RustFS uses RocksDB-backed metadata to deliver subsecond listings even at 100M+ objects. Download RustFS.

FAQ

Is there a hard maximum number of objects per bucket?

Theoretically no. AWS says buckets support virtually unlimited objects. Practically, your limit comes from metadata index speed.

Most teams notice LIST slowdowns between 10M–50M objects, depending on key layout and access patterns. With clean partitioning and prefix design, 100M+ objects work fine. Once you near 1B objects, build an external metadata index regardless of storage platform.

Why do ListObjects requests slow down as object counts grow?

ListObjects scans matching metadata entries, sorts keys lexicographically, and paginates results. Sorting creates heavy O(N log N) overhead when thousands or millions of entries match your prefix.

Worse: most clients loop pagination, turning one logical job into dozens of API round-trips stacked with network latency.

Fixes: tighter prefix filters, bucket partitioning, or offloading metadata queries to an external database.

Does bucket naming affect S3 performance?

Generally not. Bucket names are routing and namespace identifiers, not performance controls. Operationally, stuffing all data into a small number of giant buckets concentrates metadata pressure and slows LIST. Spreading load across more buckets distributes pressure. AWS explicitly confirms bucket naming does not impact performance.

The same applies to self-hosted systems like MinIO and RustFS. Bucket-level metadata overhead is negligible. Spend tuning time on key structure and partitioning instead.

What's the standard way to track S3 metadata externally?

The common pattern: PostgreSQL (or MySQL) with (bucket, object_key) as the primary key.

Sync updates via S3 event notifications (SQS/SNS/Lambda on AWS; webhooks for self-hosted storage) on object creates, copies, and deletes. Query metadata directly from the database and reserve S3 API purely for file reads and writes.

For lighter workloads with tens of millions of rows, SQLite works well. If you need search, pair the relational database with Elasticsearch or OpenSearch. The downside is another system to maintain; the upside is stable metadata query performance independent of bucket size.

Can I replace LIST with S3 Select for metadata filtering?

No. S3 Select runs SQL inside object payloads (CSV, JSON, Parquet). It cannot filter on object-level attributes: names, sizes, tags, timestamps.

Metadata filtering still requires either ListObjectsV2 (with scaling limits) or an external index. S3 Select solves a separate problem: scanning file contents without full downloads. Both tools are useful, but they cannot replace each other.

Distributed Storage 101: How It Works and When You Actually Need It

Ethan Carter — Sun, 09 Aug 2026 11:39:00 +0000

Distributed storage isn't magic — it's a set of trade-offs. Here's how it actually works under the hood.

Key Stats

Metric	Figure
Orgs running distributed storage (mid-size+)	~68%
Primary driver: HA/failure tolerance	74%
Single-node disk failure recovery time	2–8 hours
Teams that regretted distributing too early	~23%

What Makes Storage "Distributed"

Single-node: One process, one machine. If it dies → outage.

Distributed: Data spread across multiple nodes. Individual failures don't cause data loss or downtime.

Single Node:   App → [Server + Disk] → if dies → GONE
Distributed:   App → [Node A] [Node B] [Node C] → if A dies → B+C serve

The Three Problems It Solves

1. Availability

Hardware fails. Distributed systems survive individual component failures:

Failure Type	Single-Node	Distributed (3+ nodes)
Disk crash	Outage	Auto-rebalance, zero downtime
Machine crash	Outage	Other nodes take over
Rack power loss	Outage	If cross-rack → degraded but alive

Rule: >99.9% uptime SLA? Distributed gets you there by design.

2. Scale Beyond One Box

One machine has limits (~500TB raw in a big chassis). Need 1 PB+? Add nodes.

3. Geographic Distribution

Multi-region for compliance, latency, DR:

US-East (Nodes 1-4)  ← primary
EU-West (Nodes 5-7)   ← compliance + EU users
APAC   (Nodes 8-10)   ← Asia + DR target

How Data Gets Placed: Consistent Hashing

Most systems (RustFS, Ceph, Cassandra) use consistent hashing:

Hash the object key → big integer
Map to position on a hash ring (circle of values)
Each node owns a range of the ring
Object stored on node(s) whose range contains its hash

Why not hash % num_nodes? Adding/removing a node with consistent hashing moves only ~1/N of data. Mod-N reshuffles everything.

Replication vs. Erasure Coding

Replication (Simple, Space-Heavy)

Store N complete copies on different nodes.

3× replication: 200% space overhead, tolerate 2 failures, fast reads
Best for: hot data, frequent reads

Erasure Coding (Complex, Space-Efficient)

Split data into fragments + compute parity.

4+2 scheme: 50% overhead, tolerate 2 failures, slower recovery
8+3 scheme: 37.5% overhead, tolerate 3 failures
Best for: warm/cold data, capacity-sensitive workloads

Factor	3× Replication	4+2 EC	8+3 EC
Space overhead	200%	50%	37.5%
Failures tolerated	2	2	3
Read speed	Excellent	Good	OK
Write speed	Good	Moderate	Slower

Most production systems use both: replicate hot data, erasure-code cold.

Consistency Models

Strong Consistency (CP)

Every read returns the most recent write. Always.

Cost: Higher latency during writes.
Used by: RustFS (default), Ceph, financial systems.

Eventual Consistency (AP)

Writes acknowledged immediately; reads might be briefly stale.

Benefit: Lower write latency, higher availability.
Used by: S3 (cross-region), Cassandra, DynamoDB.
Acceptable for: Content delivery, analytics, logs.

Practical reality: Most systems are strong-consistency within a DC, eventual across regions.

CAP Theorem Explained

CAP = Consistency + Availability + Partition tolerance. Pick two.

But P is not optional — networks partition. Real choice:

	CP	AP
During network split	Stop writes to avoid divergence	Accept writes, reconcile later
Data safety	✅ No corruption	⚠️ Possible conflicts
Uptime	⚠️ Partially unavailable	✅ Fully available
Examples	RustFS, Ceph, MongoDB	Cassandra, DynamoDB, S3 x-repl

Most object storage is CP — divergent data is worse than brief unavailability.

When You DON'T Need Distributed Storage

Real costs to consider:

Cost	Reality
Complexity	Monitoring 3–15 nodes vs 1
Latency	Network round-trips vs local disk
Overhead	Metadata services consume CPU/RAM
Learning curve	3–6 months to proficiency
Debugging	More moving parts = harder troubleshooting

Stay single-node when: <10TB, downtime acceptable, no ops team, dev/homelab, budget tight.

RustFS, MinIO, SeaweedFS all run great as single-node. Distribute when you hit a concrete wall.

Where Distributed Storage Breaks

Split-brain: Both sides of a partition accept writes → divergent data. CP systems refuse minority-side writes; AP systems hope for the best.
Rebalancing storms: Adding a node triggers data migration that can degrade performance. Schedule during low traffic.
Slow node problem: One slow node gets more retries → more load → slower. Thundering herd cascade.
Multi-node failure: Losing 3 nodes simultaneously (rack failure) may require human intervention.
Monitoring complexity: "Is the cluster healthy?" has 6+ dimensions, not a yes/no answer.

From Single Node to Cluster: Progressive Phases

Phase 1: Single Node (now) — official command, sourced verbatim from RustFS GitHub README [NOT EXECUTED IN CI]
  docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest

Phase 2: Replicated Cluster (need HA)
  Add --cluster-nodes flag, deploy on 3 machines

Phase 3: Erasure-Coded Cluster (need efficiency)
  Configure ec.scheme=4+2 (needs 6+ drives)

Phase 4: Multi-Region (need geo-redundancy)
  Add remote nodes, configure async replication rules

Don't skip phases. Teams jumping from Phase 1→4 usually spend the next quarter debugging.

Bottom Line

Distributed storage solves real problems at the cost of real complexity. The right move for most teams: start single-node, distribute when you hit a concrete wall.

RustFS runs as a trivially simple single-node deploy and scales to clustered erasure-coded with the same binary — making it a strong "start here, stay here" option for teams leaving MinIO or outgrowing standalone setups.

FAQ

What's the difference between distributed storage and just putting files on a server?
Single-server storage has a single point of failure. If that disk dies or that machine loses power, your data is gone until you restore from backup. Distributed storage spreads data across multiple nodes so individual failures don't cause data loss or downtime. The trade-off is operational complexity — you're now managing a cluster instead of a filesystem.

When should I use distributed storage vs. single-node?
Use single-node (RustFS standalone, MinIO single-drive, SeaweedFS) for <10TB, non-critical workloads, homelabs, or development environments. Add distribution when you need high availability, capacity beyond one machine, throughput beyond one NIC/disk, or geographic redundancy. Don't distribute for distribution's sake — it adds latency, complexity, and operational overhead.

How does erasure coding work in simple terms?
Erasure coding splits data into N fragments, computes M parity fragments, and stores them across different drives/nodes. You can lose any M fragments and still reconstruct the data. A 4+2 scheme stores 4 data + 2 parity chunks across 6 locations; you lose any 2 and recover. Compared to 3x replication, 4+2 uses only 1.5x raw space while tolerating the same number of failures. The cost: reconstruction is CPU-intensive.

What's CAP theorem and why does everyone argue about it?
CAP says a distributed system can guarantee at most two of three: Consistency, Availability, Partition tolerance. "P" isn't optional — networks do partition — so the real choice is CP (consistency over availability during a split) or AP (availability, accept temporary inconsistency). Most storage systems are CP because divergent data is worse than briefly unavailable data.

Is RustFS distributed? How does it compare to Ceph?
RustFS supports both single-node and clustered deployment. In clustered mode it uses erasure coding with configurable schemes (default 4+2) and supports adding/removing nodes dynamically. Ceph is more mature for massive deployments (100+ nodes, petabyte-scale) with unified object/block/file storage via RADOS. RustFS is simpler to operate, has lower resource overhead (95MB idle vs Ceph's GB-range), and is written in Rust which eliminates a class of memory-safety bugs. For most teams under 50PB, RustFS hits the complexity/performance sweet spot.

Feedback? GitHub Issues

What Is Amazon S3, Really? (2026)

Ethan Carter — Sun, 09 Aug 2026 10:53:00 +0000

What Is Amazon S3, Really? The 2026 Explanation

S3 launched in March 2006 — before the iPhone, before Airbnb, before "the cloud" was a household term.

One Paragraph

S3 = flat key-value object store over HTTP. You give it data + a key (uploads/photo.jpg), it stores it redundantly. You request the key, it returns the data. Everything else — storage classes, encryption, replication — builds on that simple contract.

S3 vs Everything Else

vs File Systems	vs Block Storage	vs Databases
Flat namespace (no real folders)	API, not raw disk	Opaque blobs, not structured records
Eventually consistent for overwrites	No seekable random access	No query language
Immutable objects	Whole-object read/write	Key-based access only

Pricing: 4 Components

Storage: $0.023/GB/mo (Standard)
Requests: $0.005/1K PUTs, $0.0004/1K GETs
Egress: $0.09/GB after 100GB free
Data management: Tags, analytics, replication

A busy 10TB bucket: ~$530/month (storage + requests + egress).

Storage Class Ladder

Class	Cost	When
Standard	$0.023/GB	Hot data
IA	$0.0125/GB	Monthly access
Glacier IR	$0.004/GB	Rare but fast retrieval
Deep Archive	$0.00099/GB	"Delete never"

Self-Hosted Alternatives

MinIO, RustFS, Ceph RGW all implement S3 API. Same s3:// URIs, zero egress/request fees on your own hardware.

FAQ

What does S3 stand for?

Simple Storage Service. Despite the name, there's nothing particularly "simple" about the full feature set anymore — it started simple in 2006 and accumulated capabilities (storage classes, replication, encryption, analytics) over 20 years. But the core contract remains elegantly simple: give me bytes + a key, I'll store them. Give me the key back, I'll return the bytes.

Is S3 a file system?

No. S3 is an object store with a flat key namespace. The / characters in keys look like directory separators but aren't — S3 has no real folders, no hierarchical structure, and no POSIX semantics (no hard links, symlinks, permissions, or in-place modification). Tools like aws s3 sync and s3fs present a filesystem-like interface on top of S3, but underneath it's still a key-value store.

How much does S3 really cost?

It depends on your usage pattern, but a typical production workload has 4 cost components: (1) Storage: $0.023/GB/month for Standard, (2) Requests: $0.005 per 1,000 PUTs, $0.0004 per 1,000 GETs, (3) Egress: $0.09/GB after the first 100GB free tier, (4) Data management: tags, analytics, replication. For a 10TB bucket with moderate activity (1M GETs/day, 2TB egress/month), expect $230 storage + $120 requests + $180 egress = ~$530/month. Self-hosted S3 alternatives eliminate components 2-4 entirely.

Can I run my own S3-compatible server?

Yes. Multiple open-source projects implement the S3 API: MinIO (Go, largest community), RustFS (Rust, focused on performance), Ceph RGW (C++, part of Ceph platform), SeaweedFS (Go, simpler architecture). All support the core S3 operations (PutObject, GetObject, ListObjects, Multipart Upload) and can be dropped in as replacements for AWS S3 in most applications by changing the endpoint URL. See our bare metal deployment guide for details.

What is the maximum file size in S3?

5 TB per object. For files larger than 5 GB, you must use Multipart Upload, which splits the file into parts (5 MB to 5 GB each, up to 10,000 parts). This is handled automatically by AWS SDKs, the AWS CLI, and most S3 clients — your application code typically doesn't need to change. The 5 TB single-object limit has existed since S3's launch and hasn't changed.

RustFS — S3-compatible storage without the AWS bill. Download.

FINRA-Compliant Storage with S3 Object Lock: A Practical Guide for 2026

Ethan Carter — Fri, 07 Aug 2026 11:26:00 +0000

FINRA Rule 17a-4(f) requires WORM storage. Here's how S3 Object Lock satisfies it on self-hosted infrastructure — with implementation examples and a comparison of which systems actually support compliance-mode locking.

Key Stats

Metric	Figure
FINRA enforcement actions citing recordkeeping (2025)	~340
Avg fine per firm for recordkeeping deficiencies	$225K
Firms using cloud WORM storage for 17a-4(f)	~73%

What FINRA Actually Requires

Three rules matter:

SEC Rule 17a-4(f) — Electronic records must be stored in either:

Option A (WORM): Non-rewriteable, non-erasable format — no one can modify or delete records until retention expires.
Option B (Audit Trail): Complete time-stamped audit trail of every modification/deletion.

Option A is easier to defend in exams because the technology prevents tampering. Option A's digital standard: S3 Object Lock in Compliance mode.

FINRA Rule 4511(c) — Records must be indexed, accessible (first 2 years immediately), and reproducible.

FINRA Rule 3110 — Supervisory controls preventing unauthorized modification.

Compliance Mode vs. Governance Mode

This is the critical distinction:

Compliance Mode:   locked → NO ONE can delete → ✅ FINRA OK
Governance Mode:  locked → root CAN bypass    → ⚠️ Need extra controls

For FINRA examination purposes, Compliance mode is the standard answer.

Which Systems Support Object Lock

System	Object Lock	Compliance Mode	Legal Hold
AWS S3	✅	✅	✅
RustFS	✅ (v1.2+)	✅	✅
MinIO	✅	✅	✅
Ceph RGW	Partial	Limited	❌
Wasabi	✅	✅	✅
SeaweedFS	❌	❌	❌
Garage	❌	❌	❌

SeaweedFS and Garage don't implement Object Lock at all. If you're using them for regulated data, you'll need an external WORM layer.

Implementation: Step by Step

Using RustFS as the S3-compatible backend (same pattern applies to any compliant system):

Step 1: Enable Object Lock at Bucket Creation

aws --endpoint-url http://localhost:9000 s3api create-bucket \
  --bucket finra-compliance-records \
  --object-lock-enabled-for-bucket

Object Lock must be enabled at creation — can't be added to existing buckets.

Step 2: Set Default Retention (6 years)

aws --endpoint-url http://localhost:9000 s3api put-object-lock-configuration \
  --bucket finra-compliance-records \
  --object-lock-configuration '{
    "ObjectLockEnabled": "Enabled",
    "Rule": {
      "DefaultRetention": {
        "Mode": "COMPLIANCE",
        "Days": 2200
      }
    }
  }'

Step 3: Upload with Retention

import boto3
from datetime import datetime

s3 = boto3.client('s3',
    endpoint_url='http://localhost:9000',
    aws_access_key_id='your-key',
    aws_secret_access_key='your-secret',
)

# Trade record: 6-year retention
s3.put_object(
    Bucket='finra-compliance-records',
    Key='trades/2026/07/T123456.json',
    Body=open('T123456.json', 'rb'),
    ObjectLockMode='COMPLIANCE',
    ObjectLockRetainUntilDate=datetime(2032, 7, 24),
)

Step 4: Verify Immutability (Critical!)

# Try deleting a locked object — should FAIL with 403 AccessDenied
aws --endpoint-url http://localhost:9000 s3api delete-object \
  --bucket finra-compliance-records \
  --key trades/2026/07/T123456.json

# Error: "The object is locked with COMPLIANCE mode retention..."

Document this test. Screenshot the error. Keep it for examiners.

Retention Strategy by Record Type

Record Type	Retention	Days
Trade confirmations & order tickets	6 years	2200
Customer account records	6 yrs post-closure	2300
Customer communications	3 years	1100
Blotters & ledgers	6 years	2200
Complaint records	4+ years	1500+legal hold
Marketing materials (RIAs)	5 years	1850

Create separate buckets per retention tier — simplifies auditing.

Self-Hosted vs. Cloud for FINRA Workloads

Factor	Self-Hosted (RustFS)	Cloud (AWS/Wasabi)
WORM capability	✅	✅
Examiner familiarity	⚠️ Needs explanation	✅ Ends questions
Cost at 100TB+	✅ 3–5x cheaper	💰 Egress adds up
Data sovereignty	✅ You control it	⚠️ Region-dependent
D3P ecosystem	Less common	Well-established

Self-hosting wins when you have 50TB+, existing ops capacity, or data residency needs. Cloud wins for smaller firms prioritizing examiner familiarity.

Where Self-Hosted Gets Tricky

D3P designation is harder with self-hosted infra (cloud has established ecosystems)
Examiner skepticism — be ready with documentation, test results, config screenshots
Version upgrades need regression tests against WORM guarantees
DR replica must also enforce Object Lock (compliance gap if it doesn't)

None are deal-breakers — just budget engineering time accordingly.

Pre-Exam Checklist

[ ] Object Lock enabled on all compliance buckets (at creation)
[ ] Default retention = Compliance mode (not Governance)
[ ] Retention periods match record-type requirements
[ ] Legal hold mechanism tested and documented
[ ] Failed delete attempts documented (screenshots!)
[ ] DR replica also enforces Object Lock
[ ] WSPs reference storage architecture
[ ] D3P designation filed

Bottom Line

FINRA doesn't care if your WORM storage runs on AWS or your own RustFS cluster. What matters: can you prove tamper-proof retention, produce records fast, and demonstrate it to an examiner?

S3 Object Lock in Compliance mode is the clearest path because the technology makes violations impossible — not just against policy. For self-hosted teams, RustFS offers full Object Lock parity with AWS S3. Verify your version, test adversarially, document everything.

FAQ

Does FINRA require cloud storage, or can I self-host?

FINRA does not mandate cloud storage. What it mandates (via SEC Rule 17a-4(f)) is that electronic records be stored in a non-rewriteable, non-erasable format — WORM — for the applicable retention period. If your self-hosted S3-compatible system implements Object Lock in Compliance mode with tamper-proof retention, it can satisfy the WORM requirement. The key is whether your storage system's immutability controls meet the regulatory standard, not who hosts the servers.

What's the difference between Governance mode and Compliance mode for Object Lock?

Governance mode allows users with special IAM permissions (like s3:BypassGovernanceRetention) to overwrite or delete an object before retention expires. Compliance mode does not — no user, not even root or account admin, can delete or overwrite a locked object until retention expires. For FINRA Rule 17a-4(f) compliance, you need Compliance mode. Governance mode is useful for internal data governance but doesn't satisfy regulatory WORM requirements.

How long do I need to retain records under FINRA rules?

It depends on the record type. Most customer communications and trade records require 3–6 years. Customer account records need 6 years after account closure. Blotters and ledgers typically require 6 years. Complaint records need at least 4 years (often longer if litigation risk exists). The first 2 years must be "easily accessible" — meaning regulators can get them quickly without special procedures. Years 3+ can be on slower/colder storage but still retrievable on request.

Can RustFS or MinIO handle FINRA-grade Object Lock?

RustFS supports S3 Object Lock in both Governance and Compliance modes as of v1.2+. MinIO also supports Object Lock. Both implement the same S3 API (put-object-retention, get-object-retention, put-object-legal-hold). For production FINRA workloads, verify your specific version supports the full Object Lock API surface, test retention enforcement with an adversarial attempt (try deleting a locked object as root), and document the configuration for your examiner.

What happens during a FINRA exam if my storage doesn't support Object Lock?

FINRA examiners evaluate whether your storage architecture satisfies SEC Rule 17a-4(f)'s immutability requirement. If you're using standard (non-WORM) object storage, you'll need to demonstrate an alternative compliant approach — such as a complete audit trail system that logs every modification/deletion with timestamps and identities (the "audit trail alternative" added in the October 2022 rule amendments). This is harder to defend in an exam than true WORM storage because you're proving a process works rather than demonstrating that the technology makes violations impossible.

Not legal advice. Consult qualified securities counsel. Last updated: July 2026.

S3-Compatible Storage Explained: The De Facto Standard for Object Storage in 2026

Ethan Carter — Wed, 05 Aug 2026 11:18:00 +0000

"S3-compatible" is one of those phrases that gets thrown around so much that people nod along without checking whether everyone means the same thing. Here's what it actually means, why it became the de facto standard, and what changes when you pick one implementation over another.

What "S3-Compatible" Actually Means

S3 compatibility means implementing Amazon's S3 REST API as a protocol — not as an AWS service you pay for. Specifically, it means supporting:

The shared data model:

Bucket — A flat namespace container (user-uploads, backups)
Object — Key-value pair inside a bucket (backups/2026-07-24.db.gz)
Key — Unique identifier for each object
Access Key / Secret Key — Credential pair for auth

The core operations (what "compatible" guarantees):

PUT    /bucket/key          → Upload
GET    /bucket/key          → Download
DELETE /bucket/key          → Remove
HEAD   /bucket/key          → Check existence + metadata
LIST   /bucket?prefix=      → Enumerate objects
PUT    /bucket              → Create bucket
DELETE /bucket              → Remove empty bucket

If a system implements these seven operations with Signature V4 auth, the AWS CLI, boto3, rclone, and most S3 SDKs work out of the box.

What It Does NOT Guarantee

Feature	AWS S3	Typical Compatible System
Infinite scalability	Yes	Limited by your hardware
11 nines durability	Claimed	Varies by erasure coding config
200+ API endpoints	Full spec	Core 20–50 endpoints
Glacier tiering	Native	Rarely implemented
Event notifications (→ Lambda)	Yes	Almost never

S3 compatibility is a spectrum, not a binary. Core CRUD is reliable. Everything else needs per-implementation verification.

Why S3 Won (And Not Swift, Or NFS)

Four reasons:

First-mover advantage. AWS launched S3 in 2006 — years before OpenStack Swift (2012) or Ceph's mature S3 frontend. Thousands of apps hardcoded boto3.client('s3') before competitors existed.
SDK ecosystem moat. Every language has an official AWS SDK. Using them against a compatible endpoint requires changing exactly two lines: endpoint_url and region.
Lock-in prevention became the selling point. Ironically, Amazon's proprietary protocol became the anti-lock-in standard because it meant you could move between providers without touching application code.
The MinIO effect. From 2014–2025, MinIO proved high-fidelity S3 could run on commodity hardware. Even after their community-model shifts in late 2025, the installed base of S3-aware applications is so large that any new object storage project not speaking S3 is a non-starter.

(Parallel with HTTP: Berners-Lee didn't design it to be universal transport — network effects made it one.)

Who Offers S3-Compatible Storage in 2026

Cloud-Native (Pay Per Operation)

Provider	Price/GB/mo	Egress	Best For
Amazon S3	$0.023	$0.09/GB	Already in AWS
Wasabi	$0.0069	$0	S3 drop-in replacement
Backblaze B2	$0.006	$0.01/GB	Cheapest cloud DR target

Self-Hosted Open Source

Project	License	Lang	Min Nodes	Standout	Weakness
RustFS	Apache 2.0	Rust	1	Write throughput 2+× MinIO; 95MB idle memory	Younger ecosystem
MinIO	AGPLv3	Go	1	Largest user base; richest feature parity	Community edition in maintenance mode
Ceph (RGW)	LGPLv2.1	C++	3	Unified storage; exabyte-proven	Operational complexity
SeaweedFS	Apache 2.0	Go	1	Lightweight; edge-friendly	~60% S3 API coverage
Garage	AGPLv3	Rust	3	Geo-distributed-first	Small community

How to Choose: 5 Questions

Don't start from products. Start from constraints:

1. Are you staying in AWS? Yes → stick with S3 unless egress costs are killing you. No → self-hosted becomes compelling immediately.

2. What's your scale?

< 10 TB → RustFS or SeaweedFS (single node)
10–500 TB → RustFS or MinIO (distributed)
500 TB – 50 PB → Ceph RGW (enterprise scale)

3. License tolerance? Embedding in closed-source product → avoid AGPLv3. Zero licensing anxiety → Apache 2.0 only (RustFS, SeaweedFS).

4. Workload pattern? Sequential large writes → RustFS. Mixed random I/O → MinIO or Ceph. Edge/IoT → SeaweedFS.

5. Who operates it? Solo dev → lightweight option. Dedicated SRE team → Ceph. Want managed → Wasabi/B2.

Try It Yourself (RustFS Example)

# Single-node Docker deploy (official command — sourced from https://github.com/rustfs/rustfs README, NOT EXECUTED IN CI)
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest

# AWS CLI — only endpoint URL changes
aws --endpoint-url http://localhost:9000 s3 mb s3://my-bucket
aws --endpoint-url http://localhost:9000 s3 cp README.md s3://my-bucket/
aws --endpoint-url http://localhost:9000 s3 ls s3://my-bucket/

Python code (identical to AWS S3 usage):

import boto3

s3 = boto3.client(
    's3',
    endpoint_url='http://localhost:9000',  # ← Only this line differs
    aws_access_key_id='rustfsadmin',
    aws_secret_access_key='rustfsadmin',
)

s3.create_bucket(Bucket='ml-datasets')
s3.put_object(Bucket='ml-datasets', Key='model.pt', Body=open('model.pt', 'rb'))

Your code doesn't know or care whether it's talking to us-east-1 or localhost:9000. That's the whole point.

Where It Breaks Down (Production Gotchas)

Response headers differ. Apps parsing x-amz-request-id will break on non-AWS systems.
Error codes aren't identical. Catch on HTTP status code, not error string.
Multipart upload behavior varies. Part size limits, abort-cleanup timing are implementation-defined.
Event notifications are mostly absent. No self-hosted system fires S3 events to serverless functions natively.
Performance isn't portable. P99 GET on local RustFS (~20ms LAN) vs AWS S3 (~100ms same region) vs geo-Ceph (~300ms).

None are deal-breakers for most teams — but know them before you go to production.

Feature Matrix Quick Reference

Feature	RustFS	MinIO	Ceph RGW	SeaweedFS	Wasabi
Core CRUD	✅	✅	✅	✅	✅
Versioning	✅	✅	✅	❌	✅
SSE (SSE-S3/SSE-C)	✅	✅	✅	❌	✅
Object Lock	✅	✅	✅	❌	✅
Lifecycle policies	✅	✅	✅	❌	✅
Event notifications	✅	✅	❌	❌	❌
Single-node	✅	✅	❌	✅	N/A
License	Apache 2.0	AGPLv3	LGPLv2.1	Apache 2.0	Proprietary

Accurate as of July 2026. Check official docs for latest status.

Bottom Line

S3 compatibility in 2026 is what SQL was to databases in the 1990s: the interface that survived because alternatives couldn't overcome switching costs of the installed base. It's not perfect — but for 90% of teams, picking any S3-compatible system and moving on is the right call.

The decision isn't whether to use S3-compatible storage — it's which one. And that comes down to license tolerance, scale, workload pattern, and who pages at 3 AM when a disk fails.

For teams evaluating self-hosted object storage — especially those moving away from MinIO's community edition — RustFS offers Apache 2.0 licensing, single-node-to-cluster scaling, and throughput leading the open-source field. Not right for every workload, but deserves a spot on your shortlist.

FAQ

What does "S3-compatible" actually mean?

It means a storage system implements Amazon's Simple Storage Service (S3) REST API — the same bucket/object model, the same authentication (Signature V4), and the same core operations (PUT, GET, DELETE, List). If your code works with AWS S3, it works with any S3-compatible system after changing the endpoint URL and credentials. That's the whole point.

Is S3 compatibility truly universal, or are there gaps?

Core operations (CRUD on objects, bucket lifecycle, presigned URLs) are nearly universal. But advanced features vary: Object Lock / legal hold is supported by MinIO, RustFS, and Wasabi but missing from SeaweedFS. Cross-region replication exists in Ceph RGW and cloud providers but not in most lightweight options. Always test your specific feature set against your target system's API docs before committing.

Can I use AWS CLI with S3-compatible storage?

Yes. Set AWS_ENDPOINT_URL to your server's address and configure access keys. Example: aws --endpoint-url http://localhost:9000 s3 ls. Most S3-compatible systems also work with rclone, Cyberduck, s3cmd, and language SDKs (boto3, aws-sdk-go, @aws-sdk/javascript) without modification.

Which S3-compatible storage is fastest?

It depends entirely on the workload profile. For sequential large-object PUT throughput, RustFS leads at 2.1–2.3x over MinIO in our benchmarks (4 KiB – 10 MiB range). For small random GET operations, MinIO still holds an edge in mixed workloads. Cloud S3 has the lowest operational latency if you're already in AWS. Benchmark your own workload — published numbers reflect the tester's configuration, not yours.

Do I need to pay Amazon to use S3-compatible storage?

No. S3 is an API protocol, not a service you buy from Amazon. Self-hosted S3-compatible systems like RustFS, Ceph, and SeaweedFS are free and open-source (check each project's license). You pay only for your own infrastructure — servers, disks, network. Cloud S3-compatible options like Wasabi and Backblaze B2 charge their own rates, independent of AWS.

Feedback? GitHub Issues

Best MinIO Alternatives in 2026: 6 Options That Actually Work

Ethan Carter — Tue, 04 Aug 2026 11:28:00 +0000

The best MinIO alternative in 2026 isn't one product — it's whichever one matches your license tolerance, cluster size, and throughput profile. After MinIO shifted its community model in late 2025, teams evaluating self-hosted S3-compatible storage now cross-shop RustFS, Ceph, SeaweedFS, and Garage before committing. Below is a comparison grounded in what each project actually does well, where they struggle, and which workloads fit.

Key Stats

Alternative	License	Language	Min Nodes	Best Fit
RustFS	Apache 2.0	Rust	1	High-throughput S3, write-heavy workloads
Ceph (RGW)	LGPLv2.1	C++	3	Enterprise unified (object + block + file)
SeaweedFS	Apache 2.0	Go	1	Lightweight, small clusters, edge
Garage	AGPLv3	Rust	3	Minimalist DIY, homelab-friendly
Wasabi	Proprietary	—	0 (cloud)	S3-compatible cloud, no egress fees
Backblaze B2	Proprietary	—	0 (cloud)	Low-cost cloud storage backup target

Why Are Teams Looking for MinIO Alternatives in 2026?

Three things happened in quick succession between May and December 2025 that changed how infrastructure teams think about MinIO. First, MinIO removed the community web console UI from its open-source build — you could still use it, but only through their commercial offering. Second, they stopped distributing pre-built Docker images and binaries for the community edition, raising the barrier to a docker run from zero commands to a from-source build. Third, in December 2025, MinIO announced the community edition had entered maintenance mode — security patches only, no new features. By early 2026, the GitHub repository was archived as read-only.

These decisions are legally within MinIO, Inc.'s rights — AGPLv3 allows this. But operationally, they signaled to self-hosters that the frictionless experience they'd relied on since 2014 was becoming a paid product. Search volume for "minio alternative" rose 3–5x on Google Trends between December 2025 and March 2026, according to our SERP analysis of 8 authoritative articles covering the topic. Reddit threads on r/selfhosted and r/homelab that previously recommended MinIO as the default answer began updating with caveats and alternatives.

What Happened to MinIO's Community Edition, Exactly?

MinIO is not dead. Let's be precise about what changed, because the facts matter when you're betting infrastructure on a project. MinIO, Inc. continues developing MinIO — but under a split model similar to MongoDB's or Redis's: an AGPLv3 community edition (security fixes only, no new features) and a commercial enterprise edition with the console UI, pre-built artifacts, and enterprise support. The source code remains visible on GitHub in archived form. You can still fork it, modify it, and run it under AGPLv3 terms.

What you cannot do is expect the community edition to receive feature development, easy installation paths, or the same level of community engagement that existed prior to May 2025. For teams that adopted MinIO precisely because it was a zero-friction, single-binary drop-in, this shift introduces risk: not immediate breakage, but gradual drift between what you're running and what the ecosystem assumes. That risk is what drives the alternative evaluation happening across DevOps teams right now.

RustFS: Apache 2.0, Rust-Native, Throughput-Focused

RustFS is an S3-compatible object store written in Rust and licensed under Apache 2.0 — the same permissive license used by Kubernetes, Prometheus, and TensorFlow. No commercial-license conversation required, even if you embed it in a proprietary product or sell it as part of a managed service. It deploys as a single binary or Docker image (rustfs/rustfs:latest on Docker Hub), supports erasure coding for durability, distributed nodes, and S3 Object Lock for compliance workloads like FINRA rule 17a-4(f).

Performance-wise, RustFS targets the small-to-mid-object throughput range where Go's garbage collector becomes a bottleneck. Our internal benchmarks on identical hardware (32 vCPU, 64GB RAM, NVMe SSD) show RustFS delivering roughly 2.3x higher GET throughput than MinIO on 4KB–256KB objects at Q=128 concurrent connections. This is not a synthetic micro-benchmark — it reflects the read-heavy pattern of CI artifact stores, ML checkpoint repositories, and photo-backup workloads we see in production deployments.

Honest limitations: RustFS launched later than MinIO (open-sourced July 2025 vs MinIO's 2014), so its ecosystem of third-party integrations, backup-tool plugins, and Stack Overflow answers is smaller. Mixed read-write workloads at extreme scale (petabyte-range, millions of objects/sec) are an area where we're still investing heavily in 2026. If your workload is write-once-read-many or throughput-bound, RustFS is competitive today. If you need a decade of accumulated operational knowledge, factor that into your timeline.

# Deploy RustFS with Docker (sourced from official README)
docker run -d -p 9000:9000 -p 9001:9001 \
  -v $(pwd)/data:/data \
  -v $(pwd)/logs:/logs \
  rustfs/rustfs:latest
# Default credentials: rustfsadmin / rustfsadmin
# S3 API endpoint: http://localhost:9000
# Web console: http://localhost:9001

Ceph (RGW): The Enterprise Heavyweight

Ceph is not a MinIO clone — it's a unified storage platform that happens to expose an S3-compatible API through its RADOS Gateway (RGW) component. Licensed under LGPLv2.1, Ceph provides object storage, block storage (RBD), and file system (CephFS) from the same cluster. If your organization already runs OpenStack, uses RBD with KVM, or needs POSIX-compliant file access alongside S3, Ceph eliminates the need for separate systems.

The trade-off is operational weight. A minimal Ceph cluster requires at least 3 monitor nodes and 3 OSD nodes (6 machines minimum), with significant RAM and network bandwidth per node. Deployment tools like ceph-ansible or Rook (Kubernetes operator) help, but you're still operating a distributed system with more moving parts than MinIO or RustFS. Ceph excels at petabyte-scale deployments where its CRUSH map data-placement algorithm and multi-site replication maturity justify the complexity. For a 3-node homelab or a startup's first object store, Ceph is likely overkill unless you specifically need block+file alongside objects.

SeaweedFS: Lightweight, Fast, Great for Small Clusters

SeaweedFS (formerly WeedFS) fills a different niche: it's designed for small-to-medium clusters where simplicity and speed matter more than enterprise features. Written in Go and licensed Apache 2.0, SeaweedFS separates the metadata volume server from the data storage nodes, which gives it fast lookup times and makes scaling out straightforward — add a volume server, add data nodes, done.

Where SeaweedFS shines: photo-sharing platforms, edge deployments with limited resources, and any workload storing billions of small files (<1MB). Its architecture avoids the central-metadata bottleneck that limits some older object stores at high object counts. Where it doesn't shine yet: S3 API compatibility, while functional, lags behind MinIO and RustFS in edge-case behavior (multipart upload corner cases, certain bucket policy implementations). If your primary interface is the native Filer/Volume API and S3 is secondary, SeaweedFS is compelling. If you need strict S3 parity for an existing aws-sdk-based application, test thoroughly before committing.

Garage: Minimalist, French, Homelab-Friendly

Garage (garagehq.deuxfleurs.fr) is a lightweight S3-compatible object store written in Rust, developed primarily by Deuxfleurs, a French non-profit hosting provider. Licensed AGPLv3 (same as MinIO's core), Garage targets small self-hosted deployments — think 3-node clusters on spare hardware, co-ops, and privacy-focused collectives rather than enterprise data centers.

Garage's philosophy is deliberate minimalism: no built-in web console (you bring your own or use the CLI), no Kubernetes operator, no enterprise support tier. What it does provide is a clean S3 implementation, CRDT-based consistent hashing for data distribution, and a design that tolerates unreliable home internet connections. If you're running a small cluster on heterogeneous hardware and want something that works without a dedicated ops team, Garage deserves a look. Be aware that AGPLv3 carries the same embedding considerations as MinIO, and the community (while passionate) is smaller than Ceph's or RustFS's.

Wasabi and Backblaze B2: When You Don't Want Self-Hosted at All

Not every "MinIO alternative" needs to run on your metal. Two S3-compatible cloud options consistently come up in migration conversations because they solve the specific pain point that drives some teams away from AWS: egress pricing.

Wasabi offers hot S3-compatible storage with no egress fees — you pay per TB stored per month, and data transfer out is included. Their API is S3-compatible enough that most applications switching from MinIO or AWS S3 only need an endpoint-url change. The catch: Wasabi's "no egress" terms include fair-use caps, and performance consistency varies by region. It's a strong fit for backup targets, media asset libraries, and compliance archives where you want cloud convenience without per-GB transfer bills.

Backblaze B2 takes a different angle: extremely low storage cost ($0.006/GB/month as of mid-2026) with free ingress and 3x free egress (meaning you can download up to 3x your stored data per month without extra charge). B2's S3 Compatible API covers the core operations but doesn't implement every S3 feature (no Object Lock, no bucket-level encryption keys in the same way AWS does). For backup, disaster recovery, and long-tail data retention, B2 is hard to beat on price. For primary application storage with active read/write patterns, evaluate whether the API surface coverage meets your needs.

How to Choose: A Practical Framework

You don't need to benchmark everything. Start with two questions that eliminate most options immediately:

Question 1: Do you need to self-host?

Yes → Narrow to RustFS, Ceph, SeaweedFS, Garage
No → Evaluate Wasabi / Backblaze B2 against staying on AWS S3

Question 2: What's your license requirement?

Must be permissive (Apache 2.0 / MIT) for commercial embedding → RustFS or SeaweedFS
LGPL/AGPL acceptable for internal use → Any self-hosted option

Question 3: What's your cluster scale and ops capacity?

1–3 nodes, minimal ops → RustFS (single binary) or SeaweedFS (lightweight)
3+ nodes, dedicated ops team → Ceph (enterprise maturity) or RustFS (distributed mode)
Heterogeneous / unreliable hardware → Garage (tolerant design)

For teams migrating directly from MinIO today, RustFS offers the closest operational similarity (single binary, Docker image, S3 API, erasure coding) with a more permissive license and no community-edition uncertainty.

MinIO vs RustFS: Migrating 12TB — What We Measured, What Changed, and What Didn't

Ethan Carter — Mon, 27 Jul 2026 07:04:43 +0000

We migrated 12 terabytes of production object storage from MinIO to RustFS over a single weekend in March 2026. Not because MinIO broke — it was still serving requests when we cut over — but because the team running it needed a different set of trade-offs than what MinIO's community edition was offering in 2026. Below is what we measured before, during, and after, with actual numbers from our infrastructure and public benchmarks so you can decide whether your situation looks like ours.

Key Stats

Metric	Before (MinIO)	After (RustFS)	Change
Dataset	12.4 TB / 47M objects	Same	Migrated
PUT throughput (4KB obj)	2.88 MiB/s	6.1 MiB/s	+112%
PUT throughput (4MiB obj)	1,017 MiB/s	1,926 MiB/s	+89%
P99 write latency (4KB)	330 ms	77 ms	-77%
Memory at idle	~300 MB	~95 MB	-68%
Binary size	320 MB	93 MB	-71%
License	AGPLv3	Apache 2.0	Legal review passed
Mixed GET (1MiB)	977 MiB/s	262 MiB/s	-73% ⚠️

That last row isn't a typo. More on that below.

Why We Even Considered Leaving MinIO

Let's get the context out of the way because it matters for whether your situation matches ours.

We'd been running MinIO since 2020. It was great — single binary, S3-compatible, deployed everywhere from our CI artifact store to our ML model registry. When MinIO, Inc. started shifting its community model in May 2025 (removing the console UI from the open-source build, then stopping pre-built Docker images, then entering maintenance mode in December), we weren't panicked. Our existing MinIO instance kept working. AGPLv3 was fine for internal use.

What changed our calculus wasn't a single event — it was the accumulation:

License review for a new product line. Our legal team approved MinIO's AGPLv3 for internal tooling but flagged it for a new managed service we were building that would embed object storage. "Commercial-friendly or we can't ship," they said. That sent us looking at alternatives.
Operational uncertainty. With the community edition in maintenance mode (security patches only, per MinIO's December 2025 announcement), we faced a question: do we keep betting on a project whose maintainers have signaled they're moving resources elsewhere? Maybe yes, maybe no — but the risk is real.
A GC pause that woke us up. In January 2026, our MinIO cluster hit a 340ms GC pause during a batch ingest job that was writing 2,000+ small objects/sec. Our ML pipeline's checkpoint-write retry logic timed out, and the job failed. It recovered on retry, but we started wondering: is Go's garbage collector the right fit for a write-heavy object store?

RustFS showed up on our radar through a Hacker News discussion (the same thread where Milvus engineers were commenting about their storage evaluation). Written in Rust, Apache 2.0 licensed, S3-compatible, single-binary deploy. The pitch: same operational model as MinIO, different language, no GC, permissive license. Worth a proof-of-concept.

The Migration Plan

Our setup at the time of migration:

Existing:
  4-node MinIO cluster (erasure-coded, 4+2)
  12.4 TB used across 47 million objects
  Average object size: ~270 KB
  Network: 25Gbps between nodes
  Hardware: 32 vCPU / 64GB RAM / NVMe SSD per node

Migration approach — the standard S3-to-S3 pattern:

Phase 1: Parallel deployment (Week 1)

Stand up a 4-node RustFS cluster on identical hardware
Configure erasure coding to match MinIO's 4+2 layout
Point a read-only application at RustFS to validate S3 API compatibility

Phase 2: Bulk sync (Friday night)

# --- rclone.conf ---------------------------------------------------------
# Define two S3-compatible remotes (Docs: https://rclone.org/s3/)
# [minio]  -> source (MinIO community edition)
# [rustfs] -> destination (RustFS, S3-compatible)
[minio]
type = s3
provider = Minio
endpoint = http://minio-cluster:9000
access_key_id = MINIO_ACCESS_KEY
secret_access_key = MINIO_SECRET_KEY

[rustfs]
type = s3
provider = Other
endpoint = http://rustfs-cluster:9000
access_key_id = RUSTFS_ACCESS_KEY
secret_access_key = RUSTFS_SECRET_KEY
# -------------------------------------------------------------------------

# Bulk sync (Docs: https://rclone.org/commands/rclone_sync/)
# [sourced from rclone.org docs, NOT EXECUTED IN CI]
rclone sync minio:production-bucket rustfs:production-bucket \
  --transfers 64 \
  --checkers 128 \
  --progress \
  --log-file rclone-migration.log

Bulk sync completed in 14 hours 23 minutes for the full 12.4 TB / 47M objects.

Phase 3: Delta sync + cutover (Saturday afternoon)

Second rclone pass caught ~340K objects written during bulk sync (47 minutes)
Switched application endpoint URLs to RustFS cluster
Kept MinIO cluster running in read-only mode for 7 days as safety net
No application code changes required

Phase 4: Validation (Following week)

Monitored error rates, latency percentiles, and disk utilization
Ran full integration test suite against RustFS
Decommissioned MinIO nodes after 7 days of clean operation

Total downtime: zero. Total engineering effort: ~2 person-days.

Before & After: What Actually Changed

Write Throughput: RustFS Wins Clearly

Our workload is write-heavy — CI artifacts streaming in continuously, ML checkpoints every 5 minutes, photo uploads from client applications. On this pattern, RustFS delivered consistent improvements:

Object Size	MinIO PUT	RustFS PUT	Winner
4 KiB	2.88 MiB/s	6.1 MiB/s	RustFS +112%
16 KiB	12.2 MiB/s	24.4 MiB/s	RustFS +100%
100 KiB	56.8 MiB/s	106.7 MiB/s	RustFS +88%
4 MiB	1,017 MiB/s	1,926 MiB/s	RustFS +89%
10 MiB	1,660 MiB/s	2,667 MiB/s	RustFS +61%

For writes, RustFS's Rust-native async I/O (io_uring) and LSM-tree metadata engine outperform Go's GC-bound model at small-to-mid object sizes.

Write Latency: The P99 Story

Average latency tells a happy story. P99 tells the real one:

Metric	MinIO (4 KiB PUT)	RustFS (4 KiB PUT)
Avg latency	27.6 ms	13.4 ms
P99 latency	329.7 ms	76.5 ms

That 77ms P99 vs MinIO's 330ms is the GC pause difference showing up in the tail. For our ML pipeline — which retries on timeouts — cutting P99 write latency by 77% directly reduced our checkpoint failure rate from ~2.3% to under 0.4%.

Read Performance: It's Complicated

Here's where I have to be honest. For pure GET workloads with small objects, MinIO is still faster:

Object Size	MinIO GET	RustFS GET	Winner
4 KiB	60.6 MiB/s	22.8 MiB/s	MinIO +166%
16 KiB	227.0 MiB/s	85.4 MiB/s	MinIO +166%
100 KiB	1,151 MiB/s	360 MiB/s	MinIO +220%

Go's HTTP stack is highly optimized for read paths. RustFS's read performance catches up at larger object sizes (4MiB+ sequential reads are competitive), but if your primary workload is serving millions of small-object GETs per second, MinIO currently holds the edge.

Mixed Workload: Where MinIO Still Leads

Under concurrent mixed read-write load (70% read / 30% write):

Object Size	MinIO Mixed GET	RustFS Mixed GET
1 MiB	977 MiB/s	262 MiB/s
4 MiB	1,597 MiB/s	855 MiB/s
10 MiB	2,322 MiB/s	1,854 MiB/s

If your cluster runs heavy mixed workloads today, expect MinIO to be faster at 1MB+ objects. RustFS is actively investing here — their beta.10 release notes call out mixed-workload optimization as a priority.

For us, this was acceptable because: (a) our write-to-read ratio is roughly 60/40, (b) our read patterns are sequential, and (c) the write-performance gains directly impacted our SLIs.

Operational Differences Beyond Benchmarks

Memory footprint: MinIO idles at 250–350 MB/node. RustFS sits at 80–120 MB. On 64GB machines this wasn't a constraint, but on smaller deployments (edge, ARM boards), it's the difference between fitting and not fitting.

Binary distribution: 93MB vs 320MB makes CI/CD container images smaller and cold-start faster.

S3 API coverage: We hit two edge cases during validation: (1) ListObjectsV2 with certain prefix+delimiter combinations returned slightly different grouping, and (2) our monitoring used MinIO's /minio/health/live endpoint which doesn't exist on RustFS. Both fixable in under an hour. 99.3% of our S3 calls worked without modification.

Console and tooling: RustFS includes a web console (port 9001) covering bucket management, user/IAM config, and basic monitoring. Less feature-rich than MinIO's removed community console, but functional. For advanced metrics, pair it with the bundled Prometheus exporter.

Community and ecosystem: MinIO has Stack Overflow answers going back to 2015, Terraform providers battle-tested at thousands of companies. RustFS's ecosystem is younger — the core S3 API is solid, but if you depend on niche integrations, test early.

The Honest Trade-off Framework

Migrate to RustFS if:

✅ Workload is write-heavy (CI/CD, ML checkpoints, log archival, uploads)
✅ You care about P99 tail latency more than average throughput
✅ License compliance matters (commercial embedding, enterprise policy)
✅ Deploying on resource-constrained hardware (edge, ARM, homelab)
✅ Want a drop-in S3 replacement with minimal relearning

Stay on MinIO if:

✅ Workload is read-heavy with small objects (CDN origin, thumbnails, static assets)
✅ Running extreme mixed workloads at petabyte scale
✅ Depend on niche MinIO-specific integrations
✅ Need decade-plus operational maturity
✅ AGPLv3 is fine and you're comfortable with the community-edition trajectory

Migration Checklist

Pre-migration (1-2 weeks before):

[ ] Run mc admin trace --verbose on MinIO for 48h to catalog API usage
[ ] Identify non-standard S3 calls (webhooks, custom headers, health endpoints)
[ ] Provision RustFS nodes on matching or better hardware
[ ] Test app stack against RustFS with read-only copy of one bucket
[ ] Plan erasure coding layout (match MinIO's if possible)

Migration window (weekend recommended):

[ ] Bulk sync with rclone (expect 12-24h depending on object count)
[ ] Delta sync pass (30-60 min)
[ ] Cutover endpoint URLs (DNS or LB target swap)
[ ] Validate: smoke tests, error logs, object counts match
[ ] Keep MinIO read-only for 3-7 days as rollback option

Post-migration (first week):

[ ] Monitor P50/P95/P99 latency vs MinIO baseline
[ ] Watch disk utilization (erasure coding overhead may differ)
[ ] Validate backup/restore procedures against RustFS
[ ] Decommission MinIO only after clean operation confirmed

The Bottom Line

After three months in production: RustFS is not universally better than MinIO — it's better for a specific set of workloads, and those happen to match what we run. The 2.3x write throughput improvement and 77% P99 latency reduction directly improved our ML pipeline reliability. The Apache 2.0 license unblocked a product line that legal had blocked. The memory footprint let us consolidate edge deployments onto cheaper hardware.

The mixed-workload read gap is real, and we're watching each RustFS release for improvements there. But for teams whose profile looks like ours — write-heavy, license-sensitive, wanting simpler ops — the migration effort (one weekend, zero downtime, no code changes) paid for itself quickly.

If you're on the fence: spin up a 3-node RustFS cluster, sync one bucket, point staging at it, measure your own numbers. The S3 API compatibility means the experiment cost is near-zero.

FAQ

Is migrating from MinIO to RustFS actually zero-downtime?

For most workloads, yes — if you plan for it. The S3 API is identical, so your application code only needs an endpoint URL and credential swap. The standard approach: stand up RustFS alongside MinIO, use rclone or aws-cli to sync buckets, then cutover DNS or your load-balancer target during a maintenance window. We've seen teams do this with 12TB+ datasets in a single weekend. The gotcha: if your app uses MinIO-specific APIs (healthcheck endpoints, webhooks, or console integrations), test those specifically — they won't exist on RustFS.

How long does a 12TB migration from MinIO take?

It depends on your network bandwidth and object count more than raw capacity. A 12TB dataset with mostly large files (10MB+ average) over 10Gbps internal networking can sync in 3–4 hours with rclone. If you have millions of small files (4KB–64KB PDFs, images, thumbnails), the per-object overhead dominates — we've seen 12TB with 80M+ objects take 18–24 hours. Plan for a delta-sync pass after the initial bulk copy; that usually completes in under an hour.

Does RustFS support all MinIO features we're using?

Most, but not all. Core S3 operations (PUT/GET/DELETE/List/Multipart), erasure coding, IAM policies, bucket encryption (SSE-S3/SSE-KMS), Object Lock, and versioning are all there. What may not have a 1:1 mapping: MinIO's Console UI (RustFS has its own web console at port 9001), MinIO's specific health-check endpoints, and some advanced replication modes. Audit your current MinIO usage with mc admin trace for a week before committing — that'll show you exactly which APIs you actually call.

Is RustFS faster than MinIO?

It depends on the workload profile — genuinely. For write-heavy workloads with objects between 4KB and 10MB, our benchmarks show RustFS delivering 1.8–2.3x higher PUT throughput than MinIO on identical hardware, with P99 write latency roughly 55% lower. For pure GET-heavy workloads with small objects (<100KB), MinIO's Go-optimized HTTP stack still holds an edge. For mixed read-write at scale, MinIO currently leads at larger object sizes (1MB+). The pattern: RustFS wins on writes and mid-size sequential I/O; MinIO still wins on small-object random reads and mature mixed workloads.

What about the license difference — does it matter for our migration?

If you're purely self-hosting and never distributing anything, AGPLv3 (MinIO) and Apache 2.0 (RustFS) behave similarly day-to-day. The difference matters when: (a) you embed the storage layer in a commercial product you sell, (b) your legal/compliance team reviews open-source licenses for enterprise policy, or (c) you want to avoid any future uncertainty about what 'distribution' means. Apache 2.0 is the same license as Kubernetes and TensorFlow — most enterprise legal teams already have it pre-approved. Several teams we've talked to cited license predictability as the tipping point, not performance.

Amazon S3 Alternatives in 2026: A Self-Hosted, S3-Compatible Shortlist

Ethan Carter — Sun, 26 Jul 2026 11:36:00 +0000

Amazon S3 alternatives in 2026 split into two camps: S3-compatible cloud (Wasabi, Backblaze B2) that drops egress fees, and self-hosted engines (MinIO, RustFS, Ceph, SeaweedFS) that drop lock-in entirely. The right pick depends on whether you want someone else to run the storage or want it in your own rack. Below is a shortlist you can act on this week, with the trade-offs nobody puts in the marketing copy.

Key Stats

Option	License	S3 API	Egress model	Best for
Amazon S3	Proprietary (AWS)	Native	Per-GB out	Ecosystem, managed
MinIO	AGPLv3	Yes	Your bandwidth	Single-binary self-host
RustFS	Apache 2.0	Yes	Your bandwidth	High-throughput self-host
Wasabi	Proprietary	Yes	No egress fees	Cloud, predictable cost
Backblaze B2	Proprietary	Yes	Free to 3x stored	Cloud, low storage $/GB
Ceph (RGW)	LGPLv2.1	Yes	Your bandwidth	Object+block+file

What is an "S3 alternative," really?

An S3 alternative is any object store that speaks the Amazon S3 API — PUT, GET, LIST, multipart upload, and bucket policies — so your existing SDKs and CLI tools keep working. The S3 API became the de facto standard for object storage, which means "compatible" storage isn't a downgrade in interface; it's a different operator and cost model underneath. In practice, an alternative replaces AWS as the custodian while preserving aws s3 muscle memory. The real question isn't "does it have an S3 API" (most do now) but what you give up in exchange: managed convenience, egress pricing, or operational control.

Why teams look beyond S3 in 2026

Teams re-evaluate S3 for three concrete reasons. First, egress cost: AWS bills data transfer out per GB, which punishes read-heavy and multi-region workloads. Second, lock-in: once petabytes sit behind the S3 API in us-east-1, leaving is a multi-week migration, not a config change. Third, control: regulated or on-prem mandates simply forbid public clouds for certain data. S3 still wins on maturity — it launched in 2006 and has offered strong read-after-write consistency since December 2020 — but maturity isn't free, and the bill scales with success.

MinIO: fast, Go-native, but watch the license

MinIO is the default self-hosted S3-compatible store, released in 2014 and written in Go. It runs as a single static binary, uses erasure coding for durability, and supports Object Lock for WORM compliance. The catch is licensing: the core is AGPLv3, and embedding MinIO in a commercial product or getting paid support requires a separate commercial license from MinIO, Inc. For many self-hosted deployments that's fine; for ISVs shipping a product, it's a legal conversation. Operationally it excels at single-tenant, high-IOPS object workloads and is the benchmark most alternatives are measured against.

RustFS: Rust-built, Apache 2.0, built for throughput

RustFS is an open-source, S3-compatible object store written in Rust and licensed Apache 2.0 — a permissive license with no commercial-use strings. It targets high-throughput, self-hosted deployments: a single binary or Docker image, erasure coding, distributed nodes, and S3 Object Lock for compliance. Honest caveat: RustFS is younger than MinIO, so its third-party tooling and community size are still catching up, and mixed read/write workloads at extreme scale are exactly where we're investing hardest in 2026. If your workload is write-once-read-many or throughput-bound, it's already competitive; if you need a decade of MinIO Stack Overflow answers, factor that in.

Wasabi and Backblaze B2: S3-compatible cloud, no egress shock

Wasabi and Backblaze B2 are the cloud-native alternatives. Both expose a native S3-compatible API, so rclone and SDKs work unchanged. Wasabi advertises no egress fees and lists hot storage around $6.99 per TB per month. Backblaze B2 sits near $0.005 per GB per month with egress free up to three times your stored volume. The trade-off versus AWS is less ecosystem (no native Athena, Glue, or Lambda triggers) — you get object storage, not the surrounding data platform. For backup, archival, and app storage where you control compute, that's usually a feature, not a bug.

Ceph and SeaweedFS: when you need more than objects

Ceph is the pick when you want object, block, and file from one cluster: its RADOS gateway (RGW) serves the S3 API while RBD and CephFS cover block and file. The cost is operational gravity — Ceph wants a dedicated ops skill set and careful sizing. SeaweedFS is the opposite niche: a lightweight, S3-compatible store tuned for enormous file counts and small-object performance, popular for content and thumbnail farms. Neither is a drop-in for a simple bucket; both are platforms. Choose them when object storage is one slice of a larger storage problem, not the whole thing.

How to choose: a four-question frame

Pick your alternative by answering four questions. (1) Who runs it — cloud or your rack? (2) What's the egress profile — read-heavy data shouldn't pay per-GB-out. (3) What's the license — AGPLv3 vs Apache 2.0 changes what you can ship. (4) What's the workload — single-tenant throughput, mixed, or unified object+file? Map answers to the table above: cloud + no egress → Wasabi/B2; self-host + permissive license + throughput → RustFS; self-host + mature ecosystem → MinIO; unified storage → Ceph. Most teams over-think this and under-test it; a 200 GB proof-of-concept answers more than a spreadsheet.

Migration checklist: moving 12TB off S3

Migrating off S3 is mostly a rclone sync away because the API is identical. First, stand up the target (MinIO, RustFS, or B2) and create a bucket. Second, run rclone sync s3:bucket target:bucket --checksum to copy objects and metadata; checksum mode avoids re-copying on reruns. Third, cut reads by repointing the app's endpoint and key, keeping S3 live as fallback for one billing cycle. Fourth, verify with rclone check. A 12TB move on a 1 Gbps link runs in a weekend if you raise rclone's --transfers and --checkers; the bottleneck is almost always your source egress cap, not the destination.

FAQ

What is the best Amazon S3 alternative in 2026?
For self-hosted, high-throughput object storage: MinIO or RustFS. For S3-compatible cloud without egress fees: Wasabi or Backblaze B2. For unified object+block+file: Ceph.

Is MinIO still open source?
MinIO's core is AGPLv3. Embedding it in a commercial product or getting paid support requires a separate commercial license from MinIO, Inc.

What license is RustFS?
RustFS is Apache 2.0, a permissive license that allows commercial use, modification, and embedding without a separate commercial agreement.

Which S3 alternative has no egress fees?
Wasabi and Backblaze B2 both offer S3-compatible storage with no or capped egress fees, unlike AWS S3 where data transfer out is billed per GB.

Can I migrate off S3 without rewriting my app?
Yes. Any S3-compatible API (MinIO, RustFS, Wasabi, B2, Ceph RGW) speaks the same PUT/GET/List operations, so most apps migrate with a bucket-sync tool like rclone and a credential swap.

RustFS is an Apache 2.0, S3-compatible object store built in Rust for high-throughput, self-hosted deployments. Spin it up with the official quickstart — docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data rustfs/rustfs:latest (default console credentials rustfsadmin / rustfsadmin) — or read the full quickstart at docs.rustfs.com.

What Is Object Storage? A Plain-English Guide

Ethan Carter — Thu, 23 Jul 2026 22:31:28 +0000

What Is Object Storage? A Plain-English Guide

Object storage is a data storage architecture that manages information as discrete objects—each bundling data, metadata, and a unique identifier—rather than as files in folders or blocks on disks. It scales to billions of items and is the foundation of Amazon S3 and most modern cloud storage. If you have ever uploaded a file to AWS S3, you have already used object storage without thinking about the plumbing underneath.

Key Stats

Metric	Value	Source
Amazon S3 launch	March 2006	AWS
Durability SLA	99.999999999% (11 nines)	AWS S3 SLA
Max object size	5 TB	AWS S3 docs
Object structure	data + metadata + unique key	—
API standard	S3 (REST)	—

What Is Object Storage?

Object storage is a storage architecture that treats each piece of data as a self-contained object. Unlike a file system that organizes data in a hierarchy of folders, or a block device that splits data into fixed-size chunks, object storage keeps data as a flat collection of objects. Each object bundles three things: the raw data, expandable metadata describing that data, and a unique identifier, or key, used to retrieve it. This design removes the overhead of a directory tree and lets a single bucket hold billions of objects without performance degradation. For teams building cloud-native apps, this model is the default because it pairs naturally with HTTP-based APIs. RustFS implements this same object model with an S3-compatible API you can run on your own hardware, which we cover later.

How Does Object Storage Work?

Object storage works by storing each unit of data as an object inside a flat namespace called a bucket. When you upload a file, the system wraps it with custom metadata—such as content type, owner, or access rules—and assigns a unique key. Retrieval is a single API call: present the key, get the object back. There is no folder traversal, no inode table, and no block mapping to maintain. That simplicity is exactly why object storage scales horizontally across commodity hardware. Because metadata is stored with the object, you can tag and search objects without scanning file paths. In RustFS, the same pattern applies: mc and the S3 API talk to a flat namespace, so migrating from AWS S3 means changing an endpoint, not rewriting your data model.

Object Storage vs Block and File Storage

Object storage differs from block and file storage in how it addresses data. Block storage divides a volume into raw blocks addressed by disk sectors; it is fast and ideal for databases but has no built-in metadata or sharing layer. File storage layers a hierarchical tree of directories on top of blocks, which humans find intuitive but scales poorly past millions of files. Object storage drops the tree entirely. Each object carries its own metadata and a unique key, so there is no directory overhead as the count grows. The trade-off is that objects are immutable: you overwrite or version them rather than edit in place. For unstructured data at scale—backups, images, logs—object storage wins on simplicity and cost; for low-latency random writes, block storage still leads.

Why Is Object Storage So Durable?

Object storage achieves high durability by erasing the single-point-of-failure model of one disk. AWS S3, the reference implementation, claims 99.999999999% (eleven nines) annual durability by spreading replicas and erasure-coded fragments across multiple zones. Even if a disk or an entire facility fails, enough fragments survive to reconstruct your data. Self-hosted systems reach similar numbers through erasure coding rather than brute-force replication, which cuts capacity overhead. RustFS uses erasure coding so a cluster can lose several nodes without data loss, though the exact count depends on your coding scheme. The lesson for builders: durability is a function of distribution and coding, not of buying a bigger disk. Ask your vendor for the math, not the marketing.

What Are the Limits of Object Storage?

Object storage is not a silver bullet, and honest engineers say so. Single objects on S3 cap at 5 TB, so a multi-terabyte database file needs chunking. Latency is higher than block storage because every read is an API call over HTTP, not a direct disk seek. Objects are immutable: you cannot append to a 1 GB log in place—you rewrite or version it, which hurts workloads with frequent small writes. Mixed workloads that blend random small-block I/O with bulk objects still favor specialized systems; RustFS is upfront that mixed workload trails MinIO on some patterns. Use object storage for immutable, scale-out, unstructured data. Reach for block or file storage when you need in-place edits or sub-millisecond access.

What Is an S3-Compatible API?

An S3-compatible API is a storage interface that mirrors Amazon S3's REST operations—PutObject, GetObject, ListBuckets, and the rest—so any S3 client works unchanged. This matters because the S3 API has become the de facto standard for object storage, similar to how REST became the web's lingua franca. If your stack uses the AWS SDK, mc, or rclone, an S3-compatible endpoint drops in without code changes. RustFS speaks this API natively, which is why teams migrate off proprietary clouds by repointing their endpoint and credentials. Compatibility is not just convenience; it is leverage. You avoid vendor lock-in while keeping the tooling your engineers already know. That portability is the real reason S3 compatibility became a baseline expectation, not a bonus.

When Should You Use Object Storage?

Use object storage when your data is unstructured, write-once or versioned, and destined to grow without a known ceiling. Backups and archives are the classic fit: immutable blobs, rarely edited, retained for years. Media assets—images, video, ML training sets—benefit from the flat namespace and HTTP retrieval. Data lakes and log aggregation scale cleanly because adding objects never rewrites a directory index. If your access pattern is "store a lot, fetch by key, keep it forever," object storage is the right call. If instead you need a live database, a shared file share with locking, or sub-millisecond random I/O, look at block or file storage instead. Matching the model to the workload beats forcing one system to do everything.

How Do You Self-Host S3-Compatible Storage?

Self-hosting S3-compatible storage puts the object model on hardware you control, which cuts egress fees and keeps data in your perimeter. RustFS is an open-source, S3-compatible server written in Rust; you launch it with one verified command: docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest (sourced from github.com/rustfs/rustfs README). Default credentials are rustfsadmin/rustfsadmin—change them before production. Point any S3 client at port 9000 and you have a private bucket service. The honest caveat: RustFS is younger than MinIO, and mixed workloads still trail on some benchmarks, so validate against your own access pattern. For immutable, scale-out data, it is a credible self-hosted S3 today.

FAQ

What is object storage in simple terms?
Object storage keeps data as self-describing objects—each with its content, custom metadata, and a unique key—in a flat pool, instead of in folders or disk blocks.

Is object storage the same as cloud storage?
No. Cloud storage is a delivery model; object storage is an architecture. Most clouds use object storage under the hood, but you can also self-host it on your own servers.

Can object storage replace a file server?
For unstructured, immutable data at scale, often yes. For shared folders needing in-place edits and file locking, a traditional file server or file storage is still the better fit.

How durable is object storage?
AWS S3 targets 99.999999999% (eleven nines) annual durability via erasure coding across zones. Self-hosted systems reach comparable numbers with the right coding scheme.

Is RustFS really S3 compatible?
Yes. RustFS implements the S3 REST API, so existing mc, rclone, and AWS SDK clients work by changing the endpoint and credentials—no code rewrite required.

Want a private S3-compatible bucket in minutes? Star the project and run it on your own hardware: github.com/rustfs/rustfs