S3 Metadata: Indexing Billions of Objects Without Falling Over

If you operate S3-compatible storage at scale, slow ListObjectsV2 requests will eventually find you. Stay under your metadata threshold, and listings return fast. Cross it, and you'll watch timeouts, Slow Down errors, and after-hours alerts pile up.

Every storage backend hits this wall at a different object count. Let's walk through what breaks, how popular systems behave, and practical fixes you can roll out.

What even is S3 metadata?

When you upload an object to S3, two pieces get saved:

• Your raw file data
• Object metadata — all descriptive information attached to that file

Here's what metadata typically includes:

Metadata Field	Example	Typical Size
Object Key	logs/app/2026-07-24/abc123.log	Variable (50–500 bytes)
Content Length	1234567 bytes	8 bytes
Content-Type	application/json	~25 bytes
ETag	"d41d8cd98f00b204e9800998ecf8427e"	36 bytes
Last-Modified	2026-07-24T15:30:00Z	~24 bytes
Storage Class	STANDARD	~12 bytes
User Tags	env=production&team=backend	Up to 2KB (combined)
Custom Metadata Headers	x-amz-meta-owner:uid-12345	Up to 2KB (combined)
Version ID	RgvpLpBZLXhPvWXNk6H_8sN1FnNtIqF	~40 bytes

Each object adds roughly 200–800 bytes of metadata overhead, stored separately from your actual data payload.

Why LIST operations get painfully slow at scale

Every ListObjectsV2 call forces the storage service to:

1. Scan its metadata index for keys matching your prefix and delimiter
2. Apply filters and limits
3. Sort matching keys lexicographically
4. Paginate results (max 1,000 entries per page + continuation token)

Real-world performance you'll observe (typical for naive prefix scans; your mileage varies with key layout and access patterns):

• Under 10k objects: List calls finish in <100ms, no issues
• ~1M objects: Queries take 1–5s depending on how narrow your prefix is
• ~100M objects: Latency balloons to 10–60+ seconds, or requests timeout
• 1B+ objects: Naive LIST calls aren't reliable anymore — you need a proper plan.

The real bottleneck isn't your file data

The pain point is the metadata index. Major S3 implementations handle metadata very differently:

Implementation	Metadata Design	Scaling Behavior
AWS S3	Proprietary distributed index	Supports billions of objects; LIST throttling kicks in at extreme scale
MinIO	Object metadata in xl.meta alongside data, with an in-cluster .metacache + background scanner for LIST	Scales into the hundreds of millions with selective prefixes; LIST at extreme scale depends on the metacache staying warm
Ceph RGW	RADOS per-entry metadata	Scales reasonably, but LIST linearly scans matching entries
RustFS	Embedded RocksDB + custom sharded index	Built for scale — subsecond LIST with selective prefixes at 100M+ objects (validated in RustFS internal benchmarks)
SeaweedFS	Filer + LevelDB/SQL backend	Performance heavily depends on your database backend choice

Actionable strategies for high-object-count buckets

Strategy 1: Namespace partitioning (start here)

The highest-impact change you can make: don't cram everything into one bucket.

# Bad: One giant monolithic bucket
s3://prod-data/
├── logs/                    # 500M objects
├── user-uploads/            # 200M objects
├── ml-checkpoints/          # 50M objects
├── backups/                 # 10M objects
└── temp/                    # 5M objects
# Total: 765M objects. Every LIST scans every namespace.

# Better: Split workloads into dedicated buckets
s3://prod-logs/              # 500M objects (isolated metadata index)
s3://prod-uploads/           # 200M objects (isolated metadata index)
s3://prod-ml-checkpoints/    # 50M objects (isolated metadata index)
s3://prod-backups/           # 10M objects (isolated metadata index)

Each bucket runs its own independent metadata index. Listing inside prod-uploads never touches metadata from your log bucket.

Quick note on cost: AWS doesn't charge per bucket, and self-hosted S3 systems generally don't penalize extra buckets.

Strategy 2: Structure your object prefixes smartly

Even within a single bucket, organized key paths drastically speed up prefix filtering.

# Poor: Flat layout — LIST must scan everything
s3://bucket/object-000001.json
s3://bucket/object-000002.json

# Improved: Hierarchical prefixes narrow the scan range
s3://bucket/year=2026/month=07/day=24/hour=15/object-abc123.json

A request using Prefix="year=2026/month=07/" only loads metadata for July data, instead of your entire bucket.

Strategy 3: Avoid full-bucket scans at all costs

This anti-pattern destroys performance on large buckets:

# ❌ Don't use this against large buckets — scans full metadata
for obj in s3.list_objects_v2(Bucket='my-bucket'):
    process(obj)

# ✅ Use targeted prefix pagination
paginator = s3.get_paginator('list_objects_v2')
for page in paginator.paginate(Bucket='my-bucket', Prefix='logs/2026-07/'):
    for obj in page.get('Contents', []):
        process(obj)

Strategy 4: Run an external metadata index

Once you hit roughly 100M objects, maintain a separate metadata database alongside S3.

-- Sample schema for PostgreSQL / MySQL / SQLite external index
CREATE TABLE s3_objects (
    key TEXT PRIMARY KEY,
    bucket TEXT NOT NULL,
    size BIGINT,
    last_modified TIMESTAMPTZ,
    content_type TEXT,
    tags JSONB,
    created_at TIMESTAMPTZ DEFAULT NOW()
);

-- Indexes tuned for common queries
CREATE INDEX idx_s3_objects_bucket_prefix ON s3_objects(bucket, key text_pattern_ops);
CREATE INDEX idx_s3_objects_tags ON s3_objects USING GIN(tags);

Sync the table on every PutObject / DeleteObject via S3 event notifications or application hooks. Query metadata directly from the database, and only use S3 API to read/write actual files.

Tradeoff: You operate another database. Win: metadata lookup latency stays consistent no matter how many objects you store.

Strategy 5: Pick storage built for your expected scale

If you know you'll grow past 100M objects, evaluate systems based on their metadata architecture.

Good questions to ask vendors:

• What's p99 ListObjectsV2 latency on a 100M-object bucket with a selective prefix?
• Does metadata run in-process, or rely on an external coordination service?
• How does LIST performance hold up during rebalancing or recovery?
• Can I read metadata without going through the standard S3 API?

RustFS metadata overview:

Every node hosts embedded RocksDB for metadata. Clusters shard metadata by object key hash:

• Single node: Subsecond LIST responses for 100M+ objects with selective prefixes
• Clustered: Metadata split evenly across nodes
• Recovery: Metadata replicates alongside object data; LIST stays available (minor staleness possible)
• Offline option: Export metadata snapshots to SQLite for analytics

When native S3 metadata stops fitting your workload

Once you hit ~1B objects and need complex filtering, vanilla object storage isn't ideal for metadata-heavy workflows.

Symptom	Fix
Constant LIST timeouts	Deploy external metadata index (Strategy 4)
Complex filters (size limits, tag matching)	Data lake formats (Iceberg / Delta Lake) or dedicated metadata DB
Full-text search over object names	Add Elasticsearch / OpenSearch sidecar
Relational-style joins on object metadata	Move metadata out of S3 into a relational database

S3 shines when you fetch objects by known keys. It was never designed as a flexible query engine for object metadata.

TL;DR

• Every S3-compatible storage hits a metadata scaling limit. Find your threshold before production hits it.
• Split data across multiple buckets — this gives you the biggest performance return.
• Build hierarchical object keys to make prefix filtering effective.
• Never run full-bucket scans in production; always use prefix filters + pagination.
• Around 100M objects, plan an event-driven external metadata store (PostgreSQL or SQLite).
• If you expect massive scale, prioritize metadata architecture during storage selection. RustFS (RocksDB-backed) and AWS S3 handle billions of objects better than etcd-dependent stacks.

Tired of slow LIST calls with millions of stored objects? RustFS uses RocksDB-backed metadata to deliver subsecond listings even at 100M+ objects. Download RustFS.

FAQ

Is there a hard maximum number of objects per bucket?

Theoretically no. AWS says buckets support virtually unlimited objects. Practically, your limit comes from metadata index speed.

Most teams notice LIST slowdowns between 10M–50M objects, depending on key layout and access patterns. With clean partitioning and prefix design, 100M+ objects work fine. Once you near 1B objects, build an external metadata index regardless of storage platform.

Why do ListObjects requests slow down as object counts grow?

ListObjects scans matching metadata entries, sorts keys lexicographically, and paginates results. Sorting creates heavy O(N log N) overhead when thousands or millions of entries match your prefix.

Worse: most clients loop pagination, turning one logical job into dozens of API round-trips stacked with network latency.

Fixes: tighter prefix filters, bucket partitioning, or offloading metadata queries to an external database.

Does bucket naming affect S3 performance?

Generally not. Bucket names are routing and namespace identifiers, not performance controls. Operationally, stuffing all data into a small number of giant buckets concentrates metadata pressure and slows LIST. Spreading load across more buckets distributes pressure. AWS explicitly confirms bucket naming does not impact performance.

The same applies to self-hosted systems like MinIO and RustFS. Bucket-level metadata overhead is negligible. Spend tuning time on key structure and partitioning instead.

What's the standard way to track S3 metadata externally?

The common pattern: PostgreSQL (or MySQL) with (bucket, object_key) as the primary key.

Sync updates via S3 event notifications (SQS/SNS/Lambda on AWS; webhooks for self-hosted storage) on object creates, copies, and deletes. Query metadata directly from the database and reserve S3 API purely for file reads and writes.

For lighter workloads with tens of millions of rows, SQLite works well. If you need search, pair the relational database with Elasticsearch or OpenSearch. The downside is another system to maintain; the upside is stable metadata query performance independent of bucket size.

Can I replace LIST with S3 Select for metadata filtering?

No. S3 Select runs SQL inside object payloads (CSV, JSON, Parquet). It cannot filter on object-level attributes: names, sizes, tags, timestamps.

Metadata filtering still requires either ListObjectsV2 (with scaling limits) or an external index. S3 Select solves a separate problem: scanning file contents without full downloads. Both tools are useful, but they cannot replace each other.