DEV Community: MaxHuo

Why Object Storage Rewrites Whole Objects (It's Far More Than Just Consensus)

MaxHuo — Tue, 30 Jun 2026 15:02:00 +0000

A junior on my team pulled me aside yesterday with that familiar frustrated look—the kind that screams he's convinced he found a massive, stupid design flaw.

He pulled up our monitoring dashboard and pointed aggressively. "Look at this. I changed one freaking character in a 5 MB JSON file—only one single byte modified. The system reuploaded the entire file, saturated our network ingress and kicked off a full garbage collection cycle. Why can't it just flip that byte directly on disk, like my laptop's drive does?"

It's a totally fair question. Consensus and replication do factor into object storage design, but they aren’t the sole driving force. The first constraint lives much lower in the stack: how physical storage hardware performs best under different write patterns. But that’s only half the equation—distributed cluster requirements add another layer of pressure pushing object storage toward immutable write patterns.

One major foundational bottleneck sits in every storage rack: spinning hard drives, running at 7,200 RPM.

Let's pop the hood on spinning hard drives

Object storage is built for massive scale. Even in 2026, that scale still relies on spinning rust—HDDs. No team can afford to host multi-exabyte datasets entirely on all-flash arrays without draining the entire cloud budget.

The core issue comes down to basic hard drive physics: disks hate random access, and they thrive on sequential streaming.

If you edit a single byte buried in the middle of a 5 MB file on an HDD, the hardware has to execute a full read-modify-write sequence, with three distinct steps:

Pull the full 4K (or larger) sector into memory
Tweak that one tiny byte in RAM
Wait for the platter to rotate back, then move the read/write head to the exact physical track again

That mechanical seek operation costs roughly 10 milliseconds.

Ten milliseconds sounds negligible on its own, until you imagine 10,000 concurrent microservices all patching tiny snippets of data at once. Drive heads flick back and forth across platters nonstop, like a broken metronome having a seizure. IOPS collapse to single digits, and throughput tanks from 200 MB/s down to dial-up levels.

Object storage isn't optimized for frequent, fine-grained updates. Its strengths are large sequential writes, high durability, and massive scalability. The simplest way to lock in consistent high bandwidth is to avoid random head movement entirely. Instead of mutating existing object data in-place, most object storage implementations generate an entirely separate new object, then flip metadata pointers to mark this new version as live.

We sacrifice random write flexibility to keep drives performant—and save ourselves endless operational headaches.

Hardware explains why sequential writes are efficient.
Distributed systems explain why immutable objects are easier to coordinate.

Modern object storage inherits both ideas. The HDD behavior gave us the mechanical push toward append-only patterns, but the distributed coordination challenges gave us the architectural lock-in.

Yeah, but what about NVMe SSDs?

SSDs have no moving heads, but they carry their own critical flaw: write amplification.

SSD hardware writes data in 4KB pages, but erases data in large 2MB–4MB blocks. If you overwrite that tiny changed byte in-place, the controller has to read the entire block into cache, fully erase the physical block (a destructive operation), then rewrite the whole block just to apply your tiny change.

This chews through the drive's limited Program/Erase (P/E) cycles at an alarming rate. Frequent small in-place overwrites significantly accelerate SSD wear-out due to write amplification, shortening drive lifespan far faster than sequential workloads would. For most large-scale cloud deployments, the extra storage overhead from immutable full rewrites is a worthwhile tradeoff. This design delivers more stable hardware lifespans, simpler failure recovery logic, and consistent throughput—benefits that outweigh the cost of retaining duplicate object versions until garbage collection cleans them up.

Fine, but what about the distributed cluster side of things?

Now we get to the topic everyone fixates on: consistency. Consensus challenges are the other half of the story, but they're not where the physical constraints begin.

That said, supporting in-place updates across three data replicas (Nodes A, B, C) creates an unmanageable pile of edge cases.

Picture a client sending a tiny patch mid-write, and Node B crashes halfway through the operation. Node A holds the updated data, Node B is left with corrupted partial bytes, and Node C still serves the original file.

Which version do we serve for the next GET request?
Do we roll back changes entirely, or risk serving a broken mix of old and new data?
How do we manage cross-node locks without crushing read throughput?

Immutable objects eliminate this whole mess. We write the complete new object copy to every node first. Once all replicas validate the checksum successfully, the metadata layer swaps one single pointer to mark the new version as live. If the upload fails halfway? The original object stays fully intact and continues serving traffic. There's no partial state to reconcile, no emergency 2 AM incident calls to untangle split-brain data issues.

An atomic metadata swap is trivial to implement. Safe partial in-place writes across distributed replicas? That's considerably harder to implement safely and reason about.

The unadvertised downside vendors skip in marketing docs

This architecture comes with an unavoidable, messy side effect: rampant storage bloat.

If you repeatedly edit a 10 GB database backup every five minutes (please don't do this), you generate another 10 GB of orphaned stale data every cycle. Object storage quickly turns into a dumping ground unless you carefully tune lifecycle policies and garbage collection cadence.

Versioning is widely marketed as a user recovery feature, but it's really a natural side effect of immutable storage—one that vendors package as a sellable "one-click rollback" capability.

If you treat object storage like a local hard drive, you're going to have a bad time.

It's not built for frequent edits. It's built for publishing—upload once, leave it untouched, delete only when you're certain. It trades random-write flexibility for raw throughput, accepts slower edits to extend hardware lifespan, and dodges the worst distributed consensus nightmares by simply refusing to play the "modify" game at all.

Once you frame object storage as an append-first system rather than a mutable local filesystem, every counterintuitive design choice falls into place.

Why Listing Objects Is One of the Hardest Operations in Cloud Storage

MaxHuo — Wed, 24 Jun 2026 14:29:00 +0000

This is Part 4 of the Object Storage Internals series.
Previous articles covered core mental models, metadata bottlenecks, and consistency tradeoffs.

When people think about object storage performance, they usually focus on reads and writes.

That makes sense.

Uploading an object sounds expensive.

Downloading an object sounds expensive.

Listing objects sounds trivial.

After all, how hard can this be?

GET /photos/

Return all the objects and move on.

The first time I started looking at object storage internals, I assumed listing was one of the easier operations.

I was wrong.

In many large-scale storage systems, listing objects is significantly more complicated than reading a single object.

Reading one object is usually straightforward

Suppose a client requests:

GET /photos/cat.png

The storage system only needs to answer a few questions:

Does the object exist?
Where is it stored?
Which nodes hold the data?

Once metadata provides the answer, the object can be retrieved.

The operation is targeted.

The system knows exactly what it is looking for.

Listing is a completely different problem

Now consider:

LIST /photos/

The request no longer asks for one object.

It asks for every object matching a prefix.

In a small system this is easy.

In a large distributed storage system, it becomes surprisingly expensive.

Imagine:

100 billion objects
Hundreds of storage nodes
Metadata distributed across partitions

The answer to a listing request may be spread across dozens of machines.

No single node necessarily knows the complete answer.

The system has to assemble reality

A read operation usually follows a path:

Object Name
      ↓
Metadata Lookup
      ↓
Storage Node

A listing operation often looks more like:

Client
   ↓
Metadata Partition 1
Metadata Partition 2
Metadata Partition 3
Metadata Partition N
   ↓
Merge Results
Sort Results
Remove Duplicates
Return Response

The system is effectively reconstructing a view of reality from multiple sources.

That takes work.

Consistency makes it harder

Things become more interesting when objects are changing while a listing operation is running.

Imagine:

Client A uploads object X
Client B deletes object Y
Client C performs LIST

What should Client C see?

The answer depends on the consistency guarantees of the system.

Some storage systems prioritize a consistent view.

Others prioritize performance and availability.

Either way, the metadata layer now has to make decisions.

This is one reason why listing is often a metadata problem rather than a storage problem.

The first surprising lesson

Many engineers assume object storage is primarily about moving data.

In practice, large storage systems spend an enormous amount of effort managing information about data.

The actual object might be sitting safely on disk.

The hard part is determining whether that object should appear in a query result right now.

Why S3 listing behavior confused developers for years

Historically, developers occasionally encountered situations where:

Upload succeeds
Immediate LIST request occurs
Object does not appear

The object existed.

The storage system had accepted the write.

The issue was that metadata updates had not fully converged.

From the developer's perspective, it felt like a bug.

From the storage system's perspective, it was a consequence of the consistency model.

This is one of the reasons object listing became such an important topic in storage architecture.

Scale changes everything

Imagine a bucket containing:

10,000 objects

Listing is easy.

Now imagine:

10 billion objects

The problem changes completely.

Questions suddenly appear:

How should metadata be partitioned?
How are results sorted?
How is pagination handled?
How much memory should listing consume?
How many metadata servers participate?

The operation that looked simple now touches some of the most important architectural decisions in the entire system.

Why storage engineers care so much about metadata

After spending time studying object storage systems, one pattern keeps appearing:

Whenever something becomes difficult, metadata is usually involved.

Part 2 of this series argued that metadata is the real system.

Listing operations are a good example.

The object data itself is often not the challenge.

The challenge is maintaining an accurate and scalable view of billions of objects while the system is constantly changing.

The trade-off nobody sees

Users see:

LIST /photos/

Storage engineers see:

Distributed metadata
Consistency guarantees
Partitioning strategy
Pagination
Failure handling
Concurrency
Scalability

The API looks simple because the storage system absorbs the complexity.

That simplicity is expensive.

Key takeaway

Reading a single object is usually about finding data.

Listing objects is about understanding the state of the entire system.

That's why listing often becomes one of the most metadata-intensive operations in cloud storage.

The bigger the system becomes, the more difficult that problem gets.

Next in this series

Part 5: Why Object Storage Systems Avoid In-Place Updates

If updating a file seems simple on your laptop, why do many object storage systems prefer creating new versions instead of modifying existing data?

Strong vs Eventual Consistency in Distributed Storage (Without the Confusion)

MaxHuo — Mon, 22 Jun 2026 13:51:00 +0000

After writing about metadata in object storage systems, I kept coming back to the same question:

If metadata is distributed across multiple machines, how do those machines agree on what is true?

This is where consistency enters the picture.

The problem is that consistency is often explained through CAP theorem diagrams and academic terminology. While those are important, they don't always help when you're trying to understand how a storage system actually behaves.

A more practical question is:

If I upload an object right now, when should everyone else be able to see it?

That question sits at the heart of consistency.

A storage engineer's version of the problem

Imagine a storage cluster with three metadata nodes.

A client uploads:

photos/cats.png

The upload succeeds.

One second later, another service asks:

"Does photos/cats.png exist?"

Most people expect the answer to be obvious.

Either the object exists or it doesn't.

But in a distributed system, different nodes may learn about the write at different times. That's where consistency guarantees start to matter.

Strong consistency: one answer, everywhere

With strong consistency, once a write is acknowledged, every future read must observe that write.

From a developer's perspective, life is simple.

Upload an object:

Upload object
Receive success response
Read object

If step three doesn't find the object, something is wrong.

The system behaves as though there is a single source of truth, even if dozens of machines are involved behind the scenes.

This is one reason developers like strongly consistent systems: they're easier to reason about.

The complexity stays inside the storage layer rather than leaking into application code.

Why storage systems historically liked eventual consistency

Eventual consistency often gets treated as a design mistake.

I don't think that's fair.

For years, many large-scale storage systems made that trade-off intentionally because coordinating every write across every node can become expensive.

The goal wasn't to be less correct.

The goal was to remain responsive and available at scale.

A classic example involved object listing.

You upload an object.

The upload succeeds.

You immediately perform a LIST operation.

The object isn't there.

The first time you see this behavior it feels like a bug.

In reality, nothing is broken. The metadata simply hasn't converged yet.

This kind of behavior existed in enough systems that many cloud engineers learned to expect it.

The trade-off is where the complexity lives

A lot of discussions frame consistency as:

Strong consistency = good

Eventual consistency = bad

In practice, the decision is rarely that simple.

Strong consistency often pushes complexity into the storage layer through coordination, consensus, and failure handling.

Eventual consistency often pushes complexity into application code through retries, stale reads, and edge cases.

The complexity doesn't disappear.

It just moves.

That's why consistency discussions are ultimately engineering discussions, not ideological ones.

The real question isn't:

"Which model is better?"

The real question is:

"Where do we want to pay the cost?"

Key takeaway

The most useful way I've found to think about consistency is this:

Consistency determines how quickly a distributed system agrees on the result of a change.

Strong consistency prioritizes agreement before responding.

Eventual consistency prioritizes progress and convergence over time.

Neither approach is free.

Every storage system chooses a different balance between coordination, performance, availability, and operational complexity.

Why Metadata Is the Hardest Part of Object Storage Systems

MaxHuo — Wed, 17 Jun 2026 13:30:00 +0000

In the previous post, I argued that object storage is best understood as a distributed key-value system rather than a traditional file system.

That raises an interesting question:

If storing objects is relatively straightforward, what makes distributed storage systems so difficult to build?

The answer is usually not the data itself.

It's the metadata.

What is metadata?

When you upload an object, the system stores more than just the object's contents.

For example:

Key: photos/cats.png
Size: 4.2 MB
Created: 2026-06-17
Version: v3
Checksum: abc123
Location: Node-7

All of this information is metadata.

Without metadata, the system would have no idea:

where the object lives
whether it exists
how large it is
which version is current
whether the object is healthy

In other words:

Data is the payload. Metadata is the map.

Storing data is surprisingly easy

Imagine a storage cluster with ten nodes.

When an object arrives, writing the data is often the simple part:

Choose a destination
Write the object
Replicate or encode it
Return success

Modern disks, SSDs, and networks are very good at moving bytes around.

The harder question is:

How does every node agree on where that object is?

That's a metadata problem.

The hidden database inside every storage system

Many people think object storage is primarily a storage problem.

In reality, every serious storage system contains a metadata system that behaves a lot like a database.

It must answer questions such as:

Does this object exist?
Which version is current?
Which nodes store the data?
Has replication completed?
Is the object being deleted?
Has the object become corrupted?

Every read and write depends on accurate answers.

A storage cluster can survive losing a disk.

It cannot survive losing track of its metadata.

Why metadata becomes difficult at scale

Imagine a system storing:

100 billion objects

The object data may be spread across hundreds of servers.

Metadata now faces several challenges:

Consistency

When a client uploads an object:

photos/cats.png

Every node must eventually agree that the object exists.

If one node thinks the upload succeeded while another thinks it failed, strange things happen:

objects disappear
stale versions appear
reads become inconsistent

Concurrency

Two clients may attempt to update the same object simultaneously.

Which version wins?

Who decides?

The metadata layer must coordinate those decisions.

Failure Recovery

Nodes fail.

Disks fail.

Networks fail.

During failures, metadata must remain correct.

Incorrect metadata is often worse than missing data because the system may confidently return the wrong answer.

Why listing objects is so expensive

Many developers assume:

List all objects in /photos/

should be simple.

In distributed storage, it often isn't.

The system may need to:

query multiple metadata partitions
aggregate results
sort keys
remove duplicates
handle concurrent updates

The actual object data may never be touched.

The entire operation is mostly metadata work.

The trade-off every storage system faces

Storage systems constantly balance three goals:

Performance

Fast reads and writes.

Consistency

Every client sees the same truth.

Availability

The system continues operating during failures.

Improving one often impacts another.

Many architectural decisions ultimately become metadata decisions.

Why metadata often determines system architecture

When engineers discuss storage systems, conversations frequently focus on:

SSDs
throughput
networking
replication

Those things matter.

But in many systems, the metadata architecture determines whether the entire platform succeeds.

Questions such as:

centralized or distributed metadata?
strong or eventual consistency?
database-backed or custom metadata engine?

can have a larger impact than raw storage performance.

Where systems like RustFS fit

While exploring object storage architectures, one lesson has become increasingly clear:

The most interesting engineering problems are often not about storing data.

They are about coordinating information about data.

Systems like RustFS, MinIO, Ceph, and others all solve the fundamental challenge differently:

How do we maintain an accurate view of billions of objects across many machines while failures are constantly happening?

That question sits at the heart of modern storage engineering.

Key takeaway

When people think about object storage, they usually focus on where the data lives.

Experienced storage engineers often focus on something else:

Metadata is the real system. The stored objects are just the payload.

Once you understand that idea, many storage architecture decisions start to make much more sense.

Next in this series

Part 3: Strong vs Eventual Consistency in Distributed Storage (Without the Confusion)

We'll look at why two healthy nodes can sometimes disagree, and how storage systems decide what "truth" actually means.

Most People Misunderstand Object Storage (Here’s the Mental Model That Actually Helps)

MaxHuo — Tue, 16 Jun 2026 12:42:00 +0000

If you’ve used S3, MinIO, or any cloud storage API, it’s easy to assume object storage is just a “cloud folder system.”

That assumption is wrong — and it leads to confusion when you start working with distributed systems.

Object storage is not a file system.

It’s closer to a distributed key-value system with strong durability guarantees and a very specific access model.

Once you understand that shift, a lot of cloud infrastructure starts to make more sense.

The mental model most people start with

When people first see object storage, they imagine something like this:

/photos/cats.png
/photos/dogs.png

A hierarchical file system:

folders
subfolders
files inside directories

This is how traditional systems like ext4 or NTFS work.

But object storage doesn’t actually work this way.

The actual model: key → object

Object storage is much simpler at its core:

key → value

Example:

key: photos/cats.png
value: <binary data>

There are no real folders.

“folders” are just string prefixes used for organization.

That’s it.

Why this design exists

This model isn’t accidental. It solves real distributed system problems.

Traditional file systems struggle when you try to:

scale across many machines
replicate data reliably
handle partial failures
coordinate metadata changes at scale

Object storage avoids many of these problems by simplifying the model.

Instead of supporting complex file operations, it focuses on:

store object
retrieve object
delete object
list objects by prefix

Nothing more.

The most important design choice: immutability

In most object storage systems:

Objects are not modified in place.

If you “update” a file, what actually happens is:

upload a new object
replace the key pointer
old object becomes orphaned (eventually cleaned up)

This is a huge shift from file systems.

Why this matters

Immutability makes distributed systems easier because:

no concurrent write conflicts on the same object
replication becomes simpler
caching becomes safer
failure recovery is easier to reason about

What object storage optimizes for

Object storage is not trying to be fast at small operations.

It is optimized for:

high durability (data should not be lost)
horizontal scalability
large objects (MBs → TBs)
simple access patterns

This is why it works well for:

backups
media storage
data lakes
AI datasets
logs and archives

Why listing objects feels “expensive”

One confusing thing for newcomers:

Why is listing objects slower than expected?

Because there is no real directory structure.

To list:

photos/

The system actually has to:

scan keys
match prefixes
aggregate results across storage nodes

This is a distributed query problem, not a simple filesystem lookup.

Where systems like RustFS fit in

While studying object storage systems, I’ve been looking at designs like RustFS, an open-source distributed object storage system built in Rust.

It becomes easier to understand such systems once you realize:

metadata is the hardest part, not storage
consistency decisions matter more than raw throughput
failure handling defines system behavior more than normal execution paths

We’ll go deeper into these ideas in the next parts.

Key takeaway

If you remember just one thing:

Object storage is not a folder system. It is a distributed key-value system optimized for durability and scale, not navigation and mutation.

Next in this series

In Part 2, I’ll break down:
“Why distributed storage systems need metadata engines (and why they are the hardest part)”