Your Storage Network Is Part of Your Storage System

In a traditional SAN architecture, storage networking is obviously critical.

In a hyperconverged Ceph cluster, engineers sometimes forget the same rule.

Imagine three Proxmox nodes.

Each has multiple fast NVMe drives.

The disks can generate several gigabytes per second.

Now connect the nodes using a slow or oversubscribed network.

The storage system is no longer limited by NVMe.

It is limited by Ethernet.

Ceph traffic includes more than guest I/O:

replication
recovery
backfill
rebalancing
client I/O

A node failure makes this even more interesting.

At exactly the moment VMs may need to restart elsewhere, Ceph may also be moving large amounts of data.

This is why storage traffic and latency-sensitive cluster communication such as Corosync should be carefully separated.

The broader engineering rule is simple:

Distributed storage performance is bounded by the network connecting the storage nodes.

And redundancy is not backup.

I used Proxmox + Ceph as a practical architecture example here:

https://sistro.net/proxmox-ceph-cuando-conviene-hci

Tags: proxmox ceph virtualization infrastructure