Every platform team eventually gets the ticket. It reads something like: "The Postgres pod won't schedule after the node rebooted, and now the whole staging environment is down." You dig in and find the database was pinned to a hostPath volume, the node got cordoned for a kernel patch, and nobody wrote down that the PVC was effectively single-node. That's the moment persistent storage stops being an app team's problem and becomes a platform decision.
This guide is for the engineer who has to make that decision and live with it for the next five years. By the end you'll understand how Longhorn and Rook-Ceph actually move bytes to disk, where each one breaks under pressure, and how to pick the one your team can operate at 3 AM — not the one that wins a synthetic benchmark on someone's blog.
It assumes you already run Kubernetes, you have opinions about StorageClass, and you're tired of the "just use NFS" crowd.
:::note[TL;DR]
- Longhorn is an opinionated, iSCSI-backed, replica-per-volume system built for operational simplicity. Great under ~50 nodes, painful above it.
- Rook-Ceph is the full RADOS/Ceph stack under a Kubernetes operator. More performance and more features, but you're adopting an ecosystem, not a CSI driver.
- Neither is "install and forget." Both fail in interesting ways, and both need off-cluster backups.
- Pick based on the team you have, not the benchmark you envy. :::
Prerequisites
You should have a working Kubernetes cluster (v1.27+ is a reasonable floor as of this writing), kubectl with cluster-admin, helm v3, and fio available for testing. For the S3 backup examples you'll need credentials to an object store. For Ceph, expect to provision raw block devices — loopback files will teach you nothing useful.
The Storage Problem Nobody Wants to Own
Stateful workloads on Kubernetes are still painful because the abstraction leaks. A PersistentVolumeClaim looks like a simple API object, but underneath it's a promise about durability, latency, and failure semantics that the API can't express. When that promise breaks, it breaks loudly and at scale.
There are three tiers of production storage, and they fit different problems:
- Cloud provider CSI — managed, expensive, and vendor-locked. Fine if you're all-in on one cloud and the bill is acceptable.
- Network filesystems (NFS, CIFS) — fine for shared config and read-mostly data, catastrophic for databases. If you're running Postgres on NFS, stop.
- Distributed storage you run yourself — Longhorn, Rook-Ceph, OpenEBS, Portworx. This is the tier that gives you control and takes your weekends.
"Distributed storage" means data is replicated across failure domains so a single disk or node loss doesn't lose data. What vendors pretend it means is "we put a CSI driver on top of a single server and called it distributed." Check where the replicas actually live before you trust the word.
A war story worth internalizing: a team I worked with ran their primary Postgres on a hostPath volume with a nodeAffinity rule. It worked for eleven months. Then a NIC failed on that node, the pod couldn't reschedule because the data was on a disk attached to a dead machine, and the recovery window was four hours of restoring from a nightly pg_dump. The lesson isn't "don't use hostPath" — it's that storage topology is a platform concern, and if you don't own it, you're borrowing someone else's outage.
Longhorn — The Opinionated Upstart
⚠️ TRUNCATED VERSION
This is an abbreviated cross-post. Full article (all config files, architecture diagrams, images): valtersit.com
🛠️ Partner Tools for Developers
ValtersIT curates partner deals for developers and sysadmins — VPS hosting, security tools, monitoring platforms and dev productivity gear. No filler, no affiliate spam.
Top comments (0)