If every volume on a container platform lives in one Ceph cluster, that cluster is one large failure domain. A bad upgrade, a lost MON quorum or a few full OSDs can take down every tenant in every zone at once. I built a proof of concept to test a different layout: one Kubernetes cluster across three zones, with a separate, independent Ceph cluster in each zone. Then I simulated losing a zone and recorded what happened.
TL;DR
- Design: one Kubernetes cluster over three zones, one independent Ceph cluster per zone, no replication between the Ceph clusters. Each PVC is created in the Ceph cluster of the zone its pod is scheduled into.
-
Mechanism:
volumeBindingMode: WaitForFirstConsumer, zone labels on nodes, and an unofficial multi-cluster fork of ceph-csi that lets one StorageClass map a zone to a cluster. - Functional result: placement, resize, snapshot/restore and per-zone isolation worked; 15 of 15 zone-aware tests passed.
-
Simulated zone outage: stateless pods with
tolerationSeconds: 30were running in another zone about 80–90 s after the stop command; with the default 300 s, eviction alone took about 6 minutes. PVC-backed pods stayedPendinguntil the zone returned, which is the expected behavior of this design. - Harness bugs: the first run exposed two problems in the lab setup (Docker reassigning node IPs, and stale kernel RBD mappings), not in ceph-csi. Both are fixed and shown below.
Read this first: what this PoC is and is not
- One VM. All six Kubernetes nodes (kind, v1.37.0) and all three Ceph clusters ran on a single virtual machine and shared one Linux kernel, one disk subsystem and one network stack.
- Logical zones. "Zones" are node labels (
topology.kubernetes.io/zone), not separate sites, racks or power domains. There is no real inter-zone latency.- Simulated outage. "Losing zone-b" means
docker stopon zone-b's two kind node containers. Zone-b's Ceph stayed up. There was no power cut, no network partition and no disk failure.- Lab timings. All times come from a shared host with default Kubernetes controller settings. They show the order of events and which settings dominate; they are not predictions for real hardware.
- Not benchmarked. No storage performance was measured. The comparison with stretched Ceph and RBD mirroring later in the post is qualitative and sourced from documentation.
Key takeaways
- With
WaitForFirstConsumerand a topology-aware CSI driver, one StorageClass can place each volume in its pod's zone across three independent Ceph clusters. - In this design, storage does not provide cross-zone failover. A PVC-backed pod waits for its zone. Availability for stateful apps has to come from application-level replication.
- For stateless pods,
tolerationSecondsmattered far more than node failure detection: about 30 s vs 300 s of extra waiting afterNotReady. - Recovering a zone cleanly needed two checks that no functional test covered: all etcd members healthy, and no stale kernel RBD mappings left from the dead node.
- Hard anti-affinity sized to the number of nodes (the CSI provisioner) leaves a
Pendingreplica during every zone outage. That is harmless, but it creates alert noise.
A short primer
-
ceph-csi is the CSI driver for Ceph. Its RBD driver creates block images in a Ceph pool and maps them on nodes with the kernel RBD client (
krbd). Upstream, a StorageClass points at oneclusterID. -
CSI topology lets a driver say where a volume can be used. Nodes report topology keys (here
topology.rbd.csi.ceph.com/zone), and the volume's PV gets anodeAffinitythat the scheduler respects. -
WaitForFirstConsumer(WFFC) delays provisioning until a pod using the PVC is scheduled, so the volume is created where the pod will run, not before. -
Taint-based eviction: when a node stops reporting, the control plane marks it
NotReady, addsnode.kubernetes.io/unreachabletaints, and evicts pods after theirtolerationSecondsexpire (300 s by default).
Architecture
-
Kubernetes: kind
v1.37.0, 3 control-plane and 3 worker nodes, one of each per zone, labeledtopology.kubernetes.io/zone=zone-a|b|c. Three control-plane nodes means three etcd members, one per zone. -
Ceph: three Ceph 19.2.7 (Squid) clusters, each with 3 MONs, 1 MGR and 3 OSDs, and a
kubernetespool withsize 3,min_size 2. Each has its own fsid and a cephx user restricted to that pool. Norbd-mirror, no peering. -
Proxmox (side test): three RBD storages (
ceph-zone-a/b/c), each pointing only at its own zone's MONs, with a separateproxmoxpool and cephx user. PVC images and Proxmox disks never appear in each other's pools.
The ceph-csi fork: version, build, install, and what it changes
Upstream ceph-csi does not yet route one StorageClass to several clusters by zone. The feature is requested in ceph/ceph-csi#5177, which is still open. I used an unofficial fork:
| Item | Value |
|---|---|
| Repository | github.com/akaitux/ceph-csi-multiaz |
| Branch / commit |
v3.16.2-multiaz, 15b2ae4
|
| Relation to upstream | one commit ("multiaz") on top of upstream v3.16.2
|
| Build |
git clone --depth 5 -b v3.16.2-multiaz, then go build with Go 1.25.0 (-mod=vendor, CGO, -tags=<ceph-release>,ceph_preview); builder and runtime base image quay.io/cephcsi/cephcsi:v3.18.1; result tagged localhost/cephcsi-multiaz:v3.16.2-multiaz; --version reports v3.16.2-multiaz and commit 15b2ae4…
|
| Install | plain manifests from the fork's deploy/rbd/kubernetes/, not Helm, adapted with sed: own namespace, image replaced, csi-provisioner set to v5.3.0 and csi-resizer to v1.14.0 (the manifests reference v6.0.0 and v2.0.0), --feature-gates=CSIVolumeGroupSnapshot=true removed, --domainlabels=topology.kubernetes.io/zone enabled on the node plugin |
| Other sidecars | csi-snapshotter v8.4.0, csi-attacher v4.10.0, node-driver-registrar v2.15.0, snapshot-controller v8.6.0 |
What the fork commit changes, from reading its diff against v3.16.2:
-
New StorageClass parameter
clusterTopologyConfigMap.clusterIDbecomes optional. The ConfigMap lists clusters with their zones, MONs, pool and Secret name. -
Cluster selection in
CreateVolume(internal/util/topology.go,FindClusterAndTopology). The driver walks the request'sPreferredtopologies first, thenRequisite, and picks the first cluster whosezonescontain the requested zone. -
Per-cluster credentials. If the request carries no secrets, the controller reads the selected cluster's Secret (
secretName) from its own namespace, and recordsclusterSecretNamein the volume context. -
Accessible topology in the response. The created volume carries
accessible_topologywith the zone segment, and the volume context getsceph.csi/zone. - Snapshots and clones stay in the source cluster. When restoring from a snapshot, the topology requirement is pinned to the snapshot's cluster (code comment: "so the new PV lands in the correct AZ").
- Related changes in the CephFS controller, group snapshots, a helper to find a PV by volume handle, and the Helm chart values.
The cluster map used in the PoC (addresses redacted):
{"clusterTopology":[
{"clusterID":"ceph-zone-a","monitors":"<mon-a1>,<mon-a2>,<mon-a3>","zones":["zone-a"],"pool":"kubernetes","secretName":"csi-rbd-secret-zone-a"},
{"clusterID":"ceph-zone-b","monitors":"<mon-b1>,<mon-b2>,<mon-b3>","zones":["zone-b"],"pool":"kubernetes","secretName":"csi-rbd-secret-zone-b"},
{"clusterID":"ceph-zone-c","monitors":"<mon-c1>,<mon-c2>,<mon-c3>","zones":["zone-c"],"pool":"kubernetes","secretName":"csi-rbd-secret-zone-c"}
]}
And the single StorageClass:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: ceph-rbd-zonal
annotations: {storageclass.kubernetes.io/is-default-class: "true"}
provisioner: rbd.csi.ceph.com
parameters:
clusterTopologyConfigMap: ceph-cluster-topology
imageFeatures: layering
csi.storage.k8s.io/fstype: ext4
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
How a PVC reaches the right Ceph cluster
Step by step, checked against the external-provisioner v5.3.0 topology documentation and the fork source:
-
WFFC holds the PVC. The PVC stays
Pending, and no image exists yet. -
The scheduler picks a node for the pod (spread constraints, affinity, resources) and records it on the PVC (
volume.kubernetes.io/selected-node). -
external-provisioner builds
accessibility_requirements. The Topology feature is on by default since v5.0. With delayed binding, noallowedTopologiesand no strict topology,Requisiteis the set of topology values of all nodes that run the driver (taken from theirCSINodekeys), andPreferredis the same set with the selected node's zone placed first. -
The fork maps zone to cluster.
FindClusterAndTopologychecksPreferredfirst, so the selected node's zone wins, matches it to the ConfigMap entry, loads that cluster's Secret, and creates the image in that cluster's pool. -
The driver returns
accessible_topology(topology.rbd.csi.ceph.com/zone=zone-X). external-provisioner turns it into the PV'snodeAffinity. -
From then on the PV is pinned. Any pod using the PVC can only be scheduled into that zone. The proof test read the PV's
nodeAffinityvalue back and compared it with the pod's node zone and theclusterID.
Spreading a StatefulSet across zones then needs one block:
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector: {matchLabels: {app: web-sts}}
Functional checks
| Check | Result |
|---|---|
| 3-replica StatefulSet spread over zones | one replica per zone; rbd ls on each cluster showed only that zone's image; PV nodeAffinity zone = pod zone |
| Data survives pod delete and recreate | new pod (new UID), same zone, same data |
| Online resize 1Gi → 3Gi | about 7 s, no pod restart (df: 973.4M → 2.9G) |
| Snapshot and restore | restored volume in the same zone, identical contents |
| PVC plus pod ready | 6.0 s |
| Zone-aware bats suite | 15 / 15 passed |
I also stopped all three MONs of zone-c's Ceph for about 6 min 22 s. In zone-a, a new PVC bound and its pod ran. A new pod in zone-c stayed Pending with FailedMount and was not placed in another zone. An existing zone-c pod kept writing through its established kernel RBD session (the OSDs were still up). Everything recovered when the MONs returned.
The simulated zone outage
Setup
-
What was stopped: both kind node containers of zone-b (control plane and worker), with
docker stop -t 10. Zone-b's Ceph stayed up. -
Workloads:
nginxandwhoamiDeployments, ars-exampleReplicaSet, anode-agentDaemonSet, a 3-replicaweb-stsStatefulSet (one PVC per zone), a single-replica Postgres (in zone-c, not affected), aproof/zonalStatefulSet, and thecsi-rbdplugin-provisioner(3 replicas, required hostname anti-affinity, as in the upstream manifest). -
Tolerations:
nginx,whoami, Postgres andweb-stswere patched totolerationSeconds: 30. The provisioner,rs-exampleandproof/zonalkept the default 300 s, so one run shows both behaviors. - Driver and safety: a small lab dashboard called a restricted helper that can only stop or start one zone's nodes, and refuses to stop a second control-plane node while one is already down (three etcd members tolerate losing one).
-
Measurement: the dashboard recorded a per-pod and per-node timeline of state changes, and a driver script polled it every 15 s and collected Kubernetes events. The dashboard's own polling interval is not in the logs, so treat timeline values as accurate to a few seconds; events have 1 s resolution. t = 0 is when the stop command was issued.
docker stopreturned 11.2 s (run 1) and 11.3 s (run 2) later.
tolerations:
- key: node.kubernetes.io/unreachable
operator: Exists
effect: NoExecute
tolerationSeconds: 30
- key: node.kubernetes.io/not-ready
operator: Exists
effect: NoExecute
tolerationSeconds: 30
Summary of runs 1 and 2
Each run tracked the same eight pods on zone-b's nodes: 4 stateless (nginx, whoami, rs-example in run 1 or a second nginx replica in run 2, and one provisioner replica), 2 PVC-backed (web-sts-1, proof/zonal-1) and 2 DaemonSet pods.
| Event (seconds after t = 0) | Run 1 (helper v1) | Run 2 (helper v2) |
|---|---|---|
Worker NotReady + unreachable taint |
+47 | +46 |
Control-plane NotReady
|
+47 | +50 (taint +52) |
Pods with tolerationSeconds=30 evicted |
+85 | +76 |
Their stateless replacements Running in a surviving zone |
+87 to +89 | +78 to +80 |
| Pods with the default 300 s evicted | +355 | +347 |
| PVC-backed pods |
Pending, PV pinned to zone-b |
same |
| Restore job duration | 1.2 s | 61.2 s |
PVC-backed pods Running again |
not on their own; manual repair | +694 (14 s after the job returned) |
Two runs on one shared host are not a statistical sample. Read the numbers as "what happened here", not as expected values.
Run 1 in detail (helper v1)
| Time | Observation |
|---|---|
| +15 to +20 s |
LeaderElection events: three leader-election leases of the provisioner sidecars moved to replicas outside zone-b, which means the zone-b replica had stopped renewing them. |
| +44 s |
NodeNotReady warnings for pods on zone-b nodes |
| +47 s | both zone-b nodes NotReady and tainted unreachable
|
| +85 s |
nginx, whoami and web-sts-1 evicted; nginx Running in zone-c at +87 s, whoami at +89 s; web-sts-1 waiting, PV pinned to zone-b |
| +354 s |
FailedScheduling for the provisioner replacement: 0/6 nodes are available: 2 node(s) didn't match pod anti-affinity rules, 4 node(s) had untolerated taint(s)
|
| +355 s |
rs-example, proof/zonal-1 and the provisioner replica evicted; rs-example Running in zone-c at +364 s |
| +619 s | restore job finished (1.2 s); worker Ready at +624 s, control plane at +628 s |
| +622 s |
FailedMount for web-sts-1: attacher.MountDevice failed to create newCsiDriverClient (the node plugin was not registered yet) |
| +626 to +631 s | DaemonSet pods and the provisioner replica Running again; with the stateless pods already replaced, 6 of 8 tracked pods counted as recovered |
| +655 s |
BackOff restarting the csi-rbdplugin, driver-registrar and liveness-prometheus containers of the zone-b node plugin |
| until the watch ended, about 10 min after the restore |
web-sts-1 and proof/zonal-1 did not recover on their own |
Root causes, found afterwards: Docker gave the two zone-b containers each other's IP addresses on restart, which left the zone-b etcd member unhealthy (2 of 3), and two kernel RBD devices were stuck (details below). The manual repair took several attempts: containers that did not stop cleanly (did not receive an exit event), Address already in use on reconnect, failing runc exec, and I/O errors while force-removing the RBD devices. It ended with the stale devices removed, the IPs back in place, 6 of 6 nodes Ready, etcd 3 of 3 healthy, and all PVCs bound to their original zone clusters.
Run 2 in detail (helper v2: IP pinning and krbd reaper)
| Time | Observation |
|---|---|
| +46 s | worker NotReady and tainted; control plane NotReady at +50 s, taint at +52 s |
| +76 s | both nginx replicas and whoami evicted; replacements Running at +78 s (nginx: one in zone-c, one in zone-a) and +80 s (whoami, zone-c). The nginx Service had ready endpoints in zones a and c throughout. |
| +345 s |
TaintManagerEviction and FailedScheduling for the provisioner replacement (same anti-affinity message as run 1) |
| +347 s |
proof/zonal-1 and the provisioner replica evicted; proof/zonal-1 waiting |
| about +618 s | restore job started; the reaper found the stale devices, refused one orphan network namespace because it had other interfaces, revived the other, and removed 2 of 2 devices in 2 s |
| +624 s |
Starting kubelet on both zone-b nodes; worker NodeReady at +625 s, control plane at +631 s |
| +626 s |
FailedMount for web-sts-1 (newCsiDriverClient, same as run 1) |
| +627 s | provisioner replacement scheduled back to the zone-b worker, Running at +630 s |
| +659 s |
BackOff on the zone-b node plugin containers; they started again at +671 s |
| +680 s | restore job returned (61.2 s in total) |
| +690 s | the StatefulSet created a new web-sts-1, scheduled to the zone-b worker; started at +694 s; proof/zonal-1 Running at +694 s, data intact |
The job log does not break down the 61 s. The reaper reported 2 s, and the kubelets started about 6 s into the job, so most of the time came after docker start. The helper then retries mount -o remount,rw /sys inside each node, up to 20 times with a 1 s pause; that loop is the likely but unconfirmed cause. The PVC-backed pods were back about 70 s after the worker reported Ready. Most of that gap overlapped the node plugin's restart back-off.
Run 3 (quick check of helper v2)
A short down/up cycle (about 71 s between the stop and the start command) to exercise the reaper on a fresh orphan namespace. The reaper revived one orphan namespace, waited 64 s for pending I/O to drain, and removed 2 devices; the whole up took 65.5 s. Both nodes came back with their original IPs.
Why these timings would differ on real hardware
-
Node failure detection is a controller setting. kube-controller-manager marks a node unhealthy after
--node-monitor-grace-period(default 50 s) and checks every--node-monitor-period(default 5 s) (reference). The kind config set neither, so the observed +46 to +50 s are consistent with the default. Since t = 0 is the stop command, not the last heartbeat, this is an explanation of the numbers, not a separate measurement. -
Eviction is
NotReadyplustolerationSeconds. The DefaultTolerationSeconds admission plugin adds 300 s tolerations fornot-readyandunreachableto pods that do not set them (taint-based evictions). Run 1's 30 s pods were evicted at +85 s, about 8 s later than +47 plus 30. - Replacement start time depends on the host. Images were already present and the nodes shared one kernel and disk. On real nodes, image pulls, CNI and real network latency add time.
- A real outage looks different. A power loss or network partition does not send SIGTERM. A partition can also leave the "dead" zone running and writing, which this test never exercised.
- One shared VM means the surviving zones also carried the lab's other load. Neither run was isolated from the other.
Problems found and how they were fixed
1. Docker swapped node IPs and etcd became unhealthy
When zone-b's containers restarted, Docker assigned them each other's addresses. kind rewrote its manifests for the new IPs, but the etcd member's peer URL and certificates still used the old one: TLS SAN mismatch, member unhealthy, cluster on 2 of 3. Another failure would have cost quorum.
Fix: on down, record each node's IP and pin it with docker network connect --ip while the container is stopped. After runs 2 and 3, etcd was 3 of 3 healthy and the addresses did not change. The relevant part of the helper (names generalized):
NET=lab
ipof(){ docker inspect -f "{{(index .NetworkSettings.Networks \"$NET\").IPAddress}}" $1 2>/dev/null; }
pin(){ # $1 container (stopped) $2 ip
cur=$(docker inspect -f "{{with (index .NetworkSettings.Networks \"$NET\").IPAMConfig}}{{.IPv4Address}}{{end}}" $1 2>/dev/null)
[ "$cur" = "$2" ] && return 0
docker network disconnect $NET $1 && docker network connect --ip $2 $NET $1 && echo "$1 pinned to $2"; }
# down: refuse if another control plane is already stopped, record IPs, stop, pin
# up: pin, run the krbd reaper, docker start, remount /sys rw for krbd
Full zone-ctl helper (v2, sanitized)
#!/bin/bash
# SSH forced command used by the lab dashboard; never a shell. v2: IP pinning + krbd reap.
set -uo pipefail
read -r op z extra <<<"${SSH_ORIGINAL_COMMAND:-}"
[ -z "${extra:-}" ] || { echo "too many args"; exit 2; }
case "${z:-}" in a) N="lab-control-plane lab-worker";; b) N="lab-control-plane2 lab-worker2";; c) N="lab-control-plane3 lab-worker3";; *) echo "zone must be a|b|c"; exit 2;; esac
logger -t zone-ctl "request: $op $z from ${SSH_CLIENT%% *}"
S=/var/lib/zone-ctl; NET=lab
ipof(){ docker inspect -f "{{(index .NetworkSettings.Networks \"$NET\").IPAddress}}" $1 2>/dev/null; }
pin(){ # $1 container (stopped) $2 ip
cur=$(docker inspect -f "{{with (index .NetworkSettings.Networks \"$NET\").IPAMConfig}}{{.IPv4Address}}{{end}}" $1 2>/dev/null)
[ "$cur" = "$2" ] && return 0
docker network disconnect $NET $1 && docker network connect --ip $2 $NET $1 && echo "$1 pinned to $2"; }
case "${op:-}" in
down)
for cp in lab-control-plane lab-control-plane2 lab-control-plane3; do
case " $N " in *" $cp "*) continue;; esac
[ "$(docker inspect -f '{{.State.Running}}' $cp 2>/dev/null)" = true ] || { echo "refused: $cp is already down (etcd quorum would be lost)"; exit 3; }
done
for n in $N; do i=$(ipof $n); [ -n "$i" ] && [ "$i" != "invalid IP" ] && echo "$i" > $S/$n.ip; done
docker stop -t 10 $N
for n in $N; do [ -s $S/$n.ip ] && pin $n "$(cat $S/$n.ip)"; done ;;
up)
IPS=""; for n in $N; do [ -s $S/$n.ip ] && { pin $n "$(cat $S/$n.ip)"; IPS="$IPS $(cat $S/$n.ip)"; }; done
[ -n "$IPS" ] && /usr/local/lib/zone-ctl/krbd-reap.py $IPS
docker start $N
for n in $N; do for i in $(seq 1 20); do docker exec $n mount -o remount,rw /sys 2>/dev/null && break; sleep 1; done; done
echo "/sys remounted rw (krbd)" ;;
status) ;;
*) echo "op must be down|up|status"; exit 2 ;;
esac
docker inspect -f '{{.Name}} {{.State.Status}} {{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' $N
2. Stale kernel RBD mappings
kind nodes are containers on the host kernel, so /dev/rbdN is a host kernel device. When the zone-b worker stopped, the libceph client of two RBD devices stayed in that node's network namespace, which no longer had any interfaces. Reconnects failed with ENETUNREACH. After restart, the node plugin reused those devices, NodeStage hung with blkid in D state, and a forced rbd remove hung too.
Fix: before starting the nodes, a reaper finds RBD devices whose client address belongs to a stopped node, refuses to act if a running container owns that address, finds orphan network namespaces (never the host namespace or one with other interfaces), gives each a temporary veth on the lab bridge so pending I/O can drain, force-removes the devices, and removes the veths. In run 2 this took 2 s; in run 3, 64 s.
krbd-reap.py (sanitized: bridge name and address range are placeholders)
#!/usr/bin/env python3
"""Reap krbd mappings whose libceph client lives in the netns of a STOPPED kind node. usage: krbd-reap.py <node-ip>..."""
import fcntl, glob, os, socket, struct, subprocess, sys, time
IPS = set(sys.argv[1:])
BRIDGE = "br-lab" # the docker bridge the kind nodes are attached to
TMP_NET = "192.0.2" # placeholder: a free address range on that bridge
def sh(*c, ok=False):
r = subprocess.run(c, capture_output=True, text=True)
if r.returncode and not ok: raise RuntimeError(f"{' '.join(c)}: {r.stderr.strip()}")
return r.stdout
T0 = time.time()
def say(*m): print(f"krbd[+{int(time.time() - T0)}s]:", *m, flush=True)
def pending(): return [l for l in sh("ps", "-eo", "pid,args").splitlines() if "remove_single_major" in l and "sh -c" in l and "timeout" not in l]
devs = {os.path.basename(d): open(d + "/name").read().strip() for d in glob.glob("/sys/bus/rbd/devices/*")
if open(d + "/client_addr").read().split(":")[0] in IPS}
if not devs and not pending(): say("no stale mappings"); sys.exit(0)
live = sh("docker", "ps", "-q").split()
busy = set(sh("docker", "inspect", "-f", "{{range .NetworkSettings.Networks}}{{.IPAddress}} {{end}}", *live).split()) if live else set()
if IPS & busy: say("refused: a RUNNING container owns", " ".join(sorted(IPS & busy)), "- mappings are live, not stale"); sys.exit(2)
say("stale mappings from", " ".join(sorted(IPS)) + ":", " ".join(f"rbd{k}={v}" for k, v in devs.items()) or "-", "| pending removes:", len(pending()))
used = set(int(x) for x in __import__("re").findall(r"link-netnsid (\d+)", sh("ip", "-o", "-d", "link", "show")))
named = set()
os.makedirs("/run/netns", exist_ok=True)
for f in glob.glob("/run/docker/netns/*") + ["/proc/1/ns/net"]: # live docker netns + the host netns itself
n = "czc-" + (os.path.basename(f) if f.startswith("/run") else "host"); p = "/run/netns/" + n; open(p, "a").close(); sh("mount", "--bind", f, p, ok=True)
for l in sh("ip", "netns", "list-id").splitlines():
if "iproute2 netns name" in l: named.add(int(l.split()[1]))
allids = [int(l.split()[1]) for l in sh("ip", "netns", "list-id").splitlines() if l.startswith("nsid")]
for p in glob.glob("/run/netns/czc-*"): sh("umount", p, ok=True); os.unlink(p)
orphans = [i for i in allids if i not in used and i not in named]
say("orphan netns ids:", orphans or "none")
def a(t, p): l = 4 + len(p); return struct.pack("HH", l, t) + p + b"\0" * ((4 - l % 4) % 4)
def s(x): return x.encode() + b"\0"
ifi = lambda idx=0, fl=0, ch=0: struct.pack("BxHiII", 0, 0, idx, fl, ch)
def nl(t, flags, body):
sk = socket.socket(socket.AF_NETLINK, socket.SOCK_RAW, 0); sk.bind((0, 0))
sk.send(struct.pack("IHHII", 16 + len(body), t, flags, 1, 0) + body); d = sk.recv(65536); sk.close()
if struct.unpack_from("H", d, 4)[0] == 2:
e = struct.unpack_from("i", d, 16)[0]
if e: raise OSError(-e, os.strerror(-e))
revived = []
for k, nsid in enumerate(orphans):
hv, handle = f"vreap{nsid}", f"/run/netns/reap{nsid}"
try:
peer = ifi() + a(3, s(hv)) + a(19, struct.pack("I", os.getpid()))
nl(16, 1 | 4 | 0x400 | 0x200, ifi(0, 1, 1) + a(46, struct.pack("i", nsid)) + a(3, s("eth9")) +
a(18, a(1, s("veth")) + a(2, a(1, peer)))) # veth eth9 (UP) inside, peer in host
fd = os.open("/dev/net/tun", os.O_RDWR); tap = f"tapreap{nsid}"
fcntl.ioctl(fd, 0x400454ca, struct.pack("16sH", tap.encode(), 0x0002 | 0x1000))
nl(16, 1 | 4, ifi(socket.if_nametoindex(tap)) + a(46, struct.pack("i", nsid))) # move tap into it
nsfd = fcntl.ioctl(fd, 0x54e3) # TUNGETDEVNETNS -> netns fd
open(handle, "a").close(); sh("mount", "--bind", f"/proc/{os.getpid()}/fd/{nsfd}", handle)
os.close(nsfd); os.close(fd)
links = [l.split(":")[1].strip().split("@")[0] for l in sh("ip", "netns", "exec", f"reap{nsid}", "ip", "-o", "link").splitlines()]
if not set(links) <= {"lo", "eth9", tap}: raise RuntimeError(f"nsid {nsid} has other links {links}; not touching it")
ip = f"{TMP_NET}.{20 + k}"
sh("ip", "netns", "exec", f"reap{nsid}", "ip", "link", "set", "lo", "up")
sh("ip", "netns", "exec", f"reap{nsid}", "ip", "addr", "add", ip + "/24", "dev", "eth9")
sh("ip", "link", "set", hv, "master", BRIDGE); sh("ip", "link", "set", hv, "up")
say(f"nsid {nsid} revived via {hv} {ip}"); revived.append((hv, handle))
except Exception as e:
say(f"nsid {nsid}: {e}")
sh("ip", "link", "del", hv, ok=True)
if os.path.exists(handle): sh("umount", handle, ok=True); os.unlink(handle)
if not revived: say("WARNING no orphan netns revived; not force-removing (would hang)"); sys.exit(1)
procs = [subprocess.Popen(["timeout", "150", "sh", "-c", f"echo '{k} force' > /sys/bus/rbd/remove_single_major"]) for k in devs]
t1 = time.time() # removal is done when the remove writers have returned (sysfs entries vanish before I/O drains)
while time.time() - t1 < 150 and (any(p.poll() is None for p in procs) or pending()): time.sleep(2)
left = [k for k in devs if os.path.exists(f"/sys/bus/rbd/devices/{k}")] + pending()
say(f"removed {len(devs)} device(s), drained in {int(time.time() - t1)}s" + (f", LEFT {left}" if left else ""))
for hv, handle in revived:
sh("ip", "link", "del", hv, ok=True); sh("umount", handle, ok=True); os.unlink(handle)
say("temporary veths/netns handles removed")
sys.exit(1 if left else 0)
The exact failure is specific to kind, but the general case is not: a node that loses its network with live krbd mappings can come back with stuck devices. Check for them before returning a node to service.
3. Hard anti-affinity and a lost zone
The provisioner runs 3 replicas with required hostname anti-affinity. With two workers left, the third replica stayed Pending for the rest of the outage. Its sidecars run with leader election (as in the upstream manifest), and the leases moved to other replicas within about 20 s in run 1, so provisioning was not affected. If you alert on Pending pods in system namespaces, every zone outage will trigger that alert. Options: two replicas, or zone-based preferred anti-affinity.
4. tolerationSeconds sets the stateless recovery time
Failure detection was stable at about 46–50 s. The setting that changed the outcome was tolerationSeconds: 30 s led to replacements at about +80–90 s, the default 300 s to about +350 s. A lower value also means pod churn on short network interruptions, so set it per workload class.
5. DoNotSchedule did not block failover (inference)
nginx uses a hard spread constraint (whenUnsatisfiable: DoNotSchedule) and nodeTaintsPolicy: Honor, yet its replacements were scheduled into zones a and c. According to the Kubernetes documentation, with Honor the skew calculation only includes untainted nodes and tainted nodes the pod tolerates; with the default (Ignore), all nodes count. The unreachable node also carries a NoSchedule taint that the default tolerations do not cover, and the FailedScheduling message above ("4 node(s) had untolerated taint(s)", which counts the three control-plane nodes plus the zone-b worker) shows that the scheduler saw that taint.
This explanation is an inference from the manifest, the documentation and the outcome. What would confirm it:
- an A/B rerun with
nodeTaintsPolicyunset: if the explanation is right, the replacement should stayPendingwith a topology spread message; - scheduler logs at higher verbosity showing which nodes enter the PodTopologySpread calculation.
Stateful workloads: what to recommend
In this design storage does not move between zones, so a PVC-backed pod is only as available as its zone. That is a deliberate trade-off for a smaller blast radius. The platform decision is how much of the remaining work it takes on.
Recommended order for a platform team, with the conditions for each:
-
Default: recommended patterns with application-level replication. Ship blueprints for a StatefulSet with one replica per zone (the spread block above), each on its own zone's Ceph, and replication done by the application: PostgreSQL streaming replication, MongoDB replica sets, Kafka or etcd quorums. Choose this when the data store has mature built-in replication and the owning team can operate it. The outage test showed the shape:
web-sts-0andweb-sts-2in zones a and c kept running, and onlyweb-sts-1waited. - Next: operator-managed data services. The platform runs operators and owns failover, backups and replica rebuilds. Choose this when many tenant teams need databases but lack the skills to run them, and the platform team can take database on-call and SLAs. Start with one or two engines.
- Only for tolerant workloads: document the limitation. State clearly that a PVC is pinned to one zone and that a single-replica stateful app is down for as long as its zone is. Acceptable for dev/test, caches, rebuildable data, or apps whose recovery objective is longer than a typical zone outage.
Two cases where per-zone Ceph is the wrong tool: a single-instance application that cannot replicate and must survive a zone loss with no data loss (consider a stretched cluster, if the sites qualify), and disaster recovery to another site for such an application where some data loss is acceptable (consider RBD mirroring).
Moving one volume to another zone is an explicit copy, because the clusters are not connected. On the Proxmox side, an offline move of a container's rootfs (about 517 MB of data) from zone-a to zone-b took 105.5 s and 99.5 s back, with the container stopped throughout. That is a migration with downtime.
Comparison: per-zone Ceph, stretched Ceph, RBD mirroring
Not benchmarked here. This table summarizes the documentation and the design properties; none of it was measured in this PoC.
| Property | Independent Ceph per zone (this PoC) | Stretched Ceph cluster | RBD mirroring between clusters |
|---|---|---|---|
| Write path | local to one zone | synchronous across zones: each write is stored in both zones before it is acknowledged (stretch mode) | local writes, asynchronous copy (RBD mirroring); journal-based mode writes every change twice, and the docs say to "expect write latencies to nearly double" |
| Data loss on losing a zone (storage layer) | volumes of that zone are unavailable until it returns | none expected; the surviving zone serves I/O in degraded mode | up to the replication lag or snapshot interval; partially applied snapshot deltas are rolled back on failover |
| Volume usable in another zone | no | yes | after failover (image promotion on the peer cluster) |
| Failure domain | one zone | one cluster spanning sites: one monitor quorum, one version, one upgrade path | separate clusters |
| Topology constraints | none between zones | stretch mode supports exactly two data zones plus a tiebreaker monitor at a third site | peer clusters configured per pool; one-way or two-way |
| Kubernetes integration | topology-aware CSI (fork) | standard ceph-csi, one cluster | ceph-csi plus csi-addons VolumeReplication for promote/demote |
| Operational cost | three clusters to run; no cross-zone data path | sensitive to inter-site latency; tiebreaker placement | mirror daemons, failover and failback procedures; data copied twice if the app already replicates |
Benchmark plan (future work)
I did not run fio on this PoC. All three "zones" share one VM and the same disks, so the results would say nothing about cross-zone behavior. On real per-zone hardware, the plan is:
-
Baseline per zone: fio on an RBD PVC in each zone with
--direct=1 --ioengine=libaio --time_based --runtime=120, three repetitions each:- 4k random write, iodepth 1: p50 and p99 completion latency;
- 4k random read, iodepth 32: IOPS;
- 1M sequential write: throughput.
- Network baseline: RTT between zones (ping and a TCP test) recorded with each run.
- Comparisons: the same jobs against a stretched pool (if two qualifying sites exist) and against an image with journal-based mirroring enabled, to measure the cross-zone and journaling cost directly.
- Under failure: repeat the 4k write test during a MON outage and during a zone outage to see stalls, not only averages.
Next steps
The thresholds below are proposed acceptance criteria, not results.
-
Repeat the outage test on real hardware with separate nodes per zone, using three failure types: hard power-off, network partition between zones, and a graceful drain. Run each at least five times. Record
NotReadytime, eviction time, replacementReadytime and PVC pod recovery time after the zone returns, and report median and maximum. -
Zone-recovery gate. A zone counts as recovered only when etcd is 3 of 3 healthy, no
/sys/bus/rbd/devices/*/client_addrpoints at a stopped node, there are no processes in D state on the returned nodes, and every PVC-backed pod isRunning. Target: no manual steps in 5 of 5 runs. - Tolerations per workload class. Test 30, 60 and 120 s against short network flaps (for example 10 s and 20 s link drops). Target: no evictions for flaps shorter than the chosen value, failover within value plus 60 s otherwise.
-
System workloads. Change the provisioner to two replicas or zone-based preferred anti-affinity. Target: no
Pendingsystem pods during a single-zone outage. -
Confirm the
nodeTaintsPolicyinference with the A/B rerun described above. - CSI strategy. Track ceph/ceph-csi#5177. While on the fork, rebase onto each upstream release, and gate upgrades on the 15 zone-aware tests plus snapshot/restore and resize. The fallback is one StorageClass per zone with upstream ceph-csi.
- Production-style Ceph with standby MGRs, real disks and capacity sized for real load, then run the benchmark plan.
-
Alerting. Alert when a PVC-backed pod is
Pendingbecause of a volume node affinity conflict for longer than a chosen time (for example 5 minutes), and silence the expected provisioner replica during a declared zone outage.
Discussion
If you run Ceph under a multi-zone Kubernetes cluster: did you choose a stretched cluster, RBD mirroring, or independent per-zone clusters? And what tolerationSeconds do you use in production? I would like to compare notes in the comments.
Ramin Najarbashi




Top comments (1)
This experiment makes the storage failure domain argument concrete. Separating Ceph per zone with WaitForFirstConsumer and the multi-AZ ceph-csi fork is a clear design, and the measured contrast is striking: stateless pods returned in about 80 to 90 seconds with tolerationSeconds set to 30, against about six minutes with the default 300 seconds, while PVC backed pods stayed Pending by design and returned about 14 seconds after recovery in run 2. The two outage only bugs also stand out, especially the Docker IP swap that broke etcd and the orphaned kernel RBD mappings that blocked recovery in run 1. The honesty box about the single VM proof of concept keeps the results in proper context.