DEV Community

Cover image for Zone-Aware Storage on Kubernetes with Three Independent Ceph Clusters: A kind-Based PoC and a Simulated Zone Outage
Ramin Najjarbashi
Ramin Najjarbashi

Posted on Edited on

Zone-Aware Storage on Kubernetes with Three Independent Ceph Clusters: A kind-Based PoC and a Simulated Zone Outage

If every volume on a container platform lives in one Ceph cluster, that cluster is one large failure domain. A bad upgrade, a lost MON quorum or a few full OSDs can take down every tenant in every zone at once. I built a proof of concept to test a different layout: one Kubernetes cluster across three zones, with a separate, independent Ceph cluster in each zone. Then I simulated losing a zone and recorded what happened.

TL;DR

  • Design: one Kubernetes cluster over three zones, one independent Ceph cluster per zone, no replication between the Ceph clusters. Each PVC is created in the Ceph cluster of the zone its pod is scheduled into.
  • Mechanism: volumeBindingMode: WaitForFirstConsumer, zone labels on nodes, and an unofficial multi-cluster fork of ceph-csi that lets one StorageClass map a zone to a cluster.
  • Functional result: placement, resize, snapshot/restore and per-zone isolation worked; 15 of 15 zone-aware tests passed.
  • Simulated zone outage: stateless pods with tolerationSeconds: 30 were running in another zone about 80–90 s after the stop command; with the default 300 s, eviction alone took about 6 minutes. PVC-backed pods stayed Pending until the zone returned, which is the expected behavior of this design.
  • Harness bugs: the first run exposed two problems in the lab setup (Docker reassigning node IPs, and stale kernel RBD mappings), not in ceph-csi. Both are fixed and shown below.

Read this first: what this PoC is and is not

  • One VM. All six Kubernetes nodes (kind, v1.37.0) and all three Ceph clusters ran on a single virtual machine and shared one Linux kernel, one disk subsystem and one network stack.
  • Logical zones. "Zones" are node labels (topology.kubernetes.io/zone), not separate sites, racks or power domains. There is no real inter-zone latency.
  • Simulated outage. "Losing zone-b" means docker stop on zone-b's two kind node containers. Zone-b's Ceph stayed up. There was no power cut, no network partition and no disk failure.
  • Lab timings. All times come from a shared host with default Kubernetes controller settings. They show the order of events and which settings dominate; they are not predictions for real hardware.
  • Not benchmarked. No storage performance was measured. The comparison with stretched Ceph and RBD mirroring later in the post is qualitative and sourced from documentation.

Key takeaways

  1. With WaitForFirstConsumer and a topology-aware CSI driver, one StorageClass can place each volume in its pod's zone across three independent Ceph clusters.
  2. In this design, storage does not provide cross-zone failover. A PVC-backed pod waits for its zone. Availability for stateful apps has to come from application-level replication.
  3. For stateless pods, tolerationSeconds mattered far more than node failure detection: about 30 s vs 300 s of extra waiting after NotReady.
  4. Recovering a zone cleanly needed two checks that no functional test covered: all etcd members healthy, and no stale kernel RBD mappings left from the dead node.
  5. Hard anti-affinity sized to the number of nodes (the CSI provisioner) leaves a Pending replica during every zone outage. That is harmless, but it creates alert noise.

A short primer

  • ceph-csi is the CSI driver for Ceph. Its RBD driver creates block images in a Ceph pool and maps them on nodes with the kernel RBD client (krbd). Upstream, a StorageClass points at one clusterID.
  • CSI topology lets a driver say where a volume can be used. Nodes report topology keys (here topology.rbd.csi.ceph.com/zone), and the volume's PV gets a nodeAffinity that the scheduler respects.
  • WaitForFirstConsumer (WFFC) delays provisioning until a pod using the PVC is scheduled, so the volume is created where the pod will run, not before.
  • Taint-based eviction: when a node stops reporting, the control plane marks it NotReady, adds node.kubernetes.io/unreachable taints, and evicts pods after their tolerationSeconds expire (300 s by default).

Architecture

Architecture: one Kubernetes cluster across three zones, each zone with its own Ceph cluster; Proxmox guests use the same per-zone storage

  • Kubernetes: kind v1.37.0, 3 control-plane and 3 worker nodes, one of each per zone, labeled topology.kubernetes.io/zone=zone-a|b|c. Three control-plane nodes means three etcd members, one per zone.
  • Ceph: three Ceph 19.2.7 (Squid) clusters, each with 3 MONs, 1 MGR and 3 OSDs, and a kubernetes pool with size 3, min_size 2. Each has its own fsid and a cephx user restricted to that pool. No rbd-mirror, no peering.
  • Proxmox (side test): three RBD storages (ceph-zone-a/b/c), each pointing only at its own zone's MONs, with a separate proxmox pool and cephx user. PVC images and Proxmox disks never appear in each other's pools.

The ceph-csi fork: version, build, install, and what it changes

Upstream ceph-csi does not yet route one StorageClass to several clusters by zone. The feature is requested in ceph/ceph-csi#5177, which is still open. I used an unofficial fork:

Item Value
Repository github.com/akaitux/ceph-csi-multiaz
Branch / commit v3.16.2-multiaz, 15b2ae4
Relation to upstream one commit ("multiaz") on top of upstream v3.16.2
Build git clone --depth 5 -b v3.16.2-multiaz, then go build with Go 1.25.0 (-mod=vendor, CGO, -tags=<ceph-release>,ceph_preview); builder and runtime base image quay.io/cephcsi/cephcsi:v3.18.1; result tagged localhost/cephcsi-multiaz:v3.16.2-multiaz; --version reports v3.16.2-multiaz and commit 15b2ae4…
Install plain manifests from the fork's deploy/rbd/kubernetes/, not Helm, adapted with sed: own namespace, image replaced, csi-provisioner set to v5.3.0 and csi-resizer to v1.14.0 (the manifests reference v6.0.0 and v2.0.0), --feature-gates=CSIVolumeGroupSnapshot=true removed, --domainlabels=topology.kubernetes.io/zone enabled on the node plugin
Other sidecars csi-snapshotter v8.4.0, csi-attacher v4.10.0, node-driver-registrar v2.15.0, snapshot-controller v8.6.0

What the fork commit changes, from reading its diff against v3.16.2:

  • New StorageClass parameter clusterTopologyConfigMap. clusterID becomes optional. The ConfigMap lists clusters with their zones, MONs, pool and Secret name.
  • Cluster selection in CreateVolume (internal/util/topology.go, FindClusterAndTopology). The driver walks the request's Preferred topologies first, then Requisite, and picks the first cluster whose zones contain the requested zone.
  • Per-cluster credentials. If the request carries no secrets, the controller reads the selected cluster's Secret (secretName) from its own namespace, and records clusterSecretName in the volume context.
  • Accessible topology in the response. The created volume carries accessible_topology with the zone segment, and the volume context gets ceph.csi/zone.
  • Snapshots and clones stay in the source cluster. When restoring from a snapshot, the topology requirement is pinned to the snapshot's cluster (code comment: "so the new PV lands in the correct AZ").
  • Related changes in the CephFS controller, group snapshots, a helper to find a PV by volume handle, and the Helm chart values.

The cluster map used in the PoC (addresses redacted):

{"clusterTopology":[
 {"clusterID":"ceph-zone-a","monitors":"<mon-a1>,<mon-a2>,<mon-a3>","zones":["zone-a"],"pool":"kubernetes","secretName":"csi-rbd-secret-zone-a"},
 {"clusterID":"ceph-zone-b","monitors":"<mon-b1>,<mon-b2>,<mon-b3>","zones":["zone-b"],"pool":"kubernetes","secretName":"csi-rbd-secret-zone-b"},
 {"clusterID":"ceph-zone-c","monitors":"<mon-c1>,<mon-c2>,<mon-c3>","zones":["zone-c"],"pool":"kubernetes","secretName":"csi-rbd-secret-zone-c"}
]}
Enter fullscreen mode Exit fullscreen mode

And the single StorageClass:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ceph-rbd-zonal
  annotations: {storageclass.kubernetes.io/is-default-class: "true"}
provisioner: rbd.csi.ceph.com
parameters:
  clusterTopologyConfigMap: ceph-cluster-topology
  imageFeatures: layering
  csi.storage.k8s.io/fstype: ext4
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
Enter fullscreen mode Exit fullscreen mode

How a PVC reaches the right Ceph cluster

Sequence: the PVC waits for scheduling; external-provisioner sends topology requirements; the fork maps the zone to a cluster; the PV gets nodeAffinity

Step by step, checked against the external-provisioner v5.3.0 topology documentation and the fork source:

  1. WFFC holds the PVC. The PVC stays Pending, and no image exists yet.
  2. The scheduler picks a node for the pod (spread constraints, affinity, resources) and records it on the PVC (volume.kubernetes.io/selected-node).
  3. external-provisioner builds accessibility_requirements. The Topology feature is on by default since v5.0. With delayed binding, no allowedTopologies and no strict topology, Requisite is the set of topology values of all nodes that run the driver (taken from their CSINode keys), and Preferred is the same set with the selected node's zone placed first.
  4. The fork maps zone to cluster. FindClusterAndTopology checks Preferred first, so the selected node's zone wins, matches it to the ConfigMap entry, loads that cluster's Secret, and creates the image in that cluster's pool.
  5. The driver returns accessible_topology (topology.rbd.csi.ceph.com/zone=zone-X). external-provisioner turns it into the PV's nodeAffinity.
  6. From then on the PV is pinned. Any pod using the PVC can only be scheduled into that zone. The proof test read the PV's nodeAffinity value back and compared it with the pod's node zone and the clusterID.

Spreading a StatefulSet across zones then needs one block:

topologySpreadConstraints:
- maxSkew: 1
  topologyKey: topology.kubernetes.io/zone
  whenUnsatisfiable: DoNotSchedule
  labelSelector: {matchLabels: {app: web-sts}}
Enter fullscreen mode Exit fullscreen mode

Functional checks

Check Result
3-replica StatefulSet spread over zones one replica per zone; rbd ls on each cluster showed only that zone's image; PV nodeAffinity zone = pod zone
Data survives pod delete and recreate new pod (new UID), same zone, same data
Online resize 1Gi → 3Gi about 7 s, no pod restart (df: 973.4M → 2.9G)
Snapshot and restore restored volume in the same zone, identical contents
PVC plus pod ready 6.0 s
Zone-aware bats suite 15 / 15 passed

I also stopped all three MONs of zone-c's Ceph for about 6 min 22 s. In zone-a, a new PVC bound and its pod ran. A new pod in zone-c stayed Pending with FailedMount and was not placed in another zone. An existing zone-c pod kept writing through its established kernel RBD session (the OSDs were still up). Everything recovered when the MONs returned.

The simulated zone outage

Setup

  • What was stopped: both kind node containers of zone-b (control plane and worker), with docker stop -t 10. Zone-b's Ceph stayed up.
  • Workloads: nginx and whoami Deployments, a rs-example ReplicaSet, a node-agent DaemonSet, a 3-replica web-sts StatefulSet (one PVC per zone), a single-replica Postgres (in zone-c, not affected), a proof/zonal StatefulSet, and the csi-rbdplugin-provisioner (3 replicas, required hostname anti-affinity, as in the upstream manifest).
  • Tolerations: nginx, whoami, Postgres and web-sts were patched to tolerationSeconds: 30. The provisioner, rs-example and proof/zonal kept the default 300 s, so one run shows both behaviors.
  • Driver and safety: a small lab dashboard called a restricted helper that can only stop or start one zone's nodes, and refuses to stop a second control-plane node while one is already down (three etcd members tolerate losing one).
  • Measurement: the dashboard recorded a per-pod and per-node timeline of state changes, and a driver script polled it every 15 s and collected Kubernetes events. The dashboard's own polling interval is not in the logs, so treat timeline values as accurate to a few seconds; events have 1 s resolution. t = 0 is when the stop command was issued. docker stop returned 11.2 s (run 1) and 11.3 s (run 2) later.
tolerations:
- key: node.kubernetes.io/unreachable
  operator: Exists
  effect: NoExecute
  tolerationSeconds: 30
- key: node.kubernetes.io/not-ready
  operator: Exists
  effect: NoExecute
  tolerationSeconds: 30
Enter fullscreen mode Exit fullscreen mode

Summary of runs 1 and 2

Each run tracked the same eight pods on zone-b's nodes: 4 stateless (nginx, whoami, rs-example in run 1 or a second nginx replica in run 2, and one provisioner replica), 2 PVC-backed (web-sts-1, proof/zonal-1) and 2 DaemonSet pods.

Event (seconds after t = 0) Run 1 (helper v1) Run 2 (helper v2)
Worker NotReady + unreachable taint +47 +46
Control-plane NotReady +47 +50 (taint +52)
Pods with tolerationSeconds=30 evicted +85 +76
Their stateless replacements Running in a surviving zone +87 to +89 +78 to +80
Pods with the default 300 s evicted +355 +347
PVC-backed pods Pending, PV pinned to zone-b same
Restore job duration 1.2 s 61.2 s
PVC-backed pods Running again not on their own; manual repair +694 (14 s after the job returned)

Bar chart of event times in run 1 and run 2 of the simulated zone-b outage

Timeline of the simulated outage: stop, NotReady, evictions, failover, Pending PVC pods, restore

Two runs on one shared host are not a statistical sample. Read the numbers as "what happened here", not as expected values.

Run 1 in detail (helper v1)

Time Observation
+15 to +20 s LeaderElection events: three leader-election leases of the provisioner sidecars moved to replicas outside zone-b, which means the zone-b replica had stopped renewing them.
+44 s NodeNotReady warnings for pods on zone-b nodes
+47 s both zone-b nodes NotReady and tainted unreachable
+85 s nginx, whoami and web-sts-1 evicted; nginx Running in zone-c at +87 s, whoami at +89 s; web-sts-1 waiting, PV pinned to zone-b
+354 s FailedScheduling for the provisioner replacement: 0/6 nodes are available: 2 node(s) didn't match pod anti-affinity rules, 4 node(s) had untolerated taint(s)
+355 s rs-example, proof/zonal-1 and the provisioner replica evicted; rs-example Running in zone-c at +364 s
+619 s restore job finished (1.2 s); worker Ready at +624 s, control plane at +628 s
+622 s FailedMount for web-sts-1: attacher.MountDevice failed to create newCsiDriverClient (the node plugin was not registered yet)
+626 to +631 s DaemonSet pods and the provisioner replica Running again; with the stateless pods already replaced, 6 of 8 tracked pods counted as recovered
+655 s BackOff restarting the csi-rbdplugin, driver-registrar and liveness-prometheus containers of the zone-b node plugin
until the watch ended, about 10 min after the restore web-sts-1 and proof/zonal-1 did not recover on their own

Root causes, found afterwards: Docker gave the two zone-b containers each other's IP addresses on restart, which left the zone-b etcd member unhealthy (2 of 3), and two kernel RBD devices were stuck (details below). The manual repair took several attempts: containers that did not stop cleanly (did not receive an exit event), Address already in use on reconnect, failing runc exec, and I/O errors while force-removing the RBD devices. It ended with the stale devices removed, the IPs back in place, 6 of 6 nodes Ready, etcd 3 of 3 healthy, and all PVCs bound to their original zone clusters.

Run 2 in detail (helper v2: IP pinning and krbd reaper)

Time Observation
+46 s worker NotReady and tainted; control plane NotReady at +50 s, taint at +52 s
+76 s both nginx replicas and whoami evicted; replacements Running at +78 s (nginx: one in zone-c, one in zone-a) and +80 s (whoami, zone-c). The nginx Service had ready endpoints in zones a and c throughout.
+345 s TaintManagerEviction and FailedScheduling for the provisioner replacement (same anti-affinity message as run 1)
+347 s proof/zonal-1 and the provisioner replica evicted; proof/zonal-1 waiting
about +618 s restore job started; the reaper found the stale devices, refused one orphan network namespace because it had other interfaces, revived the other, and removed 2 of 2 devices in 2 s
+624 s Starting kubelet on both zone-b nodes; worker NodeReady at +625 s, control plane at +631 s
+626 s FailedMount for web-sts-1 (newCsiDriverClient, same as run 1)
+627 s provisioner replacement scheduled back to the zone-b worker, Running at +630 s
+659 s BackOff on the zone-b node plugin containers; they started again at +671 s
+680 s restore job returned (61.2 s in total)
+690 s the StatefulSet created a new web-sts-1, scheduled to the zone-b worker; started at +694 s; proof/zonal-1 Running at +694 s, data intact

The job log does not break down the 61 s. The reaper reported 2 s, and the kubelets started about 6 s into the job, so most of the time came after docker start. The helper then retries mount -o remount,rw /sys inside each node, up to 20 times with a 1 s pause; that loop is the likely but unconfirmed cause. The PVC-backed pods were back about 70 s after the worker reported Ready. Most of that gap overlapped the node plugin's restart back-off.

Run 3 (quick check of helper v2)

A short down/up cycle (about 71 s between the stop and the start command) to exercise the reaper on a fresh orphan namespace. The reaper revived one orphan namespace, waited 64 s for pending I/O to drain, and removed 2 devices; the whole up took 65.5 s. Both nodes came back with their original IPs.

Why these timings would differ on real hardware

  • Node failure detection is a controller setting. kube-controller-manager marks a node unhealthy after --node-monitor-grace-period (default 50 s) and checks every --node-monitor-period (default 5 s) (reference). The kind config set neither, so the observed +46 to +50 s are consistent with the default. Since t = 0 is the stop command, not the last heartbeat, this is an explanation of the numbers, not a separate measurement.
  • Eviction is NotReady plus tolerationSeconds. The DefaultTolerationSeconds admission plugin adds 300 s tolerations for not-ready and unreachable to pods that do not set them (taint-based evictions). Run 1's 30 s pods were evicted at +85 s, about 8 s later than +47 plus 30.
  • Replacement start time depends on the host. Images were already present and the nodes shared one kernel and disk. On real nodes, image pulls, CNI and real network latency add time.
  • A real outage looks different. A power loss or network partition does not send SIGTERM. A partition can also leave the "dead" zone running and writing, which this test never exercised.
  • One shared VM means the surviving zones also carried the lab's other load. Neither run was isolated from the other.

Problems found and how they were fixed

1. Docker swapped node IPs and etcd became unhealthy

When zone-b's containers restarted, Docker assigned them each other's addresses. kind rewrote its manifests for the new IPs, but the etcd member's peer URL and certificates still used the old one: TLS SAN mismatch, member unhealthy, cluster on 2 of 3. Another failure would have cost quorum.

Fix: on down, record each node's IP and pin it with docker network connect --ip while the container is stopped. After runs 2 and 3, etcd was 3 of 3 healthy and the addresses did not change. The relevant part of the helper (names generalized):

NET=lab
ipof(){ docker inspect -f "{{(index .NetworkSettings.Networks \"$NET\").IPAddress}}" $1 2>/dev/null; }
pin(){ # $1 container (stopped) $2 ip
  cur=$(docker inspect -f "{{with (index .NetworkSettings.Networks \"$NET\").IPAMConfig}}{{.IPv4Address}}{{end}}" $1 2>/dev/null)
  [ "$cur" = "$2" ] && return 0
  docker network disconnect $NET $1 && docker network connect --ip $2 $NET $1 && echo "$1 pinned to $2"; }
# down: refuse if another control plane is already stopped, record IPs, stop, pin
# up:   pin, run the krbd reaper, docker start, remount /sys rw for krbd
Enter fullscreen mode Exit fullscreen mode

Full zone-ctl helper (v2, sanitized)
#!/bin/bash
# SSH forced command used by the lab dashboard; never a shell. v2: IP pinning + krbd reap.
set -uo pipefail
read -r op z extra <<<"${SSH_ORIGINAL_COMMAND:-}"
[ -z "${extra:-}" ] || { echo "too many args"; exit 2; }
case "${z:-}" in a) N="lab-control-plane lab-worker";; b) N="lab-control-plane2 lab-worker2";; c) N="lab-control-plane3 lab-worker3";; *) echo "zone must be a|b|c"; exit 2;; esac
logger -t zone-ctl "request: $op $z from ${SSH_CLIENT%% *}"
S=/var/lib/zone-ctl; NET=lab
ipof(){ docker inspect -f "{{(index .NetworkSettings.Networks \"$NET\").IPAddress}}" $1 2>/dev/null; }
pin(){ # $1 container (stopped) $2 ip
  cur=$(docker inspect -f "{{with (index .NetworkSettings.Networks \"$NET\").IPAMConfig}}{{.IPv4Address}}{{end}}" $1 2>/dev/null)
  [ "$cur" = "$2" ] && return 0
  docker network disconnect $NET $1 && docker network connect --ip $2 $NET $1 && echo "$1 pinned to $2"; }
case "${op:-}" in
  down)
    for cp in lab-control-plane lab-control-plane2 lab-control-plane3; do
      case " $N " in *" $cp "*) continue;; esac
      [ "$(docker inspect -f '{{.State.Running}}' $cp 2>/dev/null)" = true ] || { echo "refused: $cp is already down (etcd quorum would be lost)"; exit 3; }
    done
    for n in $N; do i=$(ipof $n); [ -n "$i" ] && [ "$i" != "invalid IP" ] && echo "$i" > $S/$n.ip; done
    docker stop -t 10 $N
    for n in $N; do [ -s $S/$n.ip ] && pin $n "$(cat $S/$n.ip)"; done ;;
  up)
    IPS=""; for n in $N; do [ -s $S/$n.ip ] && { pin $n "$(cat $S/$n.ip)"; IPS="$IPS $(cat $S/$n.ip)"; }; done
    [ -n "$IPS" ] && /usr/local/lib/zone-ctl/krbd-reap.py $IPS
    docker start $N
    for n in $N; do for i in $(seq 1 20); do docker exec $n mount -o remount,rw /sys 2>/dev/null && break; sleep 1; done; done
    echo "/sys remounted rw (krbd)" ;;
  status) ;;
  *) echo "op must be down|up|status"; exit 2 ;;
esac
docker inspect -f '{{.Name}} {{.State.Status}} {{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' $N
Enter fullscreen mode Exit fullscreen mode

2. Stale kernel RBD mappings

kind nodes are containers on the host kernel, so /dev/rbdN is a host kernel device. When the zone-b worker stopped, the libceph client of two RBD devices stayed in that node's network namespace, which no longer had any interfaces. Reconnects failed with ENETUNREACH. After restart, the node plugin reused those devices, NodeStage hung with blkid in D state, and a forced rbd remove hung too.

Fix: before starting the nodes, a reaper finds RBD devices whose client address belongs to a stopped node, refuses to act if a running container owns that address, finds orphan network namespaces (never the host namespace or one with other interfaces), gives each a temporary veth on the lab bridge so pending I/O can drain, force-removes the devices, and removes the veths. In run 2 this took 2 s; in run 3, 64 s.

krbd-reap.py (sanitized: bridge name and address range are placeholders)
#!/usr/bin/env python3
"""Reap krbd mappings whose libceph client lives in the netns of a STOPPED kind node. usage: krbd-reap.py <node-ip>..."""
import fcntl, glob, os, socket, struct, subprocess, sys, time
IPS = set(sys.argv[1:])
BRIDGE = "br-lab"        # the docker bridge the kind nodes are attached to
TMP_NET = "192.0.2"      # placeholder: a free address range on that bridge
def sh(*c, ok=False):
    r = subprocess.run(c, capture_output=True, text=True)
    if r.returncode and not ok: raise RuntimeError(f"{' '.join(c)}: {r.stderr.strip()}")
    return r.stdout
T0 = time.time()
def say(*m): print(f"krbd[+{int(time.time() - T0)}s]:", *m, flush=True)
def pending(): return [l for l in sh("ps", "-eo", "pid,args").splitlines() if "remove_single_major" in l and "sh -c" in l and "timeout" not in l]
devs = {os.path.basename(d): open(d + "/name").read().strip() for d in glob.glob("/sys/bus/rbd/devices/*")
        if open(d + "/client_addr").read().split(":")[0] in IPS}
if not devs and not pending(): say("no stale mappings"); sys.exit(0)
live = sh("docker", "ps", "-q").split()
busy = set(sh("docker", "inspect", "-f", "{{range .NetworkSettings.Networks}}{{.IPAddress}} {{end}}", *live).split()) if live else set()
if IPS & busy: say("refused: a RUNNING container owns", " ".join(sorted(IPS & busy)), "- mappings are live, not stale"); sys.exit(2)
say("stale mappings from", " ".join(sorted(IPS)) + ":", " ".join(f"rbd{k}={v}" for k, v in devs.items()) or "-", "| pending removes:", len(pending()))
used = set(int(x) for x in __import__("re").findall(r"link-netnsid (\d+)", sh("ip", "-o", "-d", "link", "show")))
named = set()
os.makedirs("/run/netns", exist_ok=True)
for f in glob.glob("/run/docker/netns/*") + ["/proc/1/ns/net"]:  # live docker netns + the host netns itself
    n = "czc-" + (os.path.basename(f) if f.startswith("/run") else "host"); p = "/run/netns/" + n; open(p, "a").close(); sh("mount", "--bind", f, p, ok=True)
for l in sh("ip", "netns", "list-id").splitlines():
    if "iproute2 netns name" in l: named.add(int(l.split()[1]))
allids = [int(l.split()[1]) for l in sh("ip", "netns", "list-id").splitlines() if l.startswith("nsid")]
for p in glob.glob("/run/netns/czc-*"): sh("umount", p, ok=True); os.unlink(p)
orphans = [i for i in allids if i not in used and i not in named]
say("orphan netns ids:", orphans or "none")
def a(t, p): l = 4 + len(p); return struct.pack("HH", l, t) + p + b"\0" * ((4 - l % 4) % 4)
def s(x): return x.encode() + b"\0"
ifi = lambda idx=0, fl=0, ch=0: struct.pack("BxHiII", 0, 0, idx, fl, ch)
def nl(t, flags, body):
    sk = socket.socket(socket.AF_NETLINK, socket.SOCK_RAW, 0); sk.bind((0, 0))
    sk.send(struct.pack("IHHII", 16 + len(body), t, flags, 1, 0) + body); d = sk.recv(65536); sk.close()
    if struct.unpack_from("H", d, 4)[0] == 2:
        e = struct.unpack_from("i", d, 16)[0]
        if e: raise OSError(-e, os.strerror(-e))
revived = []
for k, nsid in enumerate(orphans):
    hv, handle = f"vreap{nsid}", f"/run/netns/reap{nsid}"
    try:
        peer = ifi() + a(3, s(hv)) + a(19, struct.pack("I", os.getpid()))
        nl(16, 1 | 4 | 0x400 | 0x200, ifi(0, 1, 1) + a(46, struct.pack("i", nsid)) + a(3, s("eth9")) +
           a(18, a(1, s("veth")) + a(2, a(1, peer))))                       # veth eth9 (UP) inside, peer in host
        fd = os.open("/dev/net/tun", os.O_RDWR); tap = f"tapreap{nsid}"
        fcntl.ioctl(fd, 0x400454ca, struct.pack("16sH", tap.encode(), 0x0002 | 0x1000))
        nl(16, 1 | 4, ifi(socket.if_nametoindex(tap)) + a(46, struct.pack("i", nsid)))  # move tap into it
        nsfd = fcntl.ioctl(fd, 0x54e3)                                       # TUNGETDEVNETNS -> netns fd
        open(handle, "a").close(); sh("mount", "--bind", f"/proc/{os.getpid()}/fd/{nsfd}", handle)
        os.close(nsfd); os.close(fd)
        links = [l.split(":")[1].strip().split("@")[0] for l in sh("ip", "netns", "exec", f"reap{nsid}", "ip", "-o", "link").splitlines()]
        if not set(links) <= {"lo", "eth9", tap}: raise RuntimeError(f"nsid {nsid} has other links {links}; not touching it")
        ip = f"{TMP_NET}.{20 + k}"
        sh("ip", "netns", "exec", f"reap{nsid}", "ip", "link", "set", "lo", "up")
        sh("ip", "netns", "exec", f"reap{nsid}", "ip", "addr", "add", ip + "/24", "dev", "eth9")
        sh("ip", "link", "set", hv, "master", BRIDGE); sh("ip", "link", "set", hv, "up")
        say(f"nsid {nsid} revived via {hv} {ip}"); revived.append((hv, handle))
    except Exception as e:
        say(f"nsid {nsid}: {e}")
        sh("ip", "link", "del", hv, ok=True)
        if os.path.exists(handle): sh("umount", handle, ok=True); os.unlink(handle)
if not revived: say("WARNING no orphan netns revived; not force-removing (would hang)"); sys.exit(1)
procs = [subprocess.Popen(["timeout", "150", "sh", "-c", f"echo '{k} force' > /sys/bus/rbd/remove_single_major"]) for k in devs]
t1 = time.time()   # removal is done when the remove writers have returned (sysfs entries vanish before I/O drains)
while time.time() - t1 < 150 and (any(p.poll() is None for p in procs) or pending()): time.sleep(2)
left = [k for k in devs if os.path.exists(f"/sys/bus/rbd/devices/{k}")] + pending()
say(f"removed {len(devs)} device(s), drained in {int(time.time() - t1)}s" + (f", LEFT {left}" if left else ""))
for hv, handle in revived:
    sh("ip", "link", "del", hv, ok=True); sh("umount", handle, ok=True); os.unlink(handle)
say("temporary veths/netns handles removed")
sys.exit(1 if left else 0)
Enter fullscreen mode Exit fullscreen mode

The exact failure is specific to kind, but the general case is not: a node that loses its network with live krbd mappings can come back with stuck devices. Check for them before returning a node to service.

3. Hard anti-affinity and a lost zone

The provisioner runs 3 replicas with required hostname anti-affinity. With two workers left, the third replica stayed Pending for the rest of the outage. Its sidecars run with leader election (as in the upstream manifest), and the leases moved to other replicas within about 20 s in run 1, so provisioning was not affected. If you alert on Pending pods in system namespaces, every zone outage will trigger that alert. Options: two replicas, or zone-based preferred anti-affinity.

4. tolerationSeconds sets the stateless recovery time

Failure detection was stable at about 46–50 s. The setting that changed the outcome was tolerationSeconds: 30 s led to replacements at about +80–90 s, the default 300 s to about +350 s. A lower value also means pod churn on short network interruptions, so set it per workload class.

5. DoNotSchedule did not block failover (inference)

nginx uses a hard spread constraint (whenUnsatisfiable: DoNotSchedule) and nodeTaintsPolicy: Honor, yet its replacements were scheduled into zones a and c. According to the Kubernetes documentation, with Honor the skew calculation only includes untainted nodes and tainted nodes the pod tolerates; with the default (Ignore), all nodes count. The unreachable node also carries a NoSchedule taint that the default tolerations do not cover, and the FailedScheduling message above ("4 node(s) had untolerated taint(s)", which counts the three control-plane nodes plus the zone-b worker) shows that the scheduler saw that taint.

This explanation is an inference from the manifest, the documentation and the outcome. What would confirm it:

  • an A/B rerun with nodeTaintsPolicy unset: if the explanation is right, the replacement should stay Pending with a topology spread message;
  • scheduler logs at higher verbosity showing which nodes enter the PodTopologySpread calculation.

Stateful workloads: what to recommend

In this design storage does not move between zones, so a PVC-backed pod is only as available as its zone. That is a deliberate trade-off for a smaller blast radius. The platform decision is how much of the remaining work it takes on.

Recommended order for a platform team, with the conditions for each:

  1. Default: recommended patterns with application-level replication. Ship blueprints for a StatefulSet with one replica per zone (the spread block above), each on its own zone's Ceph, and replication done by the application: PostgreSQL streaming replication, MongoDB replica sets, Kafka or etcd quorums. Choose this when the data store has mature built-in replication and the owning team can operate it. The outage test showed the shape: web-sts-0 and web-sts-2 in zones a and c kept running, and only web-sts-1 waited.
  2. Next: operator-managed data services. The platform runs operators and owns failover, backups and replica rebuilds. Choose this when many tenant teams need databases but lack the skills to run them, and the platform team can take database on-call and SLAs. Start with one or two engines.
  3. Only for tolerant workloads: document the limitation. State clearly that a PVC is pinned to one zone and that a single-replica stateful app is down for as long as its zone is. Acceptable for dev/test, caches, rebuildable data, or apps whose recovery objective is longer than a typical zone outage.

Two cases where per-zone Ceph is the wrong tool: a single-instance application that cannot replicate and must survive a zone loss with no data loss (consider a stretched cluster, if the sites qualify), and disaster recovery to another site for such an application where some data loss is acceptable (consider RBD mirroring).

Moving one volume to another zone is an explicit copy, because the clusters are not connected. On the Proxmox side, an offline move of a container's rootfs (about 517 MB of data) from zone-a to zone-b took 105.5 s and 99.5 s back, with the container stopped throughout. That is a migration with downtime.

Comparison: per-zone Ceph, stretched Ceph, RBD mirroring

Not benchmarked here. This table summarizes the documentation and the design properties; none of it was measured in this PoC.

Property Independent Ceph per zone (this PoC) Stretched Ceph cluster RBD mirroring between clusters
Write path local to one zone synchronous across zones: each write is stored in both zones before it is acknowledged (stretch mode) local writes, asynchronous copy (RBD mirroring); journal-based mode writes every change twice, and the docs say to "expect write latencies to nearly double"
Data loss on losing a zone (storage layer) volumes of that zone are unavailable until it returns none expected; the surviving zone serves I/O in degraded mode up to the replication lag or snapshot interval; partially applied snapshot deltas are rolled back on failover
Volume usable in another zone no yes after failover (image promotion on the peer cluster)
Failure domain one zone one cluster spanning sites: one monitor quorum, one version, one upgrade path separate clusters
Topology constraints none between zones stretch mode supports exactly two data zones plus a tiebreaker monitor at a third site peer clusters configured per pool; one-way or two-way
Kubernetes integration topology-aware CSI (fork) standard ceph-csi, one cluster ceph-csi plus csi-addons VolumeReplication for promote/demote
Operational cost three clusters to run; no cross-zone data path sensitive to inter-site latency; tiebreaker placement mirror daemons, failover and failback procedures; data copied twice if the app already replicates

Benchmark plan (future work)

I did not run fio on this PoC. All three "zones" share one VM and the same disks, so the results would say nothing about cross-zone behavior. On real per-zone hardware, the plan is:

  • Baseline per zone: fio on an RBD PVC in each zone with --direct=1 --ioengine=libaio --time_based --runtime=120, three repetitions each:
    • 4k random write, iodepth 1: p50 and p99 completion latency;
    • 4k random read, iodepth 32: IOPS;
    • 1M sequential write: throughput.
  • Network baseline: RTT between zones (ping and a TCP test) recorded with each run.
  • Comparisons: the same jobs against a stretched pool (if two qualifying sites exist) and against an image with journal-based mirroring enabled, to measure the cross-zone and journaling cost directly.
  • Under failure: repeat the 4k write test during a MON outage and during a zone outage to see stalls, not only averages.

Next steps

The thresholds below are proposed acceptance criteria, not results.

  1. Repeat the outage test on real hardware with separate nodes per zone, using three failure types: hard power-off, network partition between zones, and a graceful drain. Run each at least five times. Record NotReady time, eviction time, replacement Ready time and PVC pod recovery time after the zone returns, and report median and maximum.
  2. Zone-recovery gate. A zone counts as recovered only when etcd is 3 of 3 healthy, no /sys/bus/rbd/devices/*/client_addr points at a stopped node, there are no processes in D state on the returned nodes, and every PVC-backed pod is Running. Target: no manual steps in 5 of 5 runs.
  3. Tolerations per workload class. Test 30, 60 and 120 s against short network flaps (for example 10 s and 20 s link drops). Target: no evictions for flaps shorter than the chosen value, failover within value plus 60 s otherwise.
  4. System workloads. Change the provisioner to two replicas or zone-based preferred anti-affinity. Target: no Pending system pods during a single-zone outage.
  5. Confirm the nodeTaintsPolicy inference with the A/B rerun described above.
  6. CSI strategy. Track ceph/ceph-csi#5177. While on the fork, rebase onto each upstream release, and gate upgrades on the 15 zone-aware tests plus snapshot/restore and resize. The fallback is one StorageClass per zone with upstream ceph-csi.
  7. Production-style Ceph with standby MGRs, real disks and capacity sized for real load, then run the benchmark plan.
  8. Alerting. Alert when a PVC-backed pod is Pending because of a volume node affinity conflict for longer than a chosen time (for example 5 minutes), and silence the expected provisioner replica during a declared zone outage.

Discussion

If you run Ceph under a multi-zone Kubernetes cluster: did you choose a stretched cluster, RBD mirroring, or independent per-zone clusters? And what tolerationSeconds do you use in production? I would like to compare notes in the comments.

Ramin Najarbashi

Top comments (1)

Collapse
 
onelifevibe profile image
OneLifeVibe •

This experiment makes the storage failure domain argument concrete. Separating Ceph per zone with WaitForFirstConsumer and the multi-AZ ceph-csi fork is a clear design, and the measured contrast is striking: stateless pods returned in about 80 to 90 seconds with tolerationSeconds set to 30, against about six minutes with the default 300 seconds, while PVC backed pods stayed Pending by design and returned about 14 seconds after recovery in run 2. The two outage only bugs also stand out, especially the Docker IP swap that broke etcd and the orphaned kernel RBD mappings that blocked recovery in run 1. The honesty box about the single VM proof of concept keeps the results in proper context.