<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Ramin Najjarbashi</title>
    <description>The latest articles on DEV Community by Ramin Najjarbashi (@raminnietzsche).</description>
    <link>https://dev.to/raminnietzsche</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F1053598%2Fdc1aca15-37d9-4d9b-b934-665b023a9465.jpg</url>
      <title>DEV Community: Ramin Najjarbashi</title>
      <link>https://dev.to/raminnietzsche</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/raminnietzsche"/>
    <language>en</language>
    <item>
      <title>Zone-Aware Storage on Kubernetes with Three Independent Ceph Clusters: A kind-Based PoC and a Simulated Zone Outage</title>
      <dc:creator>Ramin Najjarbashi</dc:creator>
      <pubDate>Sat, 21 Nov 2026 12:47:16 +0000</pubDate>
      <link>https://dev.to/raminnietzsche/one-kubernetes-cluster-three-zones-three-cephs-we-pulled-the-plug-on-a-zone-4kbg</link>
      <guid>https://dev.to/raminnietzsche/one-kubernetes-cluster-three-zones-three-cephs-we-pulled-the-plug-on-a-zone-4kbg</guid>
      <description>&lt;p&gt;If every volume on a container platform lives in one Ceph cluster, that cluster is one large failure domain. A bad upgrade, a lost MON quorum or a few full OSDs can take down every tenant in every zone at once. I built a proof of concept to test a different layout: one Kubernetes cluster across three zones, with a separate, independent Ceph cluster in each zone. Then I simulated losing a zone and recorded what happened.&lt;/p&gt;

&lt;h2&gt;
  
  
  TL;DR
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Design:&lt;/strong&gt; one Kubernetes cluster over three zones, one independent Ceph cluster per zone, no replication between the Ceph clusters. Each PVC is created in the Ceph cluster of the zone its pod is scheduled into.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Mechanism:&lt;/strong&gt; &lt;code&gt;volumeBindingMode: WaitForFirstConsumer&lt;/code&gt;, zone labels on nodes, and an unofficial multi-cluster fork of ceph-csi that lets one StorageClass map a zone to a cluster.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Functional result:&lt;/strong&gt; placement, resize, snapshot/restore and per-zone isolation worked; 15 of 15 zone-aware tests passed.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Simulated zone outage:&lt;/strong&gt; stateless pods with &lt;code&gt;tolerationSeconds: 30&lt;/code&gt; were running in another zone about 80–90 s after the stop command; with the default 300 s, eviction alone took about 6 minutes. PVC-backed pods stayed &lt;code&gt;Pending&lt;/code&gt; until the zone returned, which is the expected behavior of this design.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Harness bugs:&lt;/strong&gt; the first run exposed two problems in the lab setup (Docker reassigning node IPs, and stale kernel RBD mappings), not in ceph-csi. Both are fixed and shown below.&lt;/li&gt;
&lt;/ul&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Read this first: what this PoC is and is not&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;One VM.&lt;/strong&gt; All six Kubernetes nodes (kind, v1.37.0) and all three Ceph clusters ran on a single virtual machine and shared one Linux kernel, one disk subsystem and one network stack.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Logical zones.&lt;/strong&gt; "Zones" are node labels (&lt;code&gt;topology.kubernetes.io/zone&lt;/code&gt;), not separate sites, racks or power domains. There is no real inter-zone latency.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Simulated outage.&lt;/strong&gt; "Losing zone-b" means &lt;code&gt;docker stop&lt;/code&gt; on zone-b's two kind node containers. Zone-b's Ceph stayed up. There was no power cut, no network partition and no disk failure.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Lab timings.&lt;/strong&gt; All times come from a shared host with default Kubernetes controller settings. They show the order of events and which settings dominate; they are not predictions for real hardware.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Not benchmarked.&lt;/strong&gt; No storage performance was measured. The comparison with stretched Ceph and RBD mirroring later in the post is qualitative and sourced from documentation.&lt;/li&gt;
&lt;/ul&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Key takeaways
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;With &lt;code&gt;WaitForFirstConsumer&lt;/code&gt; and a topology-aware CSI driver, one StorageClass can place each volume in its pod's zone across three independent Ceph clusters.&lt;/li&gt;
&lt;li&gt;In this design, storage does not provide cross-zone failover. A PVC-backed pod waits for its zone. Availability for stateful apps has to come from application-level replication.&lt;/li&gt;
&lt;li&gt;For stateless pods, &lt;code&gt;tolerationSeconds&lt;/code&gt; mattered far more than node failure detection: about 30 s vs 300 s of extra waiting after &lt;code&gt;NotReady&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Recovering a zone cleanly needed two checks that no functional test covered: all etcd members healthy, and no stale kernel RBD mappings left from the dead node.&lt;/li&gt;
&lt;li&gt;Hard anti-affinity sized to the number of nodes (the CSI provisioner) leaves a &lt;code&gt;Pending&lt;/code&gt; replica during every zone outage. That is harmless, but it creates alert noise.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  A short primer
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;ceph-csi&lt;/strong&gt; is the CSI driver for Ceph. Its RBD driver creates block images in a Ceph pool and maps them on nodes with the kernel RBD client (&lt;code&gt;krbd&lt;/code&gt;). Upstream, a StorageClass points at one &lt;code&gt;clusterID&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;CSI topology&lt;/strong&gt; lets a driver say where a volume can be used. Nodes report topology keys (here &lt;code&gt;topology.rbd.csi.ceph.com/zone&lt;/code&gt;), and the volume's PV gets a &lt;code&gt;nodeAffinity&lt;/code&gt; that the scheduler respects.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;WaitForFirstConsumer&lt;/code&gt; (WFFC)&lt;/strong&gt; delays provisioning until a pod using the PVC is scheduled, so the volume is created where the pod will run, not before.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Taint-based eviction:&lt;/strong&gt; when a node stops reporting, the control plane marks it &lt;code&gt;NotReady&lt;/code&gt;, adds &lt;code&gt;node.kubernetes.io/unreachable&lt;/code&gt; taints, and evicts pods after their &lt;code&gt;tolerationSeconds&lt;/code&gt; expire (300 s by default).&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Architecture
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Frzcam37h4w6ee2yw31ox.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Frzcam37h4w6ee2yw31ox.png" alt="Architecture: one Kubernetes cluster across three zones, each zone with its own Ceph cluster; Proxmox guests use the same per-zone storage" width="799" height="235"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Kubernetes:&lt;/strong&gt; kind &lt;code&gt;v1.37.0&lt;/code&gt;, 3 control-plane and 3 worker nodes, one of each per zone, labeled &lt;code&gt;topology.kubernetes.io/zone=zone-a|b|c&lt;/code&gt;. Three control-plane nodes means three etcd members, one per zone.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Ceph:&lt;/strong&gt; three Ceph 19.2.7 (Squid) clusters, each with 3 MONs, 1 MGR and 3 OSDs, and a &lt;code&gt;kubernetes&lt;/code&gt; pool with &lt;code&gt;size 3&lt;/code&gt;, &lt;code&gt;min_size 2&lt;/code&gt;. Each has its own fsid and a cephx user restricted to that pool. No &lt;code&gt;rbd-mirror&lt;/code&gt;, no peering.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Proxmox (side test):&lt;/strong&gt; three RBD storages (&lt;code&gt;ceph-zone-a/b/c&lt;/code&gt;), each pointing only at its own zone's MONs, with a separate &lt;code&gt;proxmox&lt;/code&gt; pool and cephx user. PVC images and Proxmox disks never appear in each other's pools.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The ceph-csi fork: version, build, install, and what it changes
&lt;/h2&gt;

&lt;p&gt;Upstream ceph-csi does not yet route one StorageClass to several clusters by zone. The feature is requested in &lt;a href="https://github.com/ceph/ceph-csi/issues/5177" rel="noopener noreferrer"&gt;ceph/ceph-csi#5177&lt;/a&gt;, which is still open. I used an unofficial fork:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Item&lt;/th&gt;
&lt;th&gt;Value&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Repository&lt;/td&gt;
&lt;td&gt;&lt;a href="https://github.com/akaitux/ceph-csi-multiaz/tree/v3.16.2-multiaz" rel="noopener noreferrer"&gt;github.com/akaitux/ceph-csi-multiaz&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Branch / commit&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;v3.16.2-multiaz&lt;/code&gt;, &lt;a href="https://github.com/akaitux/ceph-csi-multiaz/commit/15b2ae49c589bf8866bba9c0858b514be0c4b8bf" rel="noopener noreferrer"&gt;&lt;code&gt;15b2ae4&lt;/code&gt;&lt;/a&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Relation to upstream&lt;/td&gt;
&lt;td&gt;one commit ("multiaz") on top of upstream &lt;code&gt;v3.16.2&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Build&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;git clone --depth 5 -b v3.16.2-multiaz&lt;/code&gt;, then &lt;code&gt;go build&lt;/code&gt; with Go 1.25.0 (&lt;code&gt;-mod=vendor&lt;/code&gt;, CGO, &lt;code&gt;-tags=&amp;lt;ceph-release&amp;gt;,ceph_preview&lt;/code&gt;); builder and runtime base image &lt;code&gt;quay.io/cephcsi/cephcsi:v3.18.1&lt;/code&gt;; result tagged &lt;code&gt;localhost/cephcsi-multiaz:v3.16.2-multiaz&lt;/code&gt;; &lt;code&gt;--version&lt;/code&gt; reports &lt;code&gt;v3.16.2-multiaz&lt;/code&gt; and commit &lt;code&gt;15b2ae4…&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Install&lt;/td&gt;
&lt;td&gt;plain manifests from the fork's &lt;code&gt;deploy/rbd/kubernetes/&lt;/code&gt;, not Helm, adapted with &lt;code&gt;sed&lt;/code&gt;: own namespace, image replaced, &lt;code&gt;csi-provisioner&lt;/code&gt; set to v5.3.0 and &lt;code&gt;csi-resizer&lt;/code&gt; to v1.14.0 (the manifests reference v6.0.0 and v2.0.0), &lt;code&gt;--feature-gates=CSIVolumeGroupSnapshot=true&lt;/code&gt; removed, &lt;code&gt;--domainlabels=topology.kubernetes.io/zone&lt;/code&gt; enabled on the node plugin&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Other sidecars&lt;/td&gt;
&lt;td&gt;csi-snapshotter v8.4.0, csi-attacher v4.10.0, node-driver-registrar v2.15.0, snapshot-controller v8.6.0&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;What the fork commit changes, from reading its diff against &lt;code&gt;v3.16.2&lt;/code&gt;:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;New StorageClass parameter &lt;code&gt;clusterTopologyConfigMap&lt;/code&gt;.&lt;/strong&gt; &lt;code&gt;clusterID&lt;/code&gt; becomes optional. The ConfigMap lists clusters with their zones, MONs, pool and Secret name.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Cluster selection in &lt;code&gt;CreateVolume&lt;/code&gt;&lt;/strong&gt; (&lt;code&gt;internal/util/topology.go&lt;/code&gt;, &lt;code&gt;FindClusterAndTopology&lt;/code&gt;). The driver walks the request's &lt;code&gt;Preferred&lt;/code&gt; topologies first, then &lt;code&gt;Requisite&lt;/code&gt;, and picks the first cluster whose &lt;code&gt;zones&lt;/code&gt; contain the requested zone.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Per-cluster credentials.&lt;/strong&gt; If the request carries no secrets, the controller reads the selected cluster's Secret (&lt;code&gt;secretName&lt;/code&gt;) from its own namespace, and records &lt;code&gt;clusterSecretName&lt;/code&gt; in the volume context.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Accessible topology in the response.&lt;/strong&gt; The created volume carries &lt;code&gt;accessible_topology&lt;/code&gt; with the zone segment, and the volume context gets &lt;code&gt;ceph.csi/zone&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Snapshots and clones stay in the source cluster.&lt;/strong&gt; When restoring from a snapshot, the topology requirement is pinned to the snapshot's cluster (code comment: "so the new PV lands in the correct AZ").&lt;/li&gt;
&lt;li&gt;Related changes in the CephFS controller, group snapshots, a helper to find a PV by volume handle, and the Helm chart values.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The cluster map used in the PoC (addresses redacted):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nl"&gt;"clusterTopology"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;
 &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nl"&gt;"clusterID"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="s2"&gt;"ceph-zone-a"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="nl"&gt;"monitors"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="s2"&gt;"&amp;lt;mon-a1&amp;gt;,&amp;lt;mon-a2&amp;gt;,&amp;lt;mon-a3&amp;gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="nl"&gt;"zones"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"zone-a"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="nl"&gt;"pool"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="s2"&gt;"kubernetes"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="nl"&gt;"secretName"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="s2"&gt;"csi-rbd-secret-zone-a"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
 &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nl"&gt;"clusterID"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="s2"&gt;"ceph-zone-b"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="nl"&gt;"monitors"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="s2"&gt;"&amp;lt;mon-b1&amp;gt;,&amp;lt;mon-b2&amp;gt;,&amp;lt;mon-b3&amp;gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="nl"&gt;"zones"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"zone-b"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="nl"&gt;"pool"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="s2"&gt;"kubernetes"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="nl"&gt;"secretName"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="s2"&gt;"csi-rbd-secret-zone-b"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
 &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nl"&gt;"clusterID"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="s2"&gt;"ceph-zone-c"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="nl"&gt;"monitors"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="s2"&gt;"&amp;lt;mon-c1&amp;gt;,&amp;lt;mon-c2&amp;gt;,&amp;lt;mon-c3&amp;gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="nl"&gt;"zones"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"zone-c"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="nl"&gt;"pool"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="s2"&gt;"kubernetes"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="nl"&gt;"secretName"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="s2"&gt;"csi-rbd-secret-zone-c"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;And the single StorageClass:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;storage.k8s.io/v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;StorageClass&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ceph-rbd-zonal&lt;/span&gt;
  &lt;span class="na"&gt;annotations&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s"&gt;storageclass.kubernetes.io/is-default-class:&lt;/span&gt; &lt;span class="s2"&gt;"true"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="na"&gt;provisioner&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;rbd.csi.ceph.com&lt;/span&gt;
&lt;span class="na"&gt;parameters&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;clusterTopologyConfigMap&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ceph-cluster-topology&lt;/span&gt;
  &lt;span class="na"&gt;imageFeatures&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;layering&lt;/span&gt;
  &lt;span class="na"&gt;csi.storage.k8s.io/fstype&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ext4&lt;/span&gt;
&lt;span class="na"&gt;reclaimPolicy&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Delete&lt;/span&gt;
&lt;span class="na"&gt;allowVolumeExpansion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
&lt;span class="na"&gt;volumeBindingMode&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;WaitForFirstConsumer&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  How a PVC reaches the right Ceph cluster
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fh3vxpy3dkgpyxryhvuhp.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fh3vxpy3dkgpyxryhvuhp.png" alt="Sequence: the PVC waits for scheduling; external-provisioner sends topology requirements; the fork maps the zone to a cluster; the PV gets nodeAffinity" width="799" height="528"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Step by step, checked against the &lt;a href="https://github.com/kubernetes-csi/external-provisioner/tree/v5.3.0#topology-support" rel="noopener noreferrer"&gt;external-provisioner v5.3.0 topology documentation&lt;/a&gt; and the fork source:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;WFFC holds the PVC.&lt;/strong&gt; The PVC stays &lt;code&gt;Pending&lt;/code&gt;, and no image exists yet.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The scheduler picks a node&lt;/strong&gt; for the pod (spread constraints, affinity, resources) and records it on the PVC (&lt;code&gt;volume.kubernetes.io/selected-node&lt;/code&gt;).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;external-provisioner builds &lt;code&gt;accessibility_requirements&lt;/code&gt;.&lt;/strong&gt; The Topology feature is on by default since v5.0. With delayed binding, no &lt;code&gt;allowedTopologies&lt;/code&gt; and no strict topology, &lt;code&gt;Requisite&lt;/code&gt; is the set of topology values of all nodes that run the driver (taken from their &lt;code&gt;CSINode&lt;/code&gt; keys), and &lt;code&gt;Preferred&lt;/code&gt; is the same set with the selected node's zone placed first.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The fork maps zone to cluster.&lt;/strong&gt; &lt;code&gt;FindClusterAndTopology&lt;/code&gt; checks &lt;code&gt;Preferred&lt;/code&gt; first, so the selected node's zone wins, matches it to the ConfigMap entry, loads that cluster's Secret, and creates the image in that cluster's pool.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The driver returns &lt;code&gt;accessible_topology&lt;/code&gt;&lt;/strong&gt; (&lt;code&gt;topology.rbd.csi.ceph.com/zone=zone-X&lt;/code&gt;). external-provisioner turns it into the PV's &lt;code&gt;nodeAffinity&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;From then on the PV is pinned.&lt;/strong&gt; Any pod using the PVC can only be scheduled into that zone. The proof test read the PV's &lt;code&gt;nodeAffinity&lt;/code&gt; value back and compared it with the pod's node zone and the &lt;code&gt;clusterID&lt;/code&gt;.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Spreading a StatefulSet across zones then needs one block:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;topologySpreadConstraints&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;maxSkew&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
  &lt;span class="na"&gt;topologyKey&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;topology.kubernetes.io/zone&lt;/span&gt;
  &lt;span class="na"&gt;whenUnsatisfiable&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;DoNotSchedule&lt;/span&gt;
  &lt;span class="na"&gt;labelSelector&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s"&gt;matchLabels:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s"&gt;app:&lt;/span&gt; &lt;span class="s"&gt;web-sts&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Functional checks
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Check&lt;/th&gt;
&lt;th&gt;Result&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;3-replica StatefulSet spread over zones&lt;/td&gt;
&lt;td&gt;one replica per zone; &lt;code&gt;rbd ls&lt;/code&gt; on each cluster showed only that zone's image; PV &lt;code&gt;nodeAffinity&lt;/code&gt; zone = pod zone&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Data survives pod delete and recreate&lt;/td&gt;
&lt;td&gt;new pod (new UID), same zone, same data&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Online resize 1Gi → 3Gi&lt;/td&gt;
&lt;td&gt;about 7 s, no pod restart (&lt;code&gt;df&lt;/code&gt;: 973.4M → 2.9G)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Snapshot and restore&lt;/td&gt;
&lt;td&gt;restored volume in the same zone, identical contents&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;PVC plus pod ready&lt;/td&gt;
&lt;td&gt;6.0 s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Zone-aware bats suite&lt;/td&gt;
&lt;td&gt;15 / 15 passed&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;I also stopped all three MONs of zone-c's Ceph for about 6 min 22 s. In zone-a, a new PVC bound and its pod ran. A new pod in zone-c stayed &lt;code&gt;Pending&lt;/code&gt; with &lt;code&gt;FailedMount&lt;/code&gt; and was not placed in another zone. An existing zone-c pod kept writing through its established kernel RBD session (the OSDs were still up). Everything recovered when the MONs returned.&lt;/p&gt;

&lt;h2&gt;
  
  
  The simulated zone outage
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Setup
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;What was stopped:&lt;/strong&gt; both kind node containers of zone-b (control plane and worker), with &lt;code&gt;docker stop -t 10&lt;/code&gt;. Zone-b's Ceph stayed up.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Workloads:&lt;/strong&gt; &lt;code&gt;nginx&lt;/code&gt; and &lt;code&gt;whoami&lt;/code&gt; Deployments, a &lt;code&gt;rs-example&lt;/code&gt; ReplicaSet, a &lt;code&gt;node-agent&lt;/code&gt; DaemonSet, a 3-replica &lt;code&gt;web-sts&lt;/code&gt; StatefulSet (one PVC per zone), a single-replica Postgres (in zone-c, not affected), a &lt;code&gt;proof/zonal&lt;/code&gt; StatefulSet, and the &lt;code&gt;csi-rbdplugin-provisioner&lt;/code&gt; (3 replicas, required hostname anti-affinity, as in the upstream manifest).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Tolerations:&lt;/strong&gt; &lt;code&gt;nginx&lt;/code&gt;, &lt;code&gt;whoami&lt;/code&gt;, Postgres and &lt;code&gt;web-sts&lt;/code&gt; were patched to &lt;code&gt;tolerationSeconds: 30&lt;/code&gt;. The provisioner, &lt;code&gt;rs-example&lt;/code&gt; and &lt;code&gt;proof/zonal&lt;/code&gt; kept the default 300 s, so one run shows both behaviors.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Driver and safety:&lt;/strong&gt; a small lab dashboard called a restricted helper that can only stop or start one zone's nodes, and refuses to stop a second control-plane node while one is already down (three etcd members tolerate losing one).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Measurement:&lt;/strong&gt; the dashboard recorded a per-pod and per-node timeline of state changes, and a driver script polled it every 15 s and collected Kubernetes events. The dashboard's own polling interval is not in the logs, so treat timeline values as accurate to a few seconds; events have 1 s resolution. &lt;strong&gt;t = 0 is when the stop command was issued.&lt;/strong&gt; &lt;code&gt;docker stop&lt;/code&gt; returned 11.2 s (run 1) and 11.3 s (run 2) later.
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;tolerations&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;key&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;node.kubernetes.io/unreachable&lt;/span&gt;
  &lt;span class="na"&gt;operator&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Exists&lt;/span&gt;
  &lt;span class="na"&gt;effect&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;NoExecute&lt;/span&gt;
  &lt;span class="na"&gt;tolerationSeconds&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;30&lt;/span&gt;
&lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;key&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;node.kubernetes.io/not-ready&lt;/span&gt;
  &lt;span class="na"&gt;operator&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Exists&lt;/span&gt;
  &lt;span class="na"&gt;effect&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;NoExecute&lt;/span&gt;
  &lt;span class="na"&gt;tolerationSeconds&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;30&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Summary of runs 1 and 2
&lt;/h3&gt;

&lt;p&gt;Each run tracked the same eight pods on zone-b's nodes: 4 stateless (&lt;code&gt;nginx&lt;/code&gt;, &lt;code&gt;whoami&lt;/code&gt;, &lt;code&gt;rs-example&lt;/code&gt; in run 1 or a second &lt;code&gt;nginx&lt;/code&gt; replica in run 2, and one provisioner replica), 2 PVC-backed (&lt;code&gt;web-sts-1&lt;/code&gt;, &lt;code&gt;proof/zonal-1&lt;/code&gt;) and 2 DaemonSet pods.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Event (seconds after t = 0)&lt;/th&gt;
&lt;th&gt;Run 1 (helper v1)&lt;/th&gt;
&lt;th&gt;Run 2 (helper v2)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Worker &lt;code&gt;NotReady&lt;/code&gt; + &lt;code&gt;unreachable&lt;/code&gt; taint&lt;/td&gt;
&lt;td&gt;+47&lt;/td&gt;
&lt;td&gt;+46&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Control-plane &lt;code&gt;NotReady&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;+47&lt;/td&gt;
&lt;td&gt;+50 (taint +52)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Pods with &lt;code&gt;tolerationSeconds=30&lt;/code&gt; evicted&lt;/td&gt;
&lt;td&gt;+85&lt;/td&gt;
&lt;td&gt;+76&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Their stateless replacements &lt;code&gt;Running&lt;/code&gt; in a surviving zone&lt;/td&gt;
&lt;td&gt;+87 to +89&lt;/td&gt;
&lt;td&gt;+78 to +80&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Pods with the default 300 s evicted&lt;/td&gt;
&lt;td&gt;+355&lt;/td&gt;
&lt;td&gt;+347&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;PVC-backed pods&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;Pending&lt;/code&gt;, PV pinned to zone-b&lt;/td&gt;
&lt;td&gt;same&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Restore job duration&lt;/td&gt;
&lt;td&gt;1.2 s&lt;/td&gt;
&lt;td&gt;61.2 s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;PVC-backed pods &lt;code&gt;Running&lt;/code&gt; again&lt;/td&gt;
&lt;td&gt;not on their own; manual repair&lt;/td&gt;
&lt;td&gt;+694 (14 s after the job returned)&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fry59p8vqy6nbgava929h.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fry59p8vqy6nbgava929h.png" alt="Bar chart of event times in run 1 and run 2 of the simulated zone-b outage" width="800" height="368"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F86zq60gnslcui11xv7lz.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F86zq60gnslcui11xv7lz.png" alt="Timeline of the simulated outage: stop, NotReady, evictions, failover, Pending PVC pods, restore" width="799" height="167"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Two runs on one shared host are not a statistical sample. Read the numbers as "what happened here", not as expected values.&lt;/p&gt;

&lt;h3&gt;
  
  
  Run 1 in detail (helper v1)
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Time&lt;/th&gt;
&lt;th&gt;Observation&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;+15 to +20 s&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;LeaderElection&lt;/code&gt; events: three leader-election leases of the provisioner sidecars moved to replicas outside zone-b, which means the zone-b replica had stopped renewing them.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;+44 s&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;NodeNotReady&lt;/code&gt; warnings for pods on zone-b nodes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;+47 s&lt;/td&gt;
&lt;td&gt;both zone-b nodes &lt;code&gt;NotReady&lt;/code&gt; and tainted &lt;code&gt;unreachable&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;+85 s&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;nginx&lt;/code&gt;, &lt;code&gt;whoami&lt;/code&gt; and &lt;code&gt;web-sts-1&lt;/code&gt; evicted; &lt;code&gt;nginx&lt;/code&gt; &lt;code&gt;Running&lt;/code&gt; in zone-c at +87 s, &lt;code&gt;whoami&lt;/code&gt; at +89 s; &lt;code&gt;web-sts-1&lt;/code&gt; waiting, PV pinned to zone-b&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;+354 s&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;FailedScheduling&lt;/code&gt; for the provisioner replacement: &lt;code&gt;0/6 nodes are available: 2 node(s) didn't match pod anti-affinity rules, 4 node(s) had untolerated taint(s)&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;+355 s&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;rs-example&lt;/code&gt;, &lt;code&gt;proof/zonal-1&lt;/code&gt; and the provisioner replica evicted; &lt;code&gt;rs-example&lt;/code&gt; &lt;code&gt;Running&lt;/code&gt; in zone-c at +364 s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;+619 s&lt;/td&gt;
&lt;td&gt;restore job finished (1.2 s); worker &lt;code&gt;Ready&lt;/code&gt; at +624 s, control plane at +628 s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;+622 s&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;FailedMount&lt;/code&gt; for &lt;code&gt;web-sts-1&lt;/code&gt;: &lt;code&gt;attacher.MountDevice failed to create newCsiDriverClient&lt;/code&gt; (the node plugin was not registered yet)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;+626 to +631 s&lt;/td&gt;
&lt;td&gt;DaemonSet pods and the provisioner replica &lt;code&gt;Running&lt;/code&gt; again; with the stateless pods already replaced, 6 of 8 tracked pods counted as recovered&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;+655 s&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;BackOff&lt;/code&gt; restarting the &lt;code&gt;csi-rbdplugin&lt;/code&gt;, &lt;code&gt;driver-registrar&lt;/code&gt; and &lt;code&gt;liveness-prometheus&lt;/code&gt; containers of the zone-b node plugin&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;until the watch ended, about 10 min after the restore&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;web-sts-1&lt;/code&gt; and &lt;code&gt;proof/zonal-1&lt;/code&gt; did not recover on their own&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Root causes, found afterwards: Docker gave the two zone-b containers each other's IP addresses on restart, which left the zone-b etcd member unhealthy (2 of 3), and two kernel RBD devices were stuck (details below). The manual repair took several attempts: containers that did not stop cleanly (&lt;code&gt;did not receive an exit event&lt;/code&gt;), &lt;code&gt;Address already in use&lt;/code&gt; on reconnect, failing &lt;code&gt;runc exec&lt;/code&gt;, and I/O errors while force-removing the RBD devices. It ended with the stale devices removed, the IPs back in place, 6 of 6 nodes &lt;code&gt;Ready&lt;/code&gt;, etcd 3 of 3 healthy, and all PVCs bound to their original zone clusters.&lt;/p&gt;

&lt;h3&gt;
  
  
  Run 2 in detail (helper v2: IP pinning and krbd reaper)
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Time&lt;/th&gt;
&lt;th&gt;Observation&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;+46 s&lt;/td&gt;
&lt;td&gt;worker &lt;code&gt;NotReady&lt;/code&gt; and tainted; control plane &lt;code&gt;NotReady&lt;/code&gt; at +50 s, taint at +52 s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;+76 s&lt;/td&gt;
&lt;td&gt;both &lt;code&gt;nginx&lt;/code&gt; replicas and &lt;code&gt;whoami&lt;/code&gt; evicted; replacements &lt;code&gt;Running&lt;/code&gt; at +78 s (&lt;code&gt;nginx&lt;/code&gt;: one in zone-c, one in zone-a) and +80 s (&lt;code&gt;whoami&lt;/code&gt;, zone-c). The &lt;code&gt;nginx&lt;/code&gt; Service had ready endpoints in zones a and c throughout.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;+345 s&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;TaintManagerEviction&lt;/code&gt; and &lt;code&gt;FailedScheduling&lt;/code&gt; for the provisioner replacement (same anti-affinity message as run 1)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;+347 s&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;proof/zonal-1&lt;/code&gt; and the provisioner replica evicted; &lt;code&gt;proof/zonal-1&lt;/code&gt; waiting&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;about +618 s&lt;/td&gt;
&lt;td&gt;restore job started; the reaper found the stale devices, refused one orphan network namespace because it had other interfaces, revived the other, and removed 2 of 2 devices in 2 s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;+624 s&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;Starting kubelet&lt;/code&gt; on both zone-b nodes; worker &lt;code&gt;NodeReady&lt;/code&gt; at +625 s, control plane at +631 s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;+626 s&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;FailedMount&lt;/code&gt; for &lt;code&gt;web-sts-1&lt;/code&gt; (&lt;code&gt;newCsiDriverClient&lt;/code&gt;, same as run 1)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;+627 s&lt;/td&gt;
&lt;td&gt;provisioner replacement scheduled back to the zone-b worker, &lt;code&gt;Running&lt;/code&gt; at +630 s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;+659 s&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;BackOff&lt;/code&gt; on the zone-b node plugin containers; they started again at +671 s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;+680 s&lt;/td&gt;
&lt;td&gt;restore job returned (61.2 s in total)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;+690 s&lt;/td&gt;
&lt;td&gt;the StatefulSet created a new &lt;code&gt;web-sts-1&lt;/code&gt;, scheduled to the zone-b worker; started at +694 s; &lt;code&gt;proof/zonal-1&lt;/code&gt; &lt;code&gt;Running&lt;/code&gt; at +694 s, data intact&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The job log does not break down the 61 s. The reaper reported 2 s, and the kubelets started about 6 s into the job, so most of the time came after &lt;code&gt;docker start&lt;/code&gt;. The helper then retries &lt;code&gt;mount -o remount,rw /sys&lt;/code&gt; inside each node, up to 20 times with a 1 s pause; that loop is the likely but unconfirmed cause. The PVC-backed pods were back about 70 s after the worker reported &lt;code&gt;Ready&lt;/code&gt;. Most of that gap overlapped the node plugin's restart back-off.&lt;/p&gt;

&lt;h3&gt;
  
  
  Run 3 (quick check of helper v2)
&lt;/h3&gt;

&lt;p&gt;A short down/up cycle (about 71 s between the stop and the start command) to exercise the reaper on a fresh orphan namespace. The reaper revived one orphan namespace, waited 64 s for pending I/O to drain, and removed 2 devices; the whole &lt;code&gt;up&lt;/code&gt; took 65.5 s. Both nodes came back with their original IPs.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why these timings would differ on real hardware
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Node failure detection is a controller setting.&lt;/strong&gt; kube-controller-manager marks a node unhealthy after &lt;code&gt;--node-monitor-grace-period&lt;/code&gt; (default 50 s) and checks every &lt;code&gt;--node-monitor-period&lt;/code&gt; (default 5 s) (&lt;a href="https://kubernetes.io/docs/reference/command-line-tools-reference/kube-controller-manager/" rel="noopener noreferrer"&gt;reference&lt;/a&gt;). The kind config set neither, so the observed +46 to +50 s are consistent with the default. Since t = 0 is the stop command, not the last heartbeat, this is an explanation of the numbers, not a separate measurement.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Eviction is &lt;code&gt;NotReady&lt;/code&gt; plus &lt;code&gt;tolerationSeconds&lt;/code&gt;.&lt;/strong&gt; The DefaultTolerationSeconds admission plugin adds 300 s tolerations for &lt;code&gt;not-ready&lt;/code&gt; and &lt;code&gt;unreachable&lt;/code&gt; to pods that do not set them (&lt;a href="https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/#taint-based-evictions" rel="noopener noreferrer"&gt;taint-based evictions&lt;/a&gt;). Run 1's 30 s pods were evicted at +85 s, about 8 s later than +47 plus 30.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Replacement start time depends on the host.&lt;/strong&gt; Images were already present and the nodes shared one kernel and disk. On real nodes, image pulls, CNI and real network latency add time.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A real outage looks different.&lt;/strong&gt; A power loss or network partition does not send SIGTERM. A partition can also leave the "dead" zone running and writing, which this test never exercised.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;One shared VM&lt;/strong&gt; means the surviving zones also carried the lab's other load. Neither run was isolated from the other.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Problems found and how they were fixed
&lt;/h2&gt;

&lt;h3&gt;
  
  
  1. Docker swapped node IPs and etcd became unhealthy
&lt;/h3&gt;

&lt;p&gt;When zone-b's containers restarted, Docker assigned them each other's addresses. kind rewrote its manifests for the new IPs, but the etcd member's peer URL and certificates still used the old one: TLS SAN mismatch, member unhealthy, cluster on 2 of 3. Another failure would have cost quorum.&lt;/p&gt;

&lt;p&gt;Fix: on &lt;code&gt;down&lt;/code&gt;, record each node's IP and pin it with &lt;code&gt;docker network connect --ip&lt;/code&gt; while the container is stopped. After runs 2 and 3, etcd was 3 of 3 healthy and the addresses did not change. The relevant part of the helper (names generalized):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;NET&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;lab
ipof&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="o"&gt;{&lt;/span&gt; docker inspect -f &lt;span class="s2"&gt;"{{(index .NetworkSettings.Networks \"$NET\").IPAddress}}"&lt;/span&gt; &lt;span class="nv"&gt;$1&lt;/span&gt; 2&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;/dev/null&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="o"&gt;}&lt;/span&gt;
pin&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="o"&gt;{&lt;/span&gt; &lt;span class="c"&gt;# $1 container (stopped) $2 ip&lt;/span&gt;
  &lt;span class="nv"&gt;cur&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$(&lt;/span&gt;docker inspect -f &lt;span class="s2"&gt;"{{with (index .NetworkSettings.Networks \"$NET\").IPAMConfig}}{{.IPv4Address}}{{end}}"&lt;/span&gt; &lt;span class="nv"&gt;$1&lt;/span&gt; 2&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;/dev/null&lt;span class="o"&gt;)&lt;/span&gt;
  &lt;span class="o"&gt;[&lt;/span&gt; &lt;span class="s2"&gt;"$cur"&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"$2"&lt;/span&gt; &lt;span class="o"&gt;]&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="nb"&gt;return&lt;/span&gt; 0
  docker network disconnect &lt;span class="nv"&gt;$NET&lt;/span&gt; &lt;span class="nv"&gt;$1&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; docker network connect --ip &lt;span class="nv"&gt;$2&lt;/span&gt; &lt;span class="nv"&gt;$NET&lt;/span&gt; &lt;span class="nv"&gt;$1&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"$1 pinned to $2"&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="c"&gt;# down: refuse if another control plane is already stopped, record IPs, stop, pin&lt;/span&gt;
&lt;span class="c"&gt;# up:   pin, run the krbd reaper, docker start, remount /sys rw for krbd&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;/p&gt;
  Full zone-ctl helper (v2, sanitized)
  &lt;br&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;#!/bin/bash&lt;/span&gt;
&lt;span class="c"&gt;# SSH forced command used by the lab dashboard; never a shell. v2: IP pinning + krbd reap.&lt;/span&gt;
&lt;span class="nb"&gt;set&lt;/span&gt; -uo pipefail
&lt;span class="nb"&gt;read&lt;/span&gt; -r op z extra &lt;span class="o"&gt;&amp;lt;&amp;lt;&amp;lt;&lt;/span&gt;&lt;span class="s2"&gt;"${SSH_ORIGINAL_COMMAND:-}"&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt; -z &lt;span class="s2"&gt;"${extra:-}"&lt;/span&gt; &lt;span class="o"&gt;]&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt; &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"too many args"&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="nb"&gt;exit&lt;/span&gt; 2&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="k"&gt;case&lt;/span&gt; &lt;span class="s2"&gt;"${z:-}"&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; a&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="nv"&gt;N&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"lab-control-plane lab-worker"&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; b&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="nv"&gt;N&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"lab-control-plane2 lab-worker2"&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; c&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="nv"&gt;N&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"lab-control-plane3 lab-worker3"&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; *&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"zone must be a|b|c"&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="nb"&gt;exit&lt;/span&gt; 2&lt;span class="o"&gt;;&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;esac&lt;/span&gt;
logger -t zone-ctl &lt;span class="s2"&gt;"request: $op $z from ${SSH_CLIENT%% *}"&lt;/span&gt;
&lt;span class="nv"&gt;S&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;/var/lib/zone-ctl&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="nv"&gt;NET&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;lab
ipof&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="o"&gt;{&lt;/span&gt; docker inspect -f &lt;span class="s2"&gt;"{{(index .NetworkSettings.Networks \"$NET\").IPAddress}}"&lt;/span&gt; &lt;span class="nv"&gt;$1&lt;/span&gt; 2&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;/dev/null&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="o"&gt;}&lt;/span&gt;
pin&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="o"&gt;{&lt;/span&gt; &lt;span class="c"&gt;# $1 container (stopped) $2 ip&lt;/span&gt;
  &lt;span class="nv"&gt;cur&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$(&lt;/span&gt;docker inspect -f &lt;span class="s2"&gt;"{{with (index .NetworkSettings.Networks \"$NET\").IPAMConfig}}{{.IPv4Address}}{{end}}"&lt;/span&gt; &lt;span class="nv"&gt;$1&lt;/span&gt; 2&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;/dev/null&lt;span class="o"&gt;)&lt;/span&gt;
  &lt;span class="o"&gt;[&lt;/span&gt; &lt;span class="s2"&gt;"$cur"&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"$2"&lt;/span&gt; &lt;span class="o"&gt;]&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="nb"&gt;return&lt;/span&gt; 0
  docker network disconnect &lt;span class="nv"&gt;$NET&lt;/span&gt; &lt;span class="nv"&gt;$1&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; docker network connect --ip &lt;span class="nv"&gt;$2&lt;/span&gt; &lt;span class="nv"&gt;$NET&lt;/span&gt; &lt;span class="nv"&gt;$1&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"$1 pinned to $2"&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="k"&gt;case&lt;/span&gt; &lt;span class="s2"&gt;"${op:-}"&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt;
  down&lt;span class="o"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; cp &lt;span class="k"&gt;in&lt;/span&gt; lab-control-plane lab-control-plane2 lab-control-plane3&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt;
      &lt;span class="k"&gt;case&lt;/span&gt; &lt;span class="s2"&gt;" $N "&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; *&lt;span class="s2"&gt;" $cp "&lt;/span&gt;*&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="nb"&gt;continue&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;esac&lt;/span&gt;
      &lt;span class="o"&gt;[&lt;/span&gt; &lt;span class="s2"&gt;"$(docker inspect -f '{{.State.Running}}' $cp 2&amp;gt;/dev/null)"&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt; &lt;span class="o"&gt;]&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt; &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"refused: $cp is already down (etcd quorum would be lost)"&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="nb"&gt;exit&lt;/span&gt; 3&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="o"&gt;}&lt;/span&gt;
    &lt;span class="k"&gt;done&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; n &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nv"&gt;$N&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt; &lt;span class="nv"&gt;i&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$(&lt;/span&gt;ipof &lt;span class="nv"&gt;$n&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="o"&gt;[&lt;/span&gt; -n &lt;span class="s2"&gt;"$i"&lt;/span&gt; &lt;span class="o"&gt;]&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="o"&gt;[&lt;/span&gt; &lt;span class="s2"&gt;"$i"&lt;/span&gt; != &lt;span class="s2"&gt;"invalid IP"&lt;/span&gt; &lt;span class="o"&gt;]&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"$i"&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="nv"&gt;$S&lt;/span&gt;/&lt;span class="nv"&gt;$n&lt;/span&gt;.ip&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;done&lt;/span&gt;
    docker stop -t 10 &lt;span class="nv"&gt;$N&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; n &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nv"&gt;$N&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt; &lt;span class="o"&gt;[&lt;/span&gt; -s &lt;span class="nv"&gt;$S&lt;/span&gt;/&lt;span class="nv"&gt;$n&lt;/span&gt;.ip &lt;span class="o"&gt;]&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; pin &lt;span class="nv"&gt;$n&lt;/span&gt; &lt;span class="s2"&gt;"$(cat $S/$n.ip)"&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;done&lt;/span&gt; &lt;span class="o"&gt;;&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
  up&lt;span class="o"&gt;)&lt;/span&gt;
    &lt;span class="nv"&gt;IPS&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;""&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; n &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nv"&gt;$N&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt; &lt;span class="o"&gt;[&lt;/span&gt; -s &lt;span class="nv"&gt;$S&lt;/span&gt;/&lt;span class="nv"&gt;$n&lt;/span&gt;.ip &lt;span class="o"&gt;]&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt; pin &lt;span class="nv"&gt;$n&lt;/span&gt; &lt;span class="s2"&gt;"$(cat $S/$n.ip)"&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="nv"&gt;IPS&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"$IPS $(cat $S/$n.ip)"&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="o"&gt;}&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;done&lt;/span&gt;
    &lt;span class="o"&gt;[&lt;/span&gt; -n &lt;span class="s2"&gt;"$IPS"&lt;/span&gt; &lt;span class="o"&gt;]&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; /usr/local/lib/zone-ctl/krbd-reap.py &lt;span class="nv"&gt;$IPS&lt;/span&gt;
    docker start &lt;span class="nv"&gt;$N&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; n &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nv"&gt;$N&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; i &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nv"&gt;$(&lt;/span&gt;seq 1 20&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt; docker &lt;span class="nb"&gt;exec&lt;/span&gt; &lt;span class="nv"&gt;$n&lt;/span&gt; mount -o remount,rw /sys 2&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;/dev/null &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="nb"&gt;break&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; sleep 1&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;done&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;done&lt;/span&gt;
    &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"/sys remounted rw (krbd)"&lt;/span&gt; &lt;span class="o"&gt;;&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
  status&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;;&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
  *&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"op must be down|up|status"&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="nb"&gt;exit&lt;/span&gt; 2 &lt;span class="o"&gt;;&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;esac&lt;/span&gt;
docker inspect -f &lt;span class="s1"&gt;'{{.Name}} {{.State.Status}} {{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}'&lt;/span&gt; &lt;span class="nv"&gt;$N&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;




&lt;p&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  2. Stale kernel RBD mappings
&lt;/h3&gt;

&lt;p&gt;kind nodes are containers on the host kernel, so &lt;code&gt;/dev/rbdN&lt;/code&gt; is a host kernel device. When the zone-b worker stopped, the libceph client of two RBD devices stayed in that node's network namespace, which no longer had any interfaces. Reconnects failed with &lt;code&gt;ENETUNREACH&lt;/code&gt;. After restart, the node plugin reused those devices, &lt;code&gt;NodeStage&lt;/code&gt; hung with &lt;code&gt;blkid&lt;/code&gt; in D state, and a forced &lt;code&gt;rbd remove&lt;/code&gt; hung too.&lt;/p&gt;

&lt;p&gt;Fix: before starting the nodes, a reaper finds RBD devices whose client address belongs to a stopped node, refuses to act if a running container owns that address, finds orphan network namespaces (never the host namespace or one with other interfaces), gives each a temporary veth on the lab bridge so pending I/O can drain, force-removes the devices, and removes the veths. In run 2 this took 2 s; in run 3, 64 s.&lt;/p&gt;

&lt;p&gt;&lt;/p&gt;
  krbd-reap.py (sanitized: bridge name and address range are placeholders)
  &lt;br&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="c1"&gt;#!/usr/bin/env python3&lt;/span&gt;
&lt;span class="s2"&gt;"""Reap krbd mappings whose libceph client lives in the netns of a STOPPED kind node. usage: krbd-reap.py &amp;lt;node-ip&amp;gt;..."""&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;fcntl&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;glob&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;socket&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;struct&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;subprocess&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;
&lt;span class="no"&gt;IPS&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kt"&gt;set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;argv&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="no"&gt;BRIDGE&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"br-lab"&lt;/span&gt;        &lt;span class="c1"&gt;# the docker bridge the kind nodes are attached to&lt;/span&gt;
&lt;span class="no"&gt;TMP_NET&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"192.0.2"&lt;/span&gt;      &lt;span class="c1"&gt;# placeholder: a free address range on that bridge&lt;/span&gt;
&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;sh&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ok&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;False&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;subprocess&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;run&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;capture_output&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;returncode&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;ok&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;RuntimeError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"{' '.join(c)}: {r.stderr.strip()}"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;stdout&lt;/span&gt;
&lt;span class="no"&gt;T0&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;time&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;say&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"krbd[+{int(time.time() - T0)}s]:"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;flush&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;pending&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;l&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;sh&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"ps"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"-eo"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"pid,args"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;splitlines&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="s2"&gt;"remove_single_major"&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="s2"&gt;"sh -c"&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="s2"&gt;"timeout"&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="n"&gt;devs&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;basename&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="s2"&gt;"/name"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;read&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;glob&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;glob&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"/sys/bus/rbd/devices/*"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nb"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="s2"&gt;"/client_addr"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;read&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;split&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;":"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="no"&gt;IPS&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;devs&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="nf"&gt;pending&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;say&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"no stale mappings"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;exit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;live&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;sh&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"docker"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"ps"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"-q"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;split&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;busy&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kt"&gt;set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;sh&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"docker"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"inspect"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"-f"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"{{range .NetworkSettings.Networks}}{{.IPAddress}} {{end}}"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;live&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;split&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;live&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="kt"&gt;set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="no"&gt;IPS&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&lt;/span&gt; &lt;span class="n"&gt;busy&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;say&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"refused: a RUNNING container owns"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;" "&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;sorted&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="no"&gt;IPS&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&lt;/span&gt; &lt;span class="n"&gt;busy&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"- mappings are live, not stale"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;exit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="nf"&gt;say&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"stale mappings from"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;" "&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;sorted&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="no"&gt;IPS&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="s2"&gt;":"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;" "&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"rbd{k}={v}"&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;devs&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;items&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="s2"&gt;"-"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"| pending removes:"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;pending&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;used&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kt"&gt;set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;__import__&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"re"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;findall&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="s2"&gt;"link-netnsid (\d+)"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;sh&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"ip"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"-o"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"-d"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"link"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"show"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;named&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kt"&gt;set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;makedirs&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"/run/netns"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;exist_ok&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;glob&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;glob&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"/run/docker/netns/*"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"/proc/1/ns/net"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;  &lt;span class="c1"&gt;# live docker netns + the host netns itself&lt;/span&gt;
    &lt;span class="n"&gt;n&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"czc-"&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;basename&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;startswith&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"/run"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="s2"&gt;"host"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"/run/netns/"&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="nb"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"a"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;close&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="nf"&gt;sh&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"mount"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"--bind"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ok&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;sh&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"ip"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"netns"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"list-id"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;splitlines&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="s2"&gt;"iproute2 netns name"&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;named&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;split&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;allids&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;split&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;sh&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"ip"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"netns"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"list-id"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;splitlines&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;startswith&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"nsid"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;glob&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;glob&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"/run/netns/czc-*"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;sh&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"umount"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ok&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;unlink&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;orphans&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;allids&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;used&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;named&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="nf"&gt;say&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"orphan netns ids:"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;orphans&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="s2"&gt;"none"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;a&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;4&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;struct&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;pack&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"HH"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="s2"&gt;"\0"&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt; &lt;span class="o"&gt;%&lt;/span&gt; &lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;%&lt;/span&gt; &lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;s&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;encode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="s2"&gt;"\0"&lt;/span&gt;
&lt;span class="n"&gt;ifi&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;lambda&lt;/span&gt; &lt;span class="n"&gt;idx&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;fl&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ch&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;struct&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;pack&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"BxHiII"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;idx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;fl&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ch&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;nl&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;flags&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;sk&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;socket&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;socket&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;socket&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;AF_NETLINK&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;socket&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;SOCK_RAW&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;sk&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;bind&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;sk&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;send&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;struct&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;pack&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"IHHII"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;16&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;flags&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;sk&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;recv&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;65536&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;sk&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;close&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;struct&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;unpack_from&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"H"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;e&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;struct&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;unpack_from&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"i"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;16&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;OSError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;strerror&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;revived&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;nsid&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;enumerate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;orphans&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;hv&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;handle&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"vreap{nsid}"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"/run/netns/reap{nsid}"&lt;/span&gt;
    &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;peer&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;ifi&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="nf"&gt;a&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;s&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;hv&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="nf"&gt;a&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;19&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;struct&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;pack&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"I"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;getpid&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="nf"&gt;nl&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;16&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="mi"&gt;4&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="mh"&gt;0x400&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="mh"&gt;0x200&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;ifi&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="nf"&gt;a&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;46&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;struct&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;pack&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"i"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;nsid&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="nf"&gt;a&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;s&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"eth9"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt;
           &lt;span class="nf"&gt;a&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;18&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;a&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;s&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"veth"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="nf"&gt;a&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;a&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;peer&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;                       &lt;span class="c1"&gt;# veth eth9 (UP) inside, peer in host&lt;/span&gt;
        &lt;span class="n"&gt;fd&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nb"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"/dev/net/tun"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;O_RDWR&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;tap&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"tapreap{nsid}"&lt;/span&gt;
        &lt;span class="n"&gt;fcntl&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;ioctl&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;fd&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mh"&gt;0x400454ca&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;struct&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;pack&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"16sH"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tap&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;encode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mh"&gt;0x0002&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="mh"&gt;0x1000&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="nf"&gt;nl&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;16&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;ifi&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;socket&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;if_nametoindex&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tap&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="nf"&gt;a&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;46&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;struct&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;pack&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"i"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;nsid&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;  &lt;span class="c1"&gt;# move tap into it&lt;/span&gt;
        &lt;span class="n"&gt;nsfd&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;fcntl&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;ioctl&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;fd&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mh"&gt;0x54e3&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;                                       &lt;span class="c1"&gt;# TUNGETDEVNETNS -&amp;gt; netns fd&lt;/span&gt;
        &lt;span class="nb"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;handle&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"a"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;close&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="nf"&gt;sh&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"mount"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"--bind"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"/proc/{os.getpid()}/fd/{nsfd}"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;handle&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;close&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;nsfd&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;close&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;fd&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;links&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;split&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;":"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;split&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"@"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;sh&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"ip"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"netns"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"exec"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"reap{nsid}"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"ip"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"-o"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"link"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;splitlines&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="kt"&gt;set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;links&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"lo"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"eth9"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tap&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;RuntimeError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"nsid {nsid} has other links {links}; not touching it"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;ip&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"{TMP_NET}.{20 + k}"&lt;/span&gt;
        &lt;span class="nf"&gt;sh&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"ip"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"netns"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"exec"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"reap{nsid}"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"ip"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"link"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"set"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"lo"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"up"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="nf"&gt;sh&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"ip"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"netns"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"exec"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"reap{nsid}"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"ip"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"addr"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"add"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ip&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="s2"&gt;"/24"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"dev"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"eth9"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="nf"&gt;sh&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"ip"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"link"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"set"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;hv&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"master"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="no"&gt;BRIDGE&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="nf"&gt;sh&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"ip"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"link"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"set"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;hv&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"up"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="nf"&gt;say&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"nsid {nsid} revived via {hv} {ip}"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;revived&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;hv&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;handle&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="nc"&gt;Exception&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="nf"&gt;say&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"nsid {nsid}: {e}"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="nf"&gt;sh&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"ip"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"link"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"del"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;hv&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ok&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;exists&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;handle&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;sh&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"umount"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;handle&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ok&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;unlink&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;handle&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;revived&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;say&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"WARNING no orphan netns revived; not force-removing (would hang)"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;exit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;procs&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;subprocess&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Popen&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"timeout"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"150"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"sh"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"-c"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"echo '{k} force' &amp;gt; /sys/bus/rbd/remove_single_major"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;devs&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="n"&gt;t1&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;time&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;   &lt;span class="c1"&gt;# removal is done when the remove writers have returned (sysfs entries vanish before I/O drains)&lt;/span&gt;
&lt;span class="k"&gt;while&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;time&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;t1&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="mi"&gt;150&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;any&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;poll&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;is&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;procs&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="nf"&gt;pending&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sleep&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;left&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;k&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;devs&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;exists&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"/sys/bus/rbd/devices/{k}"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="nf"&gt;pending&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="nf"&gt;say&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"removed {len(devs)} device(s), drained in {int(time.time() - t1)}s"&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;", LEFT {left}"&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="s2"&gt;""&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;hv&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;handle&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;revived&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="nf"&gt;sh&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"ip"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"link"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"del"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;hv&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ok&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="nf"&gt;sh&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"umount"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;handle&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ok&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;unlink&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;handle&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="nf"&gt;say&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"temporary veths/netns handles removed"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;exit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;left&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;




&lt;p&gt;&lt;/p&gt;

&lt;p&gt;The exact failure is specific to kind, but the general case is not: a node that loses its network with live krbd mappings can come back with stuck devices. Check for them before returning a node to service.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. Hard anti-affinity and a lost zone
&lt;/h3&gt;

&lt;p&gt;The provisioner runs 3 replicas with required hostname anti-affinity. With two workers left, the third replica stayed &lt;code&gt;Pending&lt;/code&gt; for the rest of the outage. Its sidecars run with leader election (as in the upstream manifest), and the leases moved to other replicas within about 20 s in run 1, so provisioning was not affected. If you alert on &lt;code&gt;Pending&lt;/code&gt; pods in system namespaces, every zone outage will trigger that alert. Options: two replicas, or zone-based preferred anti-affinity.&lt;/p&gt;

&lt;h3&gt;
  
  
  4. &lt;code&gt;tolerationSeconds&lt;/code&gt; sets the stateless recovery time
&lt;/h3&gt;

&lt;p&gt;Failure detection was stable at about 46–50 s. The setting that changed the outcome was &lt;code&gt;tolerationSeconds&lt;/code&gt;: 30 s led to replacements at about +80–90 s, the default 300 s to about +350 s. A lower value also means pod churn on short network interruptions, so set it per workload class.&lt;/p&gt;

&lt;h3&gt;
  
  
  5. &lt;code&gt;DoNotSchedule&lt;/code&gt; did not block failover (inference)
&lt;/h3&gt;

&lt;p&gt;&lt;code&gt;nginx&lt;/code&gt; uses a hard spread constraint (&lt;code&gt;whenUnsatisfiable: DoNotSchedule&lt;/code&gt;) and &lt;code&gt;nodeTaintsPolicy: Honor&lt;/code&gt;, yet its replacements were scheduled into zones a and c. According to the &lt;a href="https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/" rel="noopener noreferrer"&gt;Kubernetes documentation&lt;/a&gt;, with &lt;code&gt;Honor&lt;/code&gt; the skew calculation only includes untainted nodes and tainted nodes the pod tolerates; with the default (&lt;code&gt;Ignore&lt;/code&gt;), all nodes count. The unreachable node also carries a &lt;code&gt;NoSchedule&lt;/code&gt; taint that the default tolerations do not cover, and the &lt;code&gt;FailedScheduling&lt;/code&gt; message above ("4 node(s) had untolerated taint(s)", which counts the three control-plane nodes plus the zone-b worker) shows that the scheduler saw that taint.&lt;/p&gt;

&lt;p&gt;This explanation is an inference from the manifest, the documentation and the outcome. What would confirm it:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;an A/B rerun with &lt;code&gt;nodeTaintsPolicy&lt;/code&gt; unset: if the explanation is right, the replacement should stay &lt;code&gt;Pending&lt;/code&gt; with a topology spread message;&lt;/li&gt;
&lt;li&gt;scheduler logs at higher verbosity showing which nodes enter the PodTopologySpread calculation.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Stateful workloads: what to recommend
&lt;/h2&gt;

&lt;p&gt;In this design storage does not move between zones, so a PVC-backed pod is only as available as its zone. That is a deliberate trade-off for a smaller blast radius. The platform decision is how much of the remaining work it takes on.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Recommended order for a platform team, with the conditions for each:&lt;/strong&gt;&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Default: recommended patterns with application-level replication.&lt;/strong&gt; Ship blueprints for a StatefulSet with one replica per zone (the spread block above), each on its own zone's Ceph, and replication done by the application: PostgreSQL streaming replication, MongoDB replica sets, Kafka or etcd quorums. Choose this when the data store has mature built-in replication and the owning team can operate it. The outage test showed the shape: &lt;code&gt;web-sts-0&lt;/code&gt; and &lt;code&gt;web-sts-2&lt;/code&gt; in zones a and c kept running, and only &lt;code&gt;web-sts-1&lt;/code&gt; waited.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Next: operator-managed data services.&lt;/strong&gt; The platform runs operators and owns failover, backups and replica rebuilds. Choose this when many tenant teams need databases but lack the skills to run them, and the platform team can take database on-call and SLAs. Start with one or two engines.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Only for tolerant workloads: document the limitation.&lt;/strong&gt; State clearly that a PVC is pinned to one zone and that a single-replica stateful app is down for as long as its zone is. Acceptable for dev/test, caches, rebuildable data, or apps whose recovery objective is longer than a typical zone outage.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Two cases where per-zone Ceph is the wrong tool: a single-instance application that cannot replicate and must survive a zone loss with no data loss (consider a stretched cluster, if the sites qualify), and disaster recovery to another site for such an application where some data loss is acceptable (consider RBD mirroring).&lt;/p&gt;

&lt;p&gt;Moving one volume to another zone is an explicit copy, because the clusters are not connected. On the Proxmox side, an offline move of a container's rootfs (about 517 MB of data) from zone-a to zone-b took 105.5 s and 99.5 s back, with the container stopped throughout. That is a migration with downtime.&lt;/p&gt;

&lt;h2&gt;
  
  
  Comparison: per-zone Ceph, stretched Ceph, RBD mirroring
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Not benchmarked here.&lt;/strong&gt; This table summarizes the documentation and the design properties; none of it was measured in this PoC.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Property&lt;/th&gt;
&lt;th&gt;Independent Ceph per zone (this PoC)&lt;/th&gt;
&lt;th&gt;Stretched Ceph cluster&lt;/th&gt;
&lt;th&gt;RBD mirroring between clusters&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Write path&lt;/td&gt;
&lt;td&gt;local to one zone&lt;/td&gt;
&lt;td&gt;synchronous across zones: each write is stored in both zones before it is acknowledged (&lt;a href="https://docs.ceph.com/en/latest/rados/operations/stretch-mode/" rel="noopener noreferrer"&gt;stretch mode&lt;/a&gt;)&lt;/td&gt;
&lt;td&gt;local writes, asynchronous copy (&lt;a href="https://docs.ceph.com/en/latest/rbd/rbd-mirroring/" rel="noopener noreferrer"&gt;RBD mirroring&lt;/a&gt;); journal-based mode writes every change twice, and the docs say to "expect write latencies to nearly double"&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Data loss on losing a zone (storage layer)&lt;/td&gt;
&lt;td&gt;volumes of that zone are unavailable until it returns&lt;/td&gt;
&lt;td&gt;none expected; the surviving zone serves I/O in degraded mode&lt;/td&gt;
&lt;td&gt;up to the replication lag or snapshot interval; partially applied snapshot deltas are rolled back on failover&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Volume usable in another zone&lt;/td&gt;
&lt;td&gt;no&lt;/td&gt;
&lt;td&gt;yes&lt;/td&gt;
&lt;td&gt;after failover (image promotion on the peer cluster)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Failure domain&lt;/td&gt;
&lt;td&gt;one zone&lt;/td&gt;
&lt;td&gt;one cluster spanning sites: one monitor quorum, one version, one upgrade path&lt;/td&gt;
&lt;td&gt;separate clusters&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Topology constraints&lt;/td&gt;
&lt;td&gt;none between zones&lt;/td&gt;
&lt;td&gt;stretch mode supports exactly two data zones plus a tiebreaker monitor at a third site&lt;/td&gt;
&lt;td&gt;peer clusters configured per pool; one-way or two-way&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Kubernetes integration&lt;/td&gt;
&lt;td&gt;topology-aware CSI (fork)&lt;/td&gt;
&lt;td&gt;standard ceph-csi, one cluster&lt;/td&gt;
&lt;td&gt;ceph-csi plus &lt;a href="https://github.com/csi-addons/kubernetes-csi-addons/blob/main/docs/volumereplication.md" rel="noopener noreferrer"&gt;csi-addons VolumeReplication&lt;/a&gt; for promote/demote&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Operational cost&lt;/td&gt;
&lt;td&gt;three clusters to run; no cross-zone data path&lt;/td&gt;
&lt;td&gt;sensitive to inter-site latency; tiebreaker placement&lt;/td&gt;
&lt;td&gt;mirror daemons, failover and failback procedures; data copied twice if the app already replicates&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  Benchmark plan (future work)
&lt;/h2&gt;

&lt;p&gt;I did not run fio on this PoC. All three "zones" share one VM and the same disks, so the results would say nothing about cross-zone behavior. On real per-zone hardware, the plan is:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Baseline per zone:&lt;/strong&gt; fio on an RBD PVC in each zone with &lt;code&gt;--direct=1 --ioengine=libaio --time_based --runtime=120&lt;/code&gt;, three repetitions each:

&lt;ul&gt;
&lt;li&gt;4k random write, iodepth 1: p50 and p99 completion latency;&lt;/li&gt;
&lt;li&gt;4k random read, iodepth 32: IOPS;&lt;/li&gt;
&lt;li&gt;1M sequential write: throughput.&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Network baseline:&lt;/strong&gt; RTT between zones (ping and a TCP test) recorded with each run.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Comparisons:&lt;/strong&gt; the same jobs against a stretched pool (if two qualifying sites exist) and against an image with journal-based mirroring enabled, to measure the cross-zone and journaling cost directly.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Under failure:&lt;/strong&gt; repeat the 4k write test during a MON outage and during a zone outage to see stalls, not only averages.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Next steps
&lt;/h2&gt;

&lt;p&gt;The thresholds below are proposed acceptance criteria, not results.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Repeat the outage test on real hardware&lt;/strong&gt; with separate nodes per zone, using three failure types: hard power-off, network partition between zones, and a graceful drain. Run each at least five times. Record &lt;code&gt;NotReady&lt;/code&gt; time, eviction time, replacement &lt;code&gt;Ready&lt;/code&gt; time and PVC pod recovery time after the zone returns, and report median and maximum.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Zone-recovery gate.&lt;/strong&gt; A zone counts as recovered only when etcd is 3 of 3 healthy, no &lt;code&gt;/sys/bus/rbd/devices/*/client_addr&lt;/code&gt; points at a stopped node, there are no processes in D state on the returned nodes, and every PVC-backed pod is &lt;code&gt;Running&lt;/code&gt;. Target: no manual steps in 5 of 5 runs.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Tolerations per workload class.&lt;/strong&gt; Test 30, 60 and 120 s against short network flaps (for example 10 s and 20 s link drops). Target: no evictions for flaps shorter than the chosen value, failover within value plus 60 s otherwise.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;System workloads.&lt;/strong&gt; Change the provisioner to two replicas or zone-based preferred anti-affinity. Target: no &lt;code&gt;Pending&lt;/code&gt; system pods during a single-zone outage.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Confirm the &lt;code&gt;nodeTaintsPolicy&lt;/code&gt; inference&lt;/strong&gt; with the A/B rerun described above.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;CSI strategy.&lt;/strong&gt; Track ceph/ceph-csi#5177. While on the fork, rebase onto each upstream release, and gate upgrades on the 15 zone-aware tests plus snapshot/restore and resize. The fallback is one StorageClass per zone with upstream ceph-csi.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Production-style Ceph&lt;/strong&gt; with standby MGRs, real disks and capacity sized for real load, then run the benchmark plan.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Alerting.&lt;/strong&gt; Alert when a PVC-backed pod is &lt;code&gt;Pending&lt;/code&gt; because of a volume node affinity conflict for longer than a chosen time (for example 5 minutes), and silence the expected provisioner replica during a declared zone outage.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Discussion
&lt;/h2&gt;

&lt;p&gt;If you run Ceph under a multi-zone Kubernetes cluster: did you choose a stretched cluster, RBD mirroring, or independent per-zone clusters? And what &lt;code&gt;tolerationSeconds&lt;/code&gt; do you use in production? I would like to compare notes in the comments.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Ramin Najarbashi&lt;/em&gt;&lt;/p&gt;

</description>
      <category>kubernetes</category>
      <category>ceph</category>
      <category>devops</category>
      <category>sre</category>
    </item>
    <item>
      <title>CVE-Radar: Multi-Source Open Source CVE Dashboard for DevSecOps Teams</title>
      <dc:creator>Ramin Najjarbashi</dc:creator>
      <pubDate>Thu, 18 Jun 2026 05:18:37 +0000</pubDate>
      <link>https://dev.to/raminnietzsche/stop-chasing-vulnerabilities-meet-cve-radar-your-automated-open-source-threat-companion-11jb</link>
      <guid>https://dev.to/raminnietzsche/stop-chasing-vulnerabilities-meet-cve-radar-your-automated-open-source-threat-companion-11jb</guid>
      <description>&lt;p&gt;Stop Chasing Vulnerabilities: Meet CVE-Radar, Your Self-Hosted Multi-Source CVE Monitoring Dashboard&lt;br&gt;
In the world of DevSecOps, staying on top of vulnerabilities is exhausting. Thousands of new CVEs drop every week, but most tools either overwhelm you with noise or miss what actually matters to your stack.&lt;br&gt;
That’s why I built CVE-Radar — a powerful, beautiful, and fully open-source vulnerability radar that aggregates multiple trusted sources and shows you exactly what you need to care about.&lt;/p&gt;


&lt;h2&gt;
  
  
  What is CVE-Radar?
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;CVE-Radar&lt;/strong&gt; is an open-source, self-hosted vulnerability intelligence platform that aggregates multiple authoritative sources (NVD, OSV, GitHub, CISA KEV, and major Linux distros) to give you a clear, real-time view of which CVEs actually affect your infrastructure.&lt;/p&gt;

&lt;p&gt;Think of it as a smart radar for your entire tech stack — with a modern dashboard, powerful filtering, trend charts, per-tool insights, and multi-channel alerting.&lt;/p&gt;

&lt;p&gt;Built with React, TypeScript, Express, and PostgreSQL, it's production-ready, fully Dockerized, and supports both individual developers and enterprise self-hosted environments.&lt;/p&gt;
&lt;h2&gt;
  
  
  &lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fdz9ugnf4zlurm5kqb0tj.png" alt="CVE-Radar Dashboard - Overview with severity donut chart and vulnerability list" width="800" height="1074"&gt;
&lt;/h2&gt;
&lt;h2&gt;
  
  
  The Architecture &amp;amp; Recent Updates
&lt;/h2&gt;

&lt;p&gt;CVE-Radar is built with a clean, modern full-stack architecture:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Frontend&lt;/strong&gt;: React + Vite + TypeScript + Tailwind&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Backend&lt;/strong&gt;: Express.js + Drizzle ORM&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Database&lt;/strong&gt;: PostgreSQL + Redis caching&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Deployment&lt;/strong&gt;: Docker Compose (one-command setup) and Kubernetes friendly&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Monitoring&lt;/strong&gt;: Prometheus metrics&lt;/li&gt;
&lt;/ul&gt;
&lt;h3&gt;
  
  
  Recent Highlights
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;Major UI/UX improvements to the notification system&lt;/li&gt;
&lt;li&gt;Stronger multi-source data aggregation and synchronization&lt;/li&gt;
&lt;li&gt;Enhanced dashboard charts and filtering&lt;/li&gt;
&lt;li&gt;Better internationalization and RTL support&lt;/li&gt;
&lt;li&gt;Overall performance and stability improvements&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The codebase is well-structured, documented, and contributor-friendly.&lt;/p&gt;


&lt;h2&gt;
  
  
  Quick Start: Give It a Spin
&lt;/h2&gt;

&lt;p&gt;Getting CVE-Radar up and running is fast and straightforward. The project includes a complete Docker-based setup with helpful Makefiles.&lt;/p&gt;
&lt;h3&gt;
  
  
  Installation Steps
&lt;/h3&gt;


&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# Clone the repository&lt;/span&gt;
git clone https://github.com/RaminNietzsche/CVE-Radar.git
&lt;span class="nb"&gt;cd &lt;/span&gt;CVE-Radar

&lt;span class="c"&gt;# Install dependencies and start development environment&lt;/span&gt;
make setup
make dev
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;

&lt;h3&gt;
  
  
  After Running
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Web UI&lt;/strong&gt; → &lt;a href="http://localhost:5173" rel="noopener noreferrer"&gt;http://localhost:5173&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Backend API&lt;/strong&gt; → &lt;a href="http://localhost:3001" rel="noopener noreferrer"&gt;http://localhost:3001&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Default admin credentials and further instructions will appear in the terminal.&lt;/p&gt;
&lt;h3&gt;
  
  
  Recommended Enhancements
&lt;/h3&gt;

&lt;p&gt;Add these to your &lt;code&gt;.env&lt;/code&gt; file for optimal performance:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;NVD_API_KEY=your_nvd_key_here
GITHUB_TOKEN=your_github_token_here
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Production Mode
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;make build
make up
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For advanced deployment options (Kubernetes, Air-gap, custom domains, etc.), &lt;a href="http://blog.raminnietzsche.ir/CVE-Radar/" rel="noopener noreferrer"&gt;check the full documentation&lt;/a&gt; in the repository.&lt;/p&gt;




&lt;h2&gt;
  
  
  Open Source &amp;amp; Proud: How You Can Help
&lt;/h2&gt;

&lt;p&gt;CVE-Radar is fully open source under the MIT license, and I built it to be &lt;strong&gt;community-driven&lt;/strong&gt; from day one.&lt;/p&gt;

&lt;p&gt;Whether you're a developer, security engineer, DevOps practitioner, or just someone who cares about better vulnerability management tools — there are many ways to contribute:&lt;/p&gt;

&lt;h3&gt;
  
  
  Ways to Get Involved
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Star the Repository&lt;/strong&gt; — It helps with visibility and shows support ⭐&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Try it out&lt;/strong&gt; and share your feedback (open an Issue or start a Discussion)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Report bugs&lt;/strong&gt; or suggest new features&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Contribute code&lt;/strong&gt; — The project has several “good first issue” labels for beginners&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Add new features&lt;/strong&gt; like:

&lt;ul&gt;
&lt;li&gt;More notification channels&lt;/li&gt;
&lt;li&gt;Additional vulnerability sources&lt;/li&gt;
&lt;li&gt;Improved matching algorithms&lt;/li&gt;
&lt;li&gt;Better Kubernetes integration&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Improve documentation&lt;/strong&gt; or write blog posts/tutorials&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Spread the word&lt;/strong&gt; in your community, Discord servers, or workplace&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Current Status
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Latest Version&lt;/strong&gt;: v1.4.0&lt;/li&gt;
&lt;li&gt;Actively maintained with regular updates&lt;/li&gt;
&lt;li&gt;Contributor-friendly codebase with clear architecture and good documentation&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Every contribution — no matter how small — helps make CVE-Radar more powerful for the entire DevSecOps community.&lt;/p&gt;




&lt;p&gt;&lt;strong&gt;Ready to join?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;👉 Check out the project here: &lt;strong&gt;&lt;a href="https://github.com/RaminNietzsche/CVE-Radar" rel="noopener noreferrer"&gt;github.com/RaminNietzsche/CVE-Radar&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Feel free to drop your thoughts, questions, or ideas in the comments below. What features would you like to see next? Let's build something great together! 🚀&lt;/p&gt;




&lt;h2&gt;
  
  
  Let’s Chat!
&lt;/h2&gt;

&lt;p&gt;I’d love to hear from you!&lt;/p&gt;

&lt;p&gt;Whether you’ve tried CVE-Radar, have feedback, found a bug, or just want to share how you handle vulnerability management in your organization — drop a comment below.&lt;/p&gt;

&lt;h3&gt;
  
  
  Questions I’d especially like your thoughts on:
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;What’s your biggest pain point with CVE monitoring today?&lt;/li&gt;
&lt;li&gt;Which features would make this tool more valuable for your workflow?&lt;/li&gt;
&lt;li&gt;Have you used similar tools (like Dependency-Track, Trivy, or Grype)? How does CVE-Radar compare?&lt;/li&gt;
&lt;li&gt;Any new data sources or notification channels you’d like to see added?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Your feedback directly shapes the future of the project.&lt;/p&gt;




&lt;p&gt;&lt;strong&gt;Thank you for reading!&lt;/strong&gt; ❤️&lt;/p&gt;

&lt;p&gt;If you found this useful, please consider:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;⭐ Starring the repo&lt;/li&gt;
&lt;li&gt;Sharing it with your team or network&lt;/li&gt;
&lt;li&gt;Trying it out and letting me know how it goes&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;👉 &lt;strong&gt;&lt;a href="https://github.com/RaminNietzsche/CVE-Radar" rel="noopener noreferrer"&gt;GitHub Repository&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Looking forward to your comments and contributions! 🚀&lt;/p&gt;

</description>
      <category>automation</category>
      <category>opensource</category>
      <category>security</category>
      <category>showdev</category>
    </item>
    <item>
      <title>The Great Orphan Object Hunt: How I Fixed Our Cloud Storage Woes</title>
      <dc:creator>Ramin Najjarbashi</dc:creator>
      <pubDate>Mon, 27 Mar 2023 21:27:41 +0000</pubDate>
      <link>https://dev.to/raminnietzsche/the-great-orphan-object-hunt-how-i-fixed-our-cloud-storage-woes-2gni</link>
      <guid>https://dev.to/raminnietzsche/the-great-orphan-object-hunt-how-i-fixed-our-cloud-storage-woes-2gni</guid>
      <description>&lt;p&gt;As an SRE working with a cloud provider and object storage, I've learned firsthand how tricky it can be to keep a Ceph cluster running smoothly. And while most of the time, everything hums along just fine, every once in a while, we encounter a problem that leaves us scratching our heads.&lt;/p&gt;

&lt;p&gt;Recently, we started getting calls from users who were confused about their usage metrics. They were seeing that the sum of all objects in their buckets was many fewer than what we were showing in our panel, and naturally, they were concerned. After some digging, we discovered that there were orphan objects stuck in the cluster - objects that couldn't be shown by any client, like s3cmd, boto3, or mc.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://res.cloudinary.com/practicaldev/image/fetch/s--EZb7ZmsL--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/63cpzo9gd54cs5bbdyzx.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://res.cloudinary.com/practicaldev/image/fetch/s--EZb7ZmsL--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/63cpzo9gd54cs5bbdyzx.jpg" alt="Oh shit! something wrong" width="563" height="665"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Now, with billions of objects in our cluster, finding and removing these orphan objects was no easy task. I tried using normal tools, but they just weren't up to the job. So, I had to get creative.&lt;/p&gt;

&lt;p&gt;I started by creating a bash script that would help me identify and remove these orphan objects. Here's the code:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;#!/bin/bash&lt;/span&gt;

&lt;span class="c"&gt;#set -e&lt;/span&gt;

&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;[&lt;/span&gt; &lt;span class="nt"&gt;-z&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$1&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="o"&gt;]&lt;/span&gt;
  &lt;span class="k"&gt;then
    &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"No bucket name provided"&lt;/span&gt;
    &lt;span class="nb"&gt;exit &lt;/span&gt;1
&lt;span class="k"&gt;fi
&lt;/span&gt;&lt;span class="nv"&gt;BUCKET&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$1&lt;/span&gt;

&lt;span class="nv"&gt;BUCKET_ID&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;radosgw-admin bucket stats &lt;span class="nt"&gt;-b&lt;/span&gt; &lt;span class="nv"&gt;$BUCKET&lt;/span&gt; | jq &lt;span class="nt"&gt;-r&lt;/span&gt; &lt;span class="s1"&gt;'.id'&lt;/span&gt;&lt;span class="si"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;[&lt;/span&gt; &lt;span class="nt"&gt;-z&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$BUCKET_ID&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="o"&gt;]&lt;/span&gt;
  &lt;span class="k"&gt;then
    &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"Failed to get bucket ID for &lt;/span&gt;&lt;span class="nv"&gt;$BUCKET&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
    &lt;span class="nb"&gt;exit &lt;/span&gt;1
&lt;span class="k"&gt;fi

&lt;/span&gt;&lt;span class="nv"&gt;INDEX_POOL&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;radosgw-admin zone get | jq &lt;span class="s1"&gt;'.placement_pools[].val.index_pool'&lt;/span&gt; &lt;span class="nt"&gt;-r&lt;/span&gt;&lt;span class="si"&gt;)&lt;/span&gt;
&lt;span class="nv"&gt;DATA_POOL&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;radosgw-admin zone get | jq &lt;span class="s1"&gt;'.placement_pools[].val.storage_classes[].data_pool'&lt;/span&gt; &lt;span class="nt"&gt;-r&lt;/span&gt;&lt;span class="si"&gt;)&lt;/span&gt;
&lt;span class="nv"&gt;ORPHANS&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;radosgw-admin bucket check &lt;span class="nt"&gt;-b&lt;/span&gt; &lt;span class="nv"&gt;$BUCKET&lt;/span&gt; | jq &lt;span class="nt"&gt;-r&lt;/span&gt; &lt;span class="s2"&gt;".[]"&lt;/span&gt; &lt;span class="si"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;[&lt;/span&gt; &lt;span class="nt"&gt;-z&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$ORPHANS&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="o"&gt;]&lt;/span&gt;
  &lt;span class="k"&gt;then
    &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"No orphan objects found in &lt;/span&gt;&lt;span class="nv"&gt;$BUCKET&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
    &lt;span class="nb"&gt;exit &lt;/span&gt;0
&lt;span class="k"&gt;fi

&lt;/span&gt;&lt;span class="nv"&gt;UPLOAD_ID&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;""&lt;/span&gt;
&lt;span class="k"&gt;for &lt;/span&gt;object &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nv"&gt;$ORPHANS&lt;/span&gt;
&lt;span class="k"&gt;do
  &lt;/span&gt;&lt;span class="nv"&gt;NEW_ID&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="nv"&gt;$object&lt;/span&gt; | &lt;span class="nb"&gt;awk&lt;/span&gt; &lt;span class="nt"&gt;-F&lt;/span&gt; &lt;span class="s1"&gt;'.'&lt;/span&gt; &lt;span class="s1"&gt;'{print $(NF-1)}'&lt;/span&gt;&lt;span class="si"&gt;)&lt;/span&gt;
  &lt;span class="nv"&gt;UPLOAD_ID&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$UPLOAD_ID&lt;/span&gt;&lt;span class="s2"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$NEW_ID&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
&lt;span class="k"&gt;done&lt;/span&gt;

&lt;span class="c"&gt;# You must Remove this objects from data pool&lt;/span&gt;
&lt;span class="nv"&gt;ORPHAN_OBJECTS_DATA&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;radosgw-admin bucket radoslist &lt;span class="nt"&gt;-b&lt;/span&gt; &lt;span class="nv"&gt;$BUCKET&lt;/span&gt; | egrep &lt;span class="nt"&gt;--text&lt;/span&gt;  &lt;span class="s2"&gt;"(&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="nv"&gt;$UPLOAD_ID&lt;/span&gt; | &lt;span class="nb"&gt;tr&lt;/span&gt; &lt;span class="s1"&gt;' '&lt;/span&gt; &lt;span class="s1"&gt;'|'&lt;/span&gt; &lt;span class="si"&gt;)&lt;/span&gt;&lt;span class="s2"&gt;)"&lt;/span&gt;&lt;span class="si"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;[&lt;/span&gt; &lt;span class="nt"&gt;-n&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$ORPHAN_OBJECTS_DATA&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="o"&gt;]&lt;/span&gt;
  &lt;span class="k"&gt;then
    while &lt;/span&gt;&lt;span class="nb"&gt;read&lt;/span&gt; &lt;span class="nt"&gt;-r&lt;/span&gt; object
    &lt;span class="k"&gt;do
      &lt;/span&gt;&lt;span class="nb"&gt;echo &lt;/span&gt;rados &lt;span class="nt"&gt;-p&lt;/span&gt; &lt;span class="nv"&gt;$DATA_POOL&lt;/span&gt; &lt;span class="nb"&gt;rm&lt;/span&gt; &lt;span class="nv"&gt;$object&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nb"&gt;true
    &lt;/span&gt;&lt;span class="k"&gt;done&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&amp;lt;&amp;lt;&lt;/span&gt;  &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;ORPHAN_OBJECTS_DATA&lt;/span&gt;&lt;span class="p"&gt;// /\\ &lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
  &lt;span class="k"&gt;fi&lt;/span&gt;

&lt;span class="c"&gt;# You must Remove this omapkeys from index pool&lt;/span&gt;
&lt;span class="nv"&gt;SHARDS&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;radosgw-admin bucket stats &lt;span class="nt"&gt;-b&lt;/span&gt; &lt;span class="nv"&gt;$BUCKET&lt;/span&gt; | jq &lt;span class="nt"&gt;-r&lt;/span&gt; &lt;span class="s2"&gt;".num_shards"&lt;/span&gt;&lt;span class="si"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="o"&gt;((&lt;/span&gt; &lt;span class="nv"&gt;COUNTER&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;0&lt;span class="p"&gt;;&lt;/span&gt; COUNTER&amp;lt;&lt;span class="nv"&gt;$SHARDS&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; COUNTER+&lt;span class="o"&gt;=&lt;/span&gt;1 &lt;span class="o"&gt;))&lt;/span&gt;
&lt;span class="k"&gt;do
  &lt;/span&gt;&lt;span class="nv"&gt;ORPHAN_OMAP&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;rados &lt;span class="nt"&gt;-p&lt;/span&gt; &lt;span class="nv"&gt;$INDEX_POOL&lt;/span&gt; listomapkeys .dir.&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;BUCKET_ID&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;.&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;COUNTER&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt; | egrep &lt;span class="nt"&gt;--text&lt;/span&gt; &lt;span class="s2"&gt;"(&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="nv"&gt;$UPLOAD_ID&lt;/span&gt; | &lt;span class="nb"&gt;tr&lt;/span&gt; &lt;span class="s1"&gt;' '&lt;/span&gt; &lt;span class="s1"&gt;'|'&lt;/span&gt;&lt;span class="si"&gt;)&lt;/span&gt;&lt;span class="s2"&gt;)"&lt;/span&gt;&lt;span class="si"&gt;)&lt;/span&gt;
  &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;[&lt;/span&gt; &lt;span class="nt"&gt;-n&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$ORPHAN_OMAP&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="o"&gt;]&lt;/span&gt;
    &lt;span class="k"&gt;then
      while &lt;/span&gt;&lt;span class="nb"&gt;read&lt;/span&gt; &lt;span class="nt"&gt;-r&lt;/span&gt; omap
      &lt;span class="k"&gt;do
        &lt;/span&gt;&lt;span class="nb"&gt;echo &lt;/span&gt;rados &lt;span class="nt"&gt;-p&lt;/span&gt; &lt;span class="nv"&gt;$INDEX_POOL&lt;/span&gt; rmomapkey .dir.&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;BUCKET_ID&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;.&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;COUNTER&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt; &lt;span class="nv"&gt;$omap&lt;/span&gt;
      &lt;span class="k"&gt;done&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&amp;lt;&amp;lt;&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;ORPHAN_OMAP&lt;/span&gt;&lt;span class="p"&gt;// /\\ &lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
    &lt;span class="k"&gt;fi
done&lt;/span&gt;

&lt;span class="c"&gt;# Fix bucket&lt;/span&gt;
&lt;span class="nb"&gt;echo &lt;/span&gt;radosgw-admin bucket check &lt;span class="nt"&gt;-b&lt;/span&gt; &lt;span class="nv"&gt;$BUCKET&lt;/span&gt; &lt;span class="nt"&gt;--fix&lt;/span&gt; 
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This script uses radosgw-admin to check for orphan objects in a specified bucket, then removes them from both the data pool and the index pool. It's not a perfect solution, but it worked for us - and it might just work for you too!&lt;/p&gt;

&lt;p&gt;&lt;a href="https://res.cloudinary.com/practicaldev/image/fetch/s--qvzfmgPO--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kfnjwuokpmlu48t6uga9.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://res.cloudinary.com/practicaldev/image/fetch/s--qvzfmgPO--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kfnjwuokpmlu48t6uga9.jpg" alt="Oh shit! something right" width="564" height="564"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  But why do orphan objects even happen in the first place?
&lt;/h2&gt;

&lt;p&gt;Well, it's a complicated issue, and one that's beyond the scope of this post. But suffice it to say that they can be caused by a number of factors, including bugs in the system, network issues, or even user error.&lt;/p&gt;

&lt;h2&gt;
  
  
  So, what can you do if you find yourself in a similar situation?
&lt;/h2&gt;

&lt;p&gt;My advice would be to start by using a script like the one I've provided here to identify and remove any orphan objects in your cluster. But beyond that, there are a few best practices you can follow to help prevent orphan objects from occurring in the first place:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Keep your software up-to-date: Orphan objects can sometimes be caused by bugs in older versions of your storage software. Make sure you're always running the latest stable release.&lt;/li&gt;
&lt;li&gt;Monitor your cluster closely: Regularly monitoring your cluster's performance can help you catch orphan objects early, before they become a big problem.&lt;/li&gt;
&lt;li&gt;Educate your users: Make sure your users understand how to use your storage system correctly to minimize the risk of orphan objects caused by user error.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;In conclusion, the Great Orphan Object Hunt was a challenging but ultimately rewarding experience. By using a bit of creativity and some handy tools, we were able to fix our cloud storage woes and keep our users happy. And with a bit of luck, we'll be able to avoid orphan objects altogether in the future!&lt;/p&gt;

&lt;h2&gt;
  
  
  UPDATE:
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://tracker.ceph.com/issues/16767"&gt;https://tracker.ceph.com/issues/16767&lt;/a&gt;&lt;/p&gt;

</description>
      <category>cloudcomputing</category>
      <category>ceph</category>
      <category>sre</category>
      <category>objectstorage</category>
    </item>
  </channel>
</rss>
