<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: KYUNGHWAN KIM</title>
    <description>The latest articles on DEV Community by KYUNGHWAN KIM (@kkhdevs).</description>
    <link>https://dev.to/kkhdevs</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F3994880%2F6d96ccce-2886-4a28-b460-1c0c500741bc.png</url>
      <title>DEV Community: KYUNGHWAN KIM</title>
      <link>https://dev.to/kkhdevs</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/kkhdevs"/>
    <language>en</language>
    <item>
      <title>EKS 1.32 with Karpenter 0.37 and AL2: how to upgrade when "karpenter version is not compatible with K8s version 1.32"?</title>
      <dc:creator>KYUNGHWAN KIM</dc:creator>
      <pubDate>Mon, 16 Nov 2026 08:58:33 +0000</pubDate>
      <link>https://dev.to/kkhdevs/eks-132-with-karpenter-037-and-al2-how-to-upgrade-when-karpenter-version-is-not-compatible-with-1jh2</link>
      <guid>https://dev.to/kkhdevs/eks-132-with-karpenter-037-and-al2-how-to-upgrade-when-karpenter-version-is-not-compatible-with-1jh2</guid>
      <description>&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Flu0juqipqvebbt9bcgnl.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Flu0juqipqvebbt9bcgnl.png" alt="The log line &amp;quot;karpenter version is not compatible with K8s version 1.32&amp;quot; above a Kubernetes minor-version axis: Karpenter 0.37's supported range ends at 1.30, the cluster is at 1.32, two minors past it, and the three steps are listed in order: Karpenter, node image, control plane, 2 h 58 m in total." width="800" height="420"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Karpenter 0.37 supports Kubernetes 1.23–1.30; the cluster ran 1.32. The order of the work on 2026-11-16: Karpenter, then AL2023, then the control plane.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;If Karpenter on your EKS cluster logs &lt;code&gt;"failed validating kubernetes version"&lt;/code&gt; with &lt;code&gt;"error":"karpenter version is not compatible with K8s version 1.32"&lt;/code&gt;, the controller is a release whose supported Kubernetes range ends below the cluster's version: 0.37.x supports 1.23–1.30, logs the mismatch once per controller start, and goes on creating nodes. Nothing is down because of that line. EKS also reports the Amazon Linux 2 (AL2) nodes as an ERROR for 1.33, so we moved Karpenter, its v1beta1 manifests and the AL2 nodes before the control plane. This article gives the order we used to upgrade EKS 1.32 to 1.36 in place, with Karpenter 0.37.0 → 1.14.3 and AL2 → AL2023. The cluster had been on extended support since 2026-03-23. The article lists the eight problems met on the way, two of them caused by our own actions, and eight read-only checks with the values that mean "same problem". The cluster is a company EKS cluster of about 11 nodes (9 created by Karpenter, 2 in a managed node group), declared in one Terraform repository and applied by a GitHub Actions workflow; all times are UTC, on 2026-11-16 unless a date is given.&lt;/p&gt;

&lt;h2&gt;
  
  
  What we saw
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Before the work
&lt;/h3&gt;

&lt;p&gt;An audit of the cluster on 2026-11-14 at 16:51 listed, as problem 5 of its top five, "Kubernetes 1.32 past its support, and Karpenter left behind" (title translated). It listed:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;control plane &lt;code&gt;v1.32.13-eks-577a51e&lt;/code&gt;, nine nodes on &lt;code&gt;v1.32.9&lt;/code&gt;;&lt;/li&gt;
&lt;li&gt;upstream patches for 1.32 ended 2026-02-28, EKS standard support ended 2026-03-23, 237 days before the report's local date (2026-11-15); support type &lt;code&gt;EXTENDED&lt;/code&gt;;&lt;/li&gt;
&lt;li&gt;Karpenter &lt;code&gt;0.37.0&lt;/code&gt;, whose supported range is Kubernetes 1.23–1.30;&lt;/li&gt;
&lt;li&gt;one "version not supported" error line in the Karpenter log at 2026-11-14 08:53, while Karpenter had still created 7 nodes in the previous 24 hours;&lt;/li&gt;
&lt;li&gt;EC2NodeClass &lt;code&gt;default&lt;/code&gt; with &lt;code&gt;amiFamily: AL2&lt;/code&gt; and no pinned AMI version;&lt;/li&gt;
&lt;li&gt;four NodePools, each with the single disruption budget &lt;code&gt;nodes: "10%"&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The same run counted one more finding under that problem, &lt;code&gt;eks.nodegroup-ami-old&lt;/code&gt;: the managed node group AMI was 341 days old.&lt;/p&gt;

&lt;p&gt;The audit's recommended order: Karpenter first, then the EC2NodeClass to AL2023 with a pinned AMI version, and only then Kubernetes 1.33.&lt;/p&gt;

&lt;p&gt;At 03:15 on 2026-11-16 our own reading agreed with the audit and added two facts. There were 11 nodes: 9 created by Karpenter on &lt;code&gt;Amazon Linux 2&lt;/code&gt; (kernel 5.10, containerd 1.7.29) and 2 in a managed node group already on &lt;code&gt;Amazon Linux 2023.9.20251208&lt;/code&gt;. &lt;code&gt;aws eks list-insights&lt;/code&gt; returned 7 insights, 6 PASSING and 1 ERROR, named "Amazon Linux 2 compatibility", for 1.33:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Amazon EKS stopped publishing EKS-optimized Amazon Linux 2 (AL2) AMIs on November 26, 2025. AL2023 and Bottlerocket based AMIs for Amazon EKS are available for all supported Kubernetes versions including 1.33 and higher.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The Karpenter Helm release was chart &lt;code&gt;karpenter-0.37.0&lt;/code&gt;, 2 replicas, &lt;code&gt;webhook.enabled: false&lt;/code&gt;, with no &lt;code&gt;karpenter-crd&lt;/code&gt; release. The three Karpenter CRDs (nodepools, nodeclaims, ec2nodeclasses) served and stored only &lt;code&gt;v1beta1&lt;/code&gt;, with &lt;code&gt;conversion: None&lt;/code&gt; and no Helm labels.&lt;/p&gt;

&lt;p&gt;A correction to the audit's "one error line at 2026-11-14 08:53": at 03:17 on 11-16 one of the two Karpenter 0.37.0 pods had 0 ERROR lines in its last 72 hours, and we did not read the other pod. We saw the message ourselves at 03:38:35, right after 0.37.8 started:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;"failed validating kubernetes version" … "error":"karpenter version is not compatible with K8s version 1.32"
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It is logged when the controller discovers the cluster version, which is once per start. Karpenter 0.37.8 kept working after it and the EC2NodeClass stayed &lt;code&gt;Ready=True&lt;/code&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  During the work
&lt;/h3&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ft1fe5ogfvtzue81rjtqa.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ft1fe5ogfvtzue81rjtqa.png" alt="Measured timeline from 03:30 to 06:40 UTC with one lane each for the Karpenter release, the node image, the control plane, the managed node group and the add-ons, and a lane of incident markers: the node drift at 03:54, the 50 minutes in which two AL2 nodes were not replaced, the pull QPS and 503 minutes around 05:13–05:16, the Redis image NotFound, the rejected 1.34 update at 05:20 and the Prometheus OOMKilled at 06:24." width="800" height="587"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;What ran when on 2026-11-16: Karpenter and AL2023 applies, the four control-plane and node group updates, the add-on updates, and the incidents between them.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;In the order they appeared:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;1. A Terraform plan that would have destroyed every NodePool (03:51).&lt;/strong&gt; Changing the manifests from &lt;code&gt;karpenter.sh/v1beta1&lt;/code&gt; to &lt;code&gt;karpenter.sh/v1&lt;/code&gt; made the first plan of PR 3 mark all five &lt;code&gt;kubectl_manifest&lt;/code&gt; resources (the EC2NodeClass and the four NodePools) "must be replaced":&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;api_version = "karpenter.sh/v1beta1" -&amp;gt; "karpenter.sh/v1" # forces replacement
Plan: 5 to add, 0 to change, 5 to destroy
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;We did not apply it.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. All eight existing NodeClaims marked &lt;code&gt;Drifted&lt;/code&gt;, by our own apply (03:54:53).&lt;/strong&gt; One second after the apply of PR 3 ended, Karpenter logged &lt;code&gt;disrupting nodeclaim(s) via replace&lt;/code&gt;. At 03:55:18 all eight NodeClaims that existed before the apply had &lt;code&gt;Drifted=True (NodeClassDrift)&lt;/code&gt; and three replacements were launching, one per NodePool that had nodes. The description of PR 3 had said the nodes would not change.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Two nodes left on AL2 for 50 minutes (04:02–04:52).&lt;/strong&gt; PR 5 switched the EC2NodeClass to AL2023 at 04:01, and the first AL2023 Karpenter node was up at 04:02. At 04:52:43 the state was 7 Karpenter nodes on AL2023, 2 still on AL2, and nothing in progress. The events:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;DisruptionBlocked … Pod has "karpenter.sh/do-not-disrupt" annotation (Pod=prod-a/api-a-…)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;and the same for &lt;code&gt;prod-b/admin-b-…&lt;/code&gt;. Three pods in the cluster carried the annotation.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. &lt;code&gt;pull QPS exceeded&lt;/code&gt;: 27 pods not Running and target groups with no healthy target (05:13–05:16).&lt;/strong&gt; At 05:13:09, 14 minutes after the 1.33 control-plane update started, 9 nodes were already on &lt;code&gt;v1.33.13&lt;/code&gt; and 27 pods were not Running. At 05:14:15, 25 of the 26 pods not Running were on one node that was 3 m 7 s old (an m8g.large in NodePool &lt;code&gt;default&lt;/code&gt;), among them coredns, the load balancer controller, the EBS CSI controller, metrics-server, external-secrets and application pods. The EBS CSI node pod's event:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Failed to pull image "….amazonaws.com/eks/aws-ebs-csi-driver:v1.37.0": pull QPS exceeded
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;At 05:15:29, 11 load balancer target groups had targets but no healthy one: in 8 the only target was &lt;code&gt;draining&lt;/code&gt;, in 3 the targets were &lt;code&gt;unhealthy&lt;/code&gt;. The cluster's owner reported &lt;code&gt;503 Service Temporarily Unavailable&lt;/code&gt; from the services in the same minutes. The limit of this observation: the 503s were not measured from outside. We have the target group states at one instant and one user report, and no duration per service. By 05:16:54 all pods were Running except Redis and pods being terminated.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;5. &lt;code&gt;public.ecr.aws/bitnami/redis&lt;/code&gt; in &lt;code&gt;ImagePullBackOff&lt;/code&gt; with &lt;code&gt;NotFound&lt;/code&gt; (05:13).&lt;/strong&gt; &lt;code&gt;redis-master-0&lt;/code&gt; was in &lt;code&gt;ImagePullBackOff&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Failed to pull image "public.ecr.aws/bitnami/redis:7.2.3-debian-11-r1": rpc error: code = NotFound
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;and the same for &lt;code&gt;bitnami/redis-exporter:1.55.0-debian-11-r2&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;6. EKS refused the 1.34 update (05:20:22).&lt;/strong&gt; The apply of PR 8 failed with:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Error: updating EKS Cluster (…) version: operation error EKS: UpdateClusterVersion, https response error StatusCode: 400, … InvalidRequestException: Atleast one subnet in each AZ should have 2 free IPs. Invalid AZs: { [zone-b] }, provided subnets: { … }
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;7. Both replicas of one API deleted at once, by our own script (05:42).&lt;/strong&gt; Before each later control-plane step we cordoned the nodes whose replacement was blocked by an annotated pod and deleted that pod. At 05:42 the script deleted both replicas of &lt;code&gt;api-a&lt;/code&gt; at once, because they ran on two different drifted nodes. Both replacements were Pending, then starting, for a few minutes.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;8. Prometheus OOMKilled (06:24).&lt;/strong&gt; &lt;code&gt;prometheus-…-0&lt;/code&gt; was in &lt;code&gt;CrashLoopBackOff&lt;/code&gt;: container &lt;code&gt;prometheus&lt;/code&gt;, 5 restarts, last state &lt;code&gt;OOMKilled/137&lt;/code&gt;, limit 1536Mi.&lt;/p&gt;

&lt;p&gt;One more delay had nothing to do with the cluster: the CI checkout step of PR 2 failed three times with &lt;code&gt;fatal: repository '…' not found&lt;/code&gt; between 03:41 and 03:44 while GitHub's status page said "All Systems Operational". The fourth run passed with no change on our side.&lt;/p&gt;

&lt;p&gt;At 06:34:48 the server was &lt;code&gt;v1.36.5-eks-91f89a9&lt;/code&gt;, all 10 nodes were on &lt;code&gt;v1.36.5-eks-9adbfea&lt;/code&gt; and &lt;code&gt;Amazon Linux 2023.12.20261112&lt;/code&gt;, 0 pods were outside Running/Completed, and 9 of 9 Argo CD applications were Synced and Healthy. From the first merge (03:36:38) to that check: 2 h 58 m. From the first control-plane update call to the last: 63 m 57 s.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why
&lt;/h2&gt;

&lt;h3&gt;
  
  
  &lt;code&gt;karpenter version is not compatible with K8s version&lt;/code&gt;: a range check that logs in 0.37 and 1.0, and returns the error in 1.1
&lt;/h3&gt;

&lt;p&gt;The supported range is two constants in &lt;code&gt;pkg/providers/version/version.go&lt;/code&gt; of aws/karpenter-provider-aws. By tag:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Karpenter tag&lt;/th&gt;
&lt;th&gt;&lt;code&gt;MinK8sVersion&lt;/code&gt;&lt;/th&gt;
&lt;th&gt;&lt;code&gt;MaxK8sVersion&lt;/code&gt;&lt;/th&gt;
&lt;th&gt;On a version outside the range&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;v0.37.8&lt;/td&gt;
&lt;td&gt;1.23&lt;/td&gt;
&lt;td&gt;1.30&lt;/td&gt;
&lt;td&gt;logs and continues (observed on 1.32)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;v1.0.14&lt;/td&gt;
&lt;td&gt;1.25&lt;/td&gt;
&lt;td&gt;1.31&lt;/td&gt;
&lt;td&gt;logs and continues (source and observed on 1.32)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;v1.1.10&lt;/td&gt;
&lt;td&gt;1.25&lt;/td&gt;
&lt;td&gt;1.31&lt;/td&gt;
&lt;td&gt;returns the error (source only; not run)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;v1.14.3&lt;/td&gt;
&lt;td&gt;1.26&lt;/td&gt;
&lt;td&gt;1.36&lt;/td&gt;
&lt;td&gt;in range for 1.32–1.36&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;In v1.0.14 an out-of-range version is only logged, and the function goes on:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="n"&gt;log&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;FromContext&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Error&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;err&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"failed validating kubernetes version"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="c1"&gt;// ...&lt;/span&gt;
&lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;version&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kc"&gt;nil&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;In v1.1.10 the same check is returned to the caller:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;fmt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Errorf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"validating kubernetes version, %w"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;We observed the logging behaviour. Karpenter 1.0.14 also logged the message (03:48:56) and then launched three nodes at 03:54:58–03:55:08, one of which registered within 41 s.&lt;/p&gt;

&lt;p&gt;The returned error in 1.1 we only read. We did not run 1.1.x or any release between 1.0.14 and 1.14.3, and did not read what the caller does with the returned error (a failed reconcile or a crash). That 1.1.x would stop launching nodes on 1.32 is our reading of the returned error; because of it we went from 1.0.14 straight to a release whose range contains the current and the target Kubernetes version.&lt;/p&gt;

&lt;p&gt;In v1.14.3 the message is reworded: &lt;code&gt;karpenter is not compatible with kubernetes version&lt;/code&gt; (&lt;code&gt;version.go&lt;/code&gt; line 125). The wording &lt;code&gt;karpenter version is not compatible with K8s version&lt;/code&gt; is the one in v0.37.x through v1.2.0. The log command in step 1 below matches both.&lt;/p&gt;

&lt;p&gt;The Karpenter compatibility matrix (karpenter.sh/docs/upgrading/compatibility, read at 03:18) gives the minimum Karpenter per Kubernetes minor:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Kubernetes&lt;/th&gt;
&lt;th&gt;1.31&lt;/th&gt;
&lt;th&gt;1.32&lt;/th&gt;
&lt;th&gt;1.33&lt;/th&gt;
&lt;th&gt;1.34&lt;/th&gt;
&lt;th&gt;1.35&lt;/th&gt;
&lt;th&gt;1.36&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Karpenter at least&lt;/td&gt;
&lt;td&gt;1.0.5&lt;/td&gt;
&lt;td&gt;1.2&lt;/td&gt;
&lt;td&gt;1.5&lt;/td&gt;
&lt;td&gt;1.6&lt;/td&gt;
&lt;td&gt;1.9&lt;/td&gt;
&lt;td&gt;1.13&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The same page lists v1.14 as an LTS release (released July 2026, end of life July 2027). The matrix gives only a minimum (1.13) for Kubernetes 1.36; that 1.14.3 itself allows 1.36 we take from the source constant (&lt;code&gt;MaxK8sVersion = "1.36"&lt;/code&gt;).&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fezxkcxwnjj0ipplyaeb8.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fezxkcxwnjj0ipplyaeb8.png" alt="Range bars on a Kubernetes minor-version axis, one row per Karpenter release: v0.37.8 covers 1.23–1.30 and is marked &amp;quot;logs and continues&amp;quot;, v1.0.14 covers 1.25–1.31 and is marked &amp;quot;logs and continues&amp;quot;, v1.1.10 covers 1.25–1.31 and is marked &amp;quot;returns an error&amp;quot;, v1.14.3 covers 1.26–1.36 and is marked &amp;quot;in range&amp;quot;; the span 1.32 to 1.36 is shaded." width="800" height="420"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Supported Kubernetes range of four Karpenter releases, from &lt;code&gt;version.go&lt;/code&gt; at each tag. Only v1.14.3 covers the whole 1.32 → 1.36 span.&lt;/em&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  Why 0.37 cannot go to 1.14 in one step
&lt;/h3&gt;

&lt;p&gt;The upgrade guide (karpenter.sh/docs/upgrading/upgrade-guide, read at 03:55) says "Do not upgrade to 1.0.0+ without following the v1 Migration Guide" and "Karpenter 1.1.0 drops the support for v1beta1 APIs". The v1 migration guide (at tag v1.0.14) gives the path: first the latest patch of the current minor with &lt;code&gt;webhook.enabled=true&lt;/code&gt; on both the &lt;code&gt;karpenter-crd&lt;/code&gt; chart and the &lt;code&gt;karpenter&lt;/code&gt; chart, then 1.0.x; before 1.1.0 every stored object must be v1. Its rollback targets are 0.37.1, 0.36.3, 0.35.6, 0.34.7 and 0.33.6.&lt;/p&gt;

&lt;p&gt;The same upgrade guide lists what later releases need: in 1.1.0 "nodeClassRef.group and nodeClassRef.kind are strictly required"; 1.7.0 needs &lt;code&gt;iam:ListInstanceProfiles&lt;/code&gt;; 1.11.0 adds &lt;code&gt;ec2:DescribePlacementGroups&lt;/code&gt;; 1.12.0 adds &lt;code&gt;ec2:DescribeInstanceStatus&lt;/code&gt;; 1.8.4 has a scheduling regression ("do not upgrade to this version"); 1.14.0 ships a new &lt;code&gt;capacitybuffers&lt;/code&gt; CRD.&lt;/p&gt;

&lt;p&gt;Our controller IAM policy is written by hand in Terraform, not taken from the upstream CloudFormation. Compared with the upstream policy by action name, nothing was missing for v1.0.14 and v1.1.10; for v1.14.3 four actions were missing: &lt;code&gt;ec2:DescribeCapacityReservations&lt;/code&gt;, &lt;code&gt;ec2:DescribeInstanceStatus&lt;/code&gt;, &lt;code&gt;ec2:DescribePlacementGroups&lt;/code&gt;, &lt;code&gt;iam:ListInstanceProfiles&lt;/code&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  &lt;code&gt;kubectl_manifest&lt;/code&gt; must be replaced: why &lt;code&gt;api_version … # forces replacement&lt;/code&gt; planned &lt;code&gt;5 to destroy&lt;/code&gt;
&lt;/h3&gt;

&lt;p&gt;The EC2NodeClass and the four NodePools are &lt;code&gt;kubectl_manifest&lt;/code&gt; resources (provider gavinbunney/kubectl) holding YAML. The provider plans a change of &lt;code&gt;api_version&lt;/code&gt; as a replacement; we know that from the plan line only and did not read the provider's schema.&lt;/p&gt;

&lt;p&gt;Deleting a NodePool deletes its NodeClaims and therefore its nodes, and all nine Karpenter nodes belonged to those four NodePools. This part is an inference: we did not apply that plan, so the mass deletion is what the plan and Karpenter's ownership model imply, not something we watched.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why the nodes were &lt;code&gt;Drifted&lt;/code&gt; (&lt;code&gt;NodeClassDrift&lt;/code&gt;) after the EC2NodeClass change
&lt;/h3&gt;

&lt;p&gt;When the conversion webhook converted the stored objects to v1 (after PR 2), it rendered the EC2NodeClass with &lt;code&gt;amiSelectorTerms: [{alias: al2@latest}]&lt;/code&gt; and &lt;code&gt;metadataOptions.httpPutResponseHopLimit: 2&lt;/code&gt;. In PR 3 we copied those fields explicitly into the EC2NodeClass YAML.&lt;/p&gt;

&lt;p&gt;Which field changed the hash was not isolated. The EC2NodeClass was the only object whose spec we rewrote by hand in that apply; that this rewrite caused the drift is an inference.&lt;/p&gt;

&lt;h3&gt;
  
  
  "Amazon Linux 2 compatibility" ERROR with &lt;code&gt;amiFamily: AL2&lt;/code&gt;, and why two nodes stayed on AL2
&lt;/h3&gt;

&lt;p&gt;The insight text above is the reason for the first: the nine Karpenter nodes ran AL2. SSM has an AL2 EKS AMI release for Kubernetes 1.32 (&lt;code&gt;1.32.9-20251209&lt;/code&gt;) and none for 1.33 (&lt;code&gt;ParameterNotFound&lt;/code&gt; for &lt;code&gt;/aws/service/eks/optimized-ami/1.33/amazon-linux-2/recommended/release_version&lt;/code&gt;). EKS never refused a 1.33 update here: by the time we called it the nodes were on AL2023 and the insight had cleared. Whether a 1.33 update with AL2 nodes would have been refused, or would have left Karpenter with no image to launch, we did not observe.&lt;/p&gt;

&lt;p&gt;For the second, the &lt;code&gt;DisruptionBlocked&lt;/code&gt; event message contains the reason. The Karpenter disruption docs (karpenter.sh/docs/concepts/disruption, read at 08:24) say that nodes with active &lt;code&gt;karpenter.sh/do-not-disrupt&lt;/code&gt; pods "will be excluded from Consolidation, and conditionally excluded from Drift. If the Node's owning NodeClaim has a terminationGracePeriod configured, it will still be eligible for disruption via drift", and that pods with the annotation "may block draining indefinitely". None of our four NodePools set &lt;code&gt;terminationGracePeriod&lt;/code&gt;, before the work or after.&lt;/p&gt;

&lt;h3&gt;
  
  
  &lt;code&gt;pull QPS exceeded&lt;/code&gt;: why the pods and the load balancers failed in the same minutes
&lt;/h3&gt;

&lt;p&gt;Four things ran inside one &lt;code&gt;terraform apply&lt;/code&gt; and one Karpenter reconcile: the control-plane update (started 04:58:50), Karpenter's drift replacement of its own nodes (starting within minutes), the managed node group update (05:06:08) and the add-on updates (05:14:38). Most Deployments in this cluster have one replica. That these four overlapping operations are why 25 pods were scheduled onto one new node and why 8 target groups had only a &lt;code&gt;draining&lt;/code&gt; target is an inference from those times; no single log line states it.&lt;/p&gt;

&lt;p&gt;After each control-plane step Karpenter replaced its nodes without any change to the EC2NodeClass. The alias &lt;code&gt;al2023@v20261112&lt;/code&gt; pins the AMI release date, and the same release exists per Kubernetes version (&lt;code&gt;1.32.13-20261112&lt;/code&gt; for 1.32, &lt;code&gt;1.36.5-20261112&lt;/code&gt; for 1.36), so the alias resolves to the AMI of the cluster's Kubernetes version.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;pull QPS exceeded&lt;/code&gt; is the kubelet's own limit on image pulls, not a refusal by the registry. The kubelet configuration reference (kubernetes.io) gives &lt;code&gt;registryPullQPS&lt;/code&gt; as "the limit of registry pulls per second", default 5, and &lt;code&gt;registryBurst&lt;/code&gt;, default 10. We did not read the kubelet configuration of the node, so that it ran with those defaults is an inference.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why &lt;code&gt;public.ecr.aws/bitnami/redis&lt;/code&gt; returned NotFound only now (&lt;code&gt;bitnamilegacy&lt;/code&gt;)
&lt;/h3&gt;

&lt;p&gt;The same two tags exist at &lt;code&gt;docker.io/bitnamilegacy/&lt;/code&gt; (the Docker Hub API answered 200). Kafka in the same repository was already pulling from &lt;code&gt;bitnamilegacy&lt;/code&gt;, with a comment that the public.ecr.aws images had been deleted. That Redis had been running from an image cached on an old node is an inference: the pod was Running before the node replacement with the same image reference, and that reference now returns NotFound.&lt;/p&gt;

&lt;h3&gt;
  
  
  &lt;code&gt;Atleast one subnet in each AZ should have 2 free IPs&lt;/code&gt;: a /24 at 0 free
&lt;/h3&gt;

&lt;p&gt;The cluster uses two private /24 subnets, one per zone. At 05:21 the subnet in zone-b had &lt;code&gt;AvailableIpAddressCount&lt;/code&gt; 0 and the one in zone-a 150. In zone-b, 21 in-use network interfaces held 251 private IPs; in zone-a, 17 held 101. Eight of the 15 nodes at that moment were in zone-b.&lt;/p&gt;

&lt;p&gt;The VPC CNI ran with &lt;code&gt;WARM_ENI_TARGET=1&lt;/code&gt;, &lt;code&gt;WARM_PREFIX_TARGET=1&lt;/code&gt;, &lt;code&gt;ENABLE_PREFIX_DELEGATION=false&lt;/code&gt;, no &lt;code&gt;WARM_IP_TARGET&lt;/code&gt; and no &lt;code&gt;MINIMUM_IP_TARGET&lt;/code&gt;. We did not measure the spare interfaces or addresses per node. The node count was 11 at 03:15 and 15 at 05:21.&lt;/p&gt;

&lt;p&gt;The EKS user guide ("Update existing cluster to new Kubernetes version") says: "To update the cluster, Amazon EKS requires up to five available IP addresses from the subnets that you specified when you created your cluster." The API error we received says 2 in each AZ. We quote both and have not reconciled them.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fjri8ksiidk85en7hdb9m.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fjri8ksiidk85en7hdb9m.png" alt="Two horizontal bars for the same /24 subnet in zone-b. Before: 21 network interfaces hold 251 addresses and 0 are free, with WARM_ENI_TARGET=1. After WARM_IP_TARGET=3 and MINIMUM_IP_TARGET=6: 127 free right after the change and 181 free at 05:42. A marker shows the 2 free addresses per zone that the EKS error asks for." width="800" height="400"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The zone-b subnet at 05:21 and after the VPC CNI setting change: addresses held by network interfaces against free addresses.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What to check in your cluster now
&lt;/h2&gt;

&lt;p&gt;All steps are read-only. &lt;code&gt;&amp;lt;cluster&amp;gt;&lt;/code&gt; is your EKS cluster name. Each step ends with the value that means "same problem as ours".&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;1. Is Karpenter inside its supported Kubernetes range, and which release is next?&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl version &lt;span class="o"&gt;|&lt;/span&gt; grep Server
kubectl -n karpenter get deploy karpenter -o &lt;span class="nv"&gt;jsonpath&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;'{.spec.template.spec.containers[0].image}{"\n"}'&lt;/span&gt;
kubectl -n karpenter logs -l app.kubernetes.io/name=karpenter --all-containers --since=72h --tail=-1 &lt;span class="o"&gt;|&lt;/span&gt; grep -cE &lt;span class="s1"&gt;'is not compatible with (K8s|kubernetes) version'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Same problem as ours: the server is 1.32 and the image is &lt;code&gt;0.37.x&lt;/code&gt; (range 1.23–1.30). A count of 1 or more from the third command means the controller has logged that it is outside its range. The log line is written once per controller start, so a count of 0 over 72 hours does not mean the version is in range. The grep matches both wordings of the message (see "Why"), across all Karpenter pods (check the label on your release first). If the image is &lt;code&gt;0.3x&lt;/code&gt;, the path is fixed: latest patch of your minor with the webhook on → 1.0.x → manifests to v1 → a release whose range contains both your current and your target Kubernetes version (matrix in the previous section). Do not stop on 1.1.x while the cluster is on 1.32 (source reading, not a run).&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Are the CRDs ready for the conversion, and does Helm own them?&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="k"&gt;for&lt;/span&gt; c &lt;span class="k"&gt;in&lt;/span&gt; nodepools.karpenter.sh nodeclaims.karpenter.sh ec2nodeclasses.karpenter.k8s.aws&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt;
  kubectl get crd &lt;span class="nv"&gt;$c&lt;/span&gt; -o &lt;span class="nv"&gt;jsonpath&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;'{.metadata.name}{" versions="}{range .spec.versions[*]}{.name}{"(storage="}{.storage}{") "}{end}{"stored="}{.status.storedVersions}{" conversion="}{.spec.conversion.strategy}{" helm="}{.metadata.labels.app\.kubernetes\.io/managed-by}{"\n"}'&lt;/span&gt;
&lt;span class="k"&gt;done&lt;/span&gt;
helm list -A &lt;span class="o"&gt;|&lt;/span&gt; grep -i karpenter
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Same as ours before the work: only &lt;code&gt;v1beta1&lt;/code&gt;, &lt;code&gt;conversion=None&lt;/code&gt;, &lt;code&gt;helm=&lt;/code&gt; empty, and no &lt;code&gt;karpenter-crd&lt;/code&gt; release. Before you move to a release that drops v1beta1 (1.1.0 and later), &lt;code&gt;stored=&lt;/code&gt; must be &lt;code&gt;["v1"]&lt;/code&gt; for all three.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Will Terraform delete the NodePools when you change their apiVersion?&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;grep -rn &lt;span class="s1"&gt;'kubectl_manifest'&lt;/span&gt; --include=&lt;span class="s1"&gt;'*.tf'&lt;/span&gt; . &lt;span class="o"&gt;|&lt;/span&gt; head
&lt;span class="c"&gt;# after changing apiVersion in a branch:&lt;/span&gt;
terraform plan -no-color &lt;span class="o"&gt;|&lt;/span&gt; grep -E &lt;span class="s1"&gt;'must be replaced|forces replacement|^Plan:'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Same problem: &lt;code&gt;api_version = "karpenter.sh/v1beta1" -&amp;gt; "karpenter.sh/v1" # forces replacement&lt;/code&gt; and a &lt;code&gt;Plan:&lt;/code&gt; line with as many &lt;code&gt;to destroy&lt;/code&gt; as you have NodePools and EC2NodeClasses. Deleting a NodePool deletes its nodes, so do not apply that plan. What a safe plan looked like for us after &lt;code&gt;terraform state rm&lt;/code&gt; and &lt;code&gt;import&lt;/code&gt; blocks: &lt;code&gt;Plan: 5 to import, 0 to add, 5 to change, 0 to destroy&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. Does EKS already report an ERROR insight for the next version?&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;aws eks list-insights --cluster-name &lt;span class="o"&gt;&amp;lt;&lt;/span&gt;cluster&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; --query &lt;span class="s1"&gt;'insights[?insightStatus.status!=`PASSING`].[name,kubernetesVersion,insightStatus.status]'&lt;/span&gt; --output text
kubectl get nodes -o custom-columns=&lt;span class="s1"&gt;'NAME:.metadata.name,KUBELET:.status.nodeInfo.kubeletVersion,OS:.status.nodeInfo.osImage'&lt;/span&gt;
kubectl get ec2nodeclass -o &lt;span class="nv"&gt;jsonpath&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;'{range .items[*]}{.metadata.name}{" amiFamily="}{.spec.amiFamily}{" terms="}{.spec.amiSelectorTerms}{"\n"}{end}'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Same problem: &lt;code&gt;Amazon Linux 2 compatibility … ERROR&lt;/code&gt;, nodes whose OS is &lt;code&gt;Amazon Linux 2&lt;/code&gt;, an EC2NodeClass with &lt;code&gt;amiFamily=AL2&lt;/code&gt;. After the node image change, &lt;code&gt;aws eks start-insights-refresh --cluster-name &amp;lt;cluster&amp;gt;&lt;/code&gt; re-evaluates without waiting for the daily run.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;5. Which pods will stop Karpenter from replacing a node?&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl get pods -A -o json &lt;span class="o"&gt;|&lt;/span&gt; jq -r &lt;span class="s1"&gt;'.items[] | select(.metadata.annotations["karpenter.sh/do-not-disrupt"]=="true") | [.metadata.namespace,.metadata.name,.spec.nodeName] | @tsv'&lt;/span&gt;
kubectl get events -A --field-selector &lt;span class="nv"&gt;reason&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;DisruptionBlocked --sort-by=.lastTimestamp &lt;span class="o"&gt;|&lt;/span&gt; tail
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Same problem: any line from the first command; an event from the second command whose message is &lt;code&gt;Pod has "karpenter.sh/do-not-disrupt" annotation&lt;/code&gt; means a replacement is blocked now. Karpenter excludes a node with such a pod from drift replacement unless the NodePool sets &lt;code&gt;terminationGracePeriod&lt;/code&gt; (Karpenter disruption docs, quoted in "Why"). Check with &lt;code&gt;kubectl get nodepools -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.template.spec.terminationGracePeriod}{"\n"}{end}'&lt;/code&gt;: an empty second column means no limit. Ours: three annotated pods and no &lt;code&gt;terminationGracePeriod&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;6. Do the subnets have room for the node churn, and for the control-plane update itself?&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;aws ec2 describe-subnets --subnet-ids &lt;span class="nv"&gt;$(&lt;/span&gt;aws eks describe-cluster --name &lt;span class="o"&gt;&amp;lt;&lt;/span&gt;cluster&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; --query &lt;span class="s1"&gt;'cluster.resourcesVpcConfig.subnetIds'&lt;/span&gt; --output text&lt;span class="o"&gt;)&lt;/span&gt; \
  --query &lt;span class="s1"&gt;'Subnets[].[AvailabilityZone,CidrBlock,AvailableIpAddressCount]'&lt;/span&gt; --output text
kubectl -n kube-system get ds aws-node -o &lt;span class="nv"&gt;jsonpath&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;'{range .spec.template.spec.containers[?(@.name=="aws-node")].env[*]}{.name}={.value}{"\n"}{end}'&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; grep -E &lt;span class="s1"&gt;'WARM_|MINIMUM_IP|PREFIX_DELEGATION'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Same problem: a subnet whose &lt;code&gt;AvailableIpAddressCount&lt;/code&gt; is below (IPs per ENI of your instance type × the nodes a roll adds), and &lt;code&gt;WARM_ENI_TARGET=1&lt;/code&gt; with no &lt;code&gt;WARM_IP_TARGET&lt;/code&gt;. The EKS error asks for 2 free addresses per zone and the AWS documentation for up to five (see "Why"). Ours: the zone-b /24 at 0 free and the zone-a /24 at 150 free at 05:21, with 11 nodes at 03:15 and 15 at 05:21.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;7. Does every image the cluster runs still exist in its registry?&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl get pods -A -o &lt;span class="nv"&gt;jsonpath&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;'{range .items[*]}{range .spec.containers[*]}{.image}{"\n"}{end}{range .spec.initContainers[*]}{.image}{"\n"}{end}{end}'&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; sort -u &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; images.txt
&lt;span class="k"&gt;while&lt;/span&gt; &lt;span class="nb"&gt;read&lt;/span&gt; i&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt; docker buildx imagetools inspect &lt;span class="s2"&gt;"$i"&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt;/dev/null 2&lt;span class="o"&gt;&amp;gt;&amp;amp;&lt;/span&gt;1 &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"MISSING $i"&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;done&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; images.txt
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Private registries need a login first; &lt;code&gt;crane manifest&lt;/code&gt; or &lt;code&gt;skopeo inspect docker://…&lt;/code&gt; do the same. Same problem: any &lt;code&gt;MISSING&lt;/code&gt; line. That pod runs only because the image is in the cache of the node it is on, and it will not start on a new node.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;8. What goes down when every node is replaced once?&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl get deploy,sts -A -o json &lt;span class="o"&gt;|&lt;/span&gt; jq -r &lt;span class="s1"&gt;'.items[] | select(.spec.replicas==1) | [.kind,.metadata.namespace,.metadata.name] | @tsv'&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; wc -l
kubectl get targetgroupbindings -A --no-headers &lt;span class="o"&gt;|&lt;/span&gt; wc -l
aws elbv2 describe-target-groups --query &lt;span class="s1"&gt;'length(TargetGroups)'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Same problem: a first number above 0 that includes a workload serving traffic (each is unavailable while its pod moves; ours: most Deployments and all 7 production databases), and a third number larger than the second plus the groups your ingresses created (a group with hand-registered pod IPs keeps the old IP when the pod moves). &lt;code&gt;describe-target-groups&lt;/code&gt; counts every group in the account and region, so subtract the groups of other clusters.&lt;/p&gt;

&lt;p&gt;The audit quoted at the top of "What we saw" was a run of &lt;a href="https://github.com/KyungHwanKim-devs/kubegap" rel="noopener noreferrer"&gt;kubegap&lt;/a&gt;, an open-source audit of Kubernetes cluster operations that I maintain. It automates the readings of steps 1 and 4: its checks &lt;code&gt;platform.k8s-version-eol&lt;/code&gt;, &lt;code&gt;platform.karpenter-version-compat&lt;/code&gt;, &lt;code&gt;karpenter.ec2nodeclass-al2&lt;/code&gt;, &lt;code&gt;logs.signatures.karpenter.version-unsupported&lt;/code&gt; and &lt;code&gt;eks.nodegroup-ami-old&lt;/code&gt; are the five findings behind the problem reported here. It does not read the CRD versions of step 2 or run the Terraform plan of step 3. Steps 5 to 8 were done by hand in this work; whether kubegap reports them was not checked for this article.&lt;/p&gt;

&lt;h2&gt;
  
  
  What we changed, and what we have not
&lt;/h2&gt;

&lt;p&gt;No non-production cluster existed and none was built; every step below ran in production first. That is a limit of this report; we do not recommend it.&lt;/p&gt;

&lt;h3&gt;
  
  
  Before the first change
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;Thirteen EBS snapshots at 03:33:16, one per PersistentVolume, and the cluster objects exported to YAML. The cluster has 13 PersistentVolumes (232 Gi in total), among them 7 production databases, each a single-replica StatefulSet, with no AWS Backup plan and no DLM policy. The snapshots were deleted at 07:04 after all 13 stateful pods were Running; they were never used for a restore.&lt;/li&gt;
&lt;li&gt;The three existing CRDs were labelled &lt;code&gt;app.kubernetes.io/managed-by=Helm&lt;/code&gt; and annotated &lt;code&gt;meta.helm.sh/release-name=karpenter-crd&lt;/code&gt; and &lt;code&gt;meta.helm.sh/release-namespace=karpenter&lt;/code&gt; at 03:34:55, so that a new &lt;code&gt;karpenter-crd&lt;/code&gt; Helm release could take them over.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Karpenter upgrade 0.37 to 1.0, then 1.14.3: conversion webhook and the karpenter-crd chart, in four pull requests
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;PR&lt;/th&gt;
&lt;th&gt;Change&lt;/th&gt;
&lt;th&gt;Plan&lt;/th&gt;
&lt;th&gt;Apply (UTC)&lt;/th&gt;
&lt;th&gt;State after&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;PR 1&lt;/td&gt;
&lt;td&gt;Karpenter 0.37.0 → 0.37.8, new &lt;code&gt;karpenter-crd&lt;/code&gt; release with the webhook settings, &lt;code&gt;webhook.enabled=true&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;&lt;code&gt;1 to add, 1 to change, 0 to destroy&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;03:36:40–03:38:42&lt;/td&gt;
&lt;td&gt;Both charts at 0.37.8; CRDs serve &lt;code&gt;v1&lt;/code&gt; and &lt;code&gt;v1beta1&lt;/code&gt;, &lt;code&gt;v1beta1&lt;/code&gt; stored, &lt;code&gt;conversion: Webhook&lt;/code&gt;; 0 Pending pods&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;PR 2&lt;/td&gt;
&lt;td&gt;0.37.8 → 1.0.14, four read actions added to the controller IAM policy&lt;/td&gt;
&lt;td&gt;&lt;code&gt;0 to add, 3 to change, 0 to destroy&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;03:47:04–03:49:01&lt;/td&gt;
&lt;td&gt;Both charts 1.0.14; &lt;code&gt;storedVersions ["v1beta1","v1"]&lt;/code&gt; at 03:49:28 and &lt;code&gt;["v1"]&lt;/code&gt; for all three CRDs at 03:55&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;PR 3&lt;/td&gt;
&lt;td&gt;The five manifests to v1, through &lt;code&gt;terraform state rm&lt;/code&gt; and &lt;code&gt;import&lt;/code&gt; blocks&lt;/td&gt;
&lt;td&gt;&lt;code&gt;5 to import, 0 to add, 5 to change, 0 to destroy&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;03:53:49–03:54:52&lt;/td&gt;
&lt;td&gt;Manifests v1; all eight NodeClaims that existed before the apply &lt;code&gt;Drifted&lt;/code&gt; (see above)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;PR 4&lt;/td&gt;
&lt;td&gt;1.0.14 → 1.14.3, webhook settings and import blocks removed&lt;/td&gt;
&lt;td&gt;&lt;code&gt;0 to add, 2 to change, 0 to destroy&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;03:57:13–03:59:07&lt;/td&gt;
&lt;td&gt;2 pods Running on 1.14.3; new CRDs &lt;code&gt;capacitybuffers.autoscaling.x-k8s.io&lt;/code&gt; and &lt;code&gt;nodeoverlays.karpenter.sh&lt;/code&gt;; EC2NodeClass and four NodePools Ready&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Details that mattered:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;After PR 1, &lt;code&gt;kubectl get nodepools.v1.karpenter.sh default&lt;/code&gt; returned &lt;code&gt;karpenter.sh/v1&lt;/code&gt; with &lt;code&gt;nodeClassRef {group, kind, name}&lt;/code&gt;. The node security group already allowed port 8443 from the control plane, which the conversion webhook needs.&lt;/li&gt;
&lt;li&gt;For PR 3 we backed up the state with &lt;code&gt;terraform state pull&lt;/code&gt; (03:51:25; the state bucket has versioning off), ran &lt;code&gt;terraform state rm&lt;/code&gt; on the five addresses ("Successfully removed 5 resource instance(s)"), and added five &lt;code&gt;import&lt;/code&gt; blocks with ids of the form &lt;code&gt;karpenter.sh/v1//NodePool//default&lt;/code&gt; and &lt;code&gt;karpenter.k8s.aws/v1//EC2NodeClass//default&lt;/code&gt;. Each resource then planned as "will be updated in-place (imported from …)". Local Terraform was 1.5.7, CI 1.5.0.&lt;/li&gt;
&lt;li&gt;At 03:56 we removed the leftover &lt;code&gt;compatibility.karpenter.sh/v1beta1-nodeclass-reference&lt;/code&gt; annotation from the four NodePools.&lt;/li&gt;
&lt;li&gt;At 06:40 Karpenter had 0 error lines in the last 30 minutes.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Amazon Linux 2 → AL2023, with a pinned alias
&lt;/h3&gt;

&lt;p&gt;PR 5 replaced &lt;code&gt;amiFamily: AL2&lt;/code&gt; and &lt;code&gt;al2@latest&lt;/code&gt; with &lt;code&gt;amiSelectorTerms: [alias: al2023@v20261112]&lt;/code&gt; (plan &lt;code&gt;0 to add, 1 to change, 0 to destroy&lt;/code&gt;, apply 04:00:56–04:01:59). The release version came from SSM, &lt;code&gt;/aws/service/eks/optimized-ami/&amp;lt;k8s&amp;gt;/amazon-linux-2023/&amp;lt;arch&amp;gt;/standard/recommended/release_version&lt;/code&gt;: &lt;code&gt;1.32.13-20261112&lt;/code&gt; for 1.32 and &lt;code&gt;1.36.5-20261112&lt;/code&gt; for 1.36, x86_64 and arm64 alike. We pinned the alias so that an AMI release by AWS does not replace the nodes. The nodes are still replaced when we change the alias value and when the control plane version changes (see "Why").&lt;/p&gt;

&lt;p&gt;For the two nodes whose replacement was blocked by annotated pods we ran &lt;code&gt;kubectl cordon&lt;/code&gt; on the nodes and &lt;code&gt;kubectl delete pod&lt;/code&gt; on the two pods at 04:53; the Deployments recreated them elsewhere. By 04:56 all 12 nodes were on AL2023. After &lt;code&gt;aws eks start-insights-refresh&lt;/code&gt;, the refresh was COMPLETED at 04:56:52 and no insight was non-PASSING.&lt;/p&gt;

&lt;h3&gt;
  
  
  Upgrade EKS 1.32 to 1.36: the control plane, one minor per apply
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Step&lt;/th&gt;
&lt;th&gt;Control-plane update start&lt;/th&gt;
&lt;th&gt;Node group update start&lt;/th&gt;
&lt;th&gt;CI apply run&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;1.32 → 1.33 (PR 6)&lt;/td&gt;
&lt;td&gt;04:58:50&lt;/td&gt;
&lt;td&gt;05:06:08&lt;/td&gt;
&lt;td&gt;04:57:58–05:17:53 (19 m 55 s)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;1.33 → 1.34 (PR 8 refused, done by PR 9)&lt;/td&gt;
&lt;td&gt;05:24:58&lt;/td&gt;
&lt;td&gt;05:31:43&lt;/td&gt;
&lt;td&gt;05:24:09–05:41:33 (17 m 24 s)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;1.34 → 1.35 (PR 10)&lt;/td&gt;
&lt;td&gt;05:43:54&lt;/td&gt;
&lt;td&gt;05:50:52&lt;/td&gt;
&lt;td&gt;05:43:03–06:00:26 (17 m 23 s)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;1.35 → 1.36 (PR 11)&lt;/td&gt;
&lt;td&gt;06:02:47&lt;/td&gt;
&lt;td&gt;06:09:56&lt;/td&gt;
&lt;td&gt;06:01:58–06:24:27 (22 m 29 s)&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;All four EKS &lt;code&gt;VersionUpdate&lt;/code&gt; entries are Successful. The node group update started 7 m 18 s, 6 m 45 s, 6 m 58 s and 7 m 9 s after the control-plane update. Each plan changed the cluster version, the node group version and the add-on versions in place. PR 10 was merged at 05:43:01 and PR 11 at 06:01:56.&lt;/p&gt;

&lt;p&gt;PR 6 also moved the four add-on versions from the module directory into a variable set in the directory the CI workflow watches. The workflow triggers only on that directory, so a pull request that changed only the add-on versions where they were written would have run nothing. The add-ons went from aws-ebs-csi-driver v1.37.0-eksbuild.1, coredns v1.11.3-eksbuild.1, kube-proxy v1.32.0-eksbuild.2 and vpc-cni v1.19.0-eksbuild.1 to the default versions per Kubernetes version from &lt;code&gt;aws eks describe-addon-versions&lt;/code&gt;: coredns v1.12.4-eksbuild.82 (1.33, 1.34), v1.13.2-eksbuild.75 (1.35), v1.14.7-eksbuild.35 (1.36); kube-proxy v1.33.10-eksbuild.69, v1.34.6-eksbuild.69, v1.35.3-eksbuild.69, v1.36.0-eksbuild.65; vpc-cni v1.22.4-eksbuild.14 and aws-ebs-csi-driver v1.68.0-eksbuild.1 for all four.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fv6cywtvxnilsj0rqjlfw.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fv6cywtvxnilsj0rqjlfw.png" alt="Stacked bars of node counts by kubelet minor version at nine readings between 03:15 and 06:34 UTC, with the control-plane version drawn as a step line above them, rising from 1.32 to 1.36." width="799" height="493"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Nodes by kubelet minor at each reading on 2026-11-16, against the control-plane version. The first reading has 11 nodes on 1.32, the last 10 nodes on 1.36.5.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The control plane was at most two minors ahead of the oldest node (06:24–06:32: one node on 1.34, control plane on 1.36); the Kubernetes version skew policy allows a kubelet up to three minors older than the API server. After the 05:42 mistake with &lt;code&gt;api-a&lt;/code&gt;, the script that clears blocked nodes was changed to delete one pod per workload per run and to skip a workload that has a pod not Ready.&lt;/p&gt;

&lt;h3&gt;
  
  
  Fixes made on the way
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Redis image (PR 7, merged 05:15:30).&lt;/strong&gt; Images moved to &lt;code&gt;docker.io/bitnamilegacy&lt;/code&gt;, same tags. The StatefulSet had the new image, but the stuck pod was not replaced and the Helm release stayed in &lt;code&gt;pending-upgrade&lt;/code&gt;; after &lt;code&gt;kubectl delete pod redis-master-0&lt;/code&gt; at 05:20 the pod was &lt;code&gt;2/2 Running&lt;/code&gt; in 51 s.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Subnet addresses (05:21, then PR 9).&lt;/strong&gt; &lt;code&gt;kubectl -n kube-system set env ds/aws-node -c aws-node WARM_IP_TARGET=3 MINIMUM_IP_TARGET=6&lt;/code&gt;. The next read: zone-b 127 free, zone-a 174 free; at 05:42, 181 and 156. PR 9 (merged 05:24:06) put the same two values into the add-on's &lt;code&gt;configuration_values&lt;/code&gt;, and its apply performed the 1.34 update that EKS had refused in PR 8's apply.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Prometheus memory (PR 12, merged 06:26:38).&lt;/strong&gt; Requests to 1536Mi and the limit to 3Gi; at 06:34 the pod was &lt;code&gt;2/2 Running&lt;/code&gt; with 0 restarts. Why it ran out of memory was not investigated: no memory series was read before or after. The audit of 2026-11-14 had listed the same container as OOMKilled once, at 14:34 that day, with 178,024 active series.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Redis target group (PR 13, merged 06:37:19).&lt;/strong&gt; An NLB target group for Redis had one target, registered by hand: the pod's old IP, now &lt;code&gt;unhealthy&lt;/code&gt;. We registered the current IP, then added a TargetGroupBinding; the target is &lt;code&gt;healthy&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Support type (PR 14).&lt;/strong&gt; &lt;code&gt;cluster_upgrade_policy.support_type = "STANDARD"&lt;/code&gt;; the EKS &lt;code&gt;UpgradePolicyUpdate&lt;/code&gt; was Successful at 07:08:06, and &lt;code&gt;describe-cluster&lt;/code&gt; reads 1.36 / ACTIVE / STANDARD. Standard support for 1.36 ends 2027-08-02.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  The audit, run again
&lt;/h3&gt;

&lt;p&gt;A fresh kubegap full collection at 08:07:59–08:16 (kubegap 0adb216, rubric 0.1.96, read-only) no longer lists the five findings named at the end of "What to check in your cluster now". Only those five findings are compared: 180 of 762 collection commands failed in the second run (21 of 652 in the first), so the totals are not comparable. The second run has one new Karpenter finding, &lt;code&gt;karpenter.metrics-unreadable&lt;/code&gt; (warn, "0 of 2 pods /metrics"); the cause was not looked into.&lt;/p&gt;

&lt;h3&gt;
  
  
  Not changed
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The &lt;code&gt;karpenter.sh/do-not-disrupt&lt;/code&gt; annotations and &lt;code&gt;terminationGracePeriod&lt;/code&gt;.&lt;/strong&gt; The three annotated pods still carry the annotation and no NodePool sets &lt;code&gt;terminationGracePeriod&lt;/code&gt;. We expect the next drift to be blocked on the same pods.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Single-replica Deployments.&lt;/strong&gt; Not changed.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Four ALB target groups whose only targets are &lt;code&gt;unhealthy&lt;/code&gt;&lt;/strong&gt;, with &lt;code&gt;Health checks failed with these codes: [404]&lt;/code&gt; (two) and &lt;code&gt;[307]&lt;/code&gt; (two): the health check path &lt;code&gt;/&lt;/code&gt; does not return 200 on those apps. Two of the four were already &lt;code&gt;unhealthy&lt;/code&gt; at 05:15. Not changed.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A deprecated API for 1.37.&lt;/strong&gt; The insights now list "Deprecated APIs removed in Kubernetes v1.37": &lt;code&gt;/apis/storage.k8s.io/v1beta1/volumeattributesclasses&lt;/code&gt;, replaced by v1, used by &lt;code&gt;csi-resizer&lt;/code&gt; and &lt;code&gt;argocd-application-controller&lt;/code&gt;, one request each, both at 05:33. Not changed.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Older Helm charts&lt;/strong&gt; (load balancer controller v2.7.2, Argo CD v2.13.0, external-secrets v0.9.13, kube-prometheus-stack 56.0.0, KEDA 2.17.2, external-dns 0.14.0) were not upgraded. On 1.36 all were Ready and had 0 error lines in 15 minutes. That is all we checked.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Backups.&lt;/strong&gt; Still no AWS Backup plan and no DLM policy for the 13 volumes, and the Terraform state bucket still has versioning off.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A statement we have not verified.&lt;/strong&gt; We told the owner during the work that EKS cannot roll a version back. The insights list now contains "rollback compatibility" entries; we did not test or read about that feature.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Where else this happened
&lt;/h2&gt;

&lt;p&gt;The Incident Library, our searchable collection of public post-incident reports on benchlet.com, has no case of an upgrade held back by Karpenter's own version range or of a control-plane update refused for free subnet addresses. The nearest public cases:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;UK Ministry of Justice, 2025-03-18: during an EKS 1.30 upgrade the descheduler tried to reschedule every pod on the cordoned nodes at once and production workloads went down. It is the nearest case to our 05:13–05:16 overlap. &lt;a href="https://runbooks.cloud-platform.service.justice.gov.uk/incidents/2025-03-18-aws-eks-upgrade-to-1-30.html" rel="noopener noreferrer"&gt;Source&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Stytch, 2023-02-23: deleting EKS managed node groups also deleted Karpenter's instance profile, and every worker node shut down. Our &lt;code&gt;5 to destroy&lt;/code&gt; plan implied the same outcome by a different cause. &lt;a href="https://stytch.com/blog/stytch-postmortem-2023-02-23/" rel="noopener noreferrer"&gt;Source&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;PortSwigger, 2024-10-24: all customer pods on EKS crash-looped after Karpenter moved nodes to Bottlerocket v1.26.0, which broke Java; the recorded contributing factor is an unpinned version; our EC2NodeClass had the same condition (&lt;code&gt;al2@latest&lt;/code&gt;) until PR 5. &lt;a href="https://github.com/portswigger-cloud/postmortems/blob/main/2024/10/saas-outage.md" rel="noopener noreferrer"&gt;Source&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;More: &lt;a href="https://benchlet.com/incidentlib/app?symptom=upgrade-stuck" rel="noopener noreferrer"&gt;Upgrade stuck, 11 cases&lt;/a&gt; (two of them are not Kubernetes) and &lt;a href="https://benchlet.com/incidentlib/app?component=karpenter" rel="noopener noreferrer"&gt;Karpenter as a component, 5 cases&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Timeline
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Time (UTC)&lt;/th&gt;
&lt;th&gt;Event&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;2026-11-14 16:51&lt;/td&gt;
&lt;td&gt;Audit run lists Kubernetes 1.32 and Karpenter 0.37.0 as problem 5 of its top five.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2026-11-16 03:15&lt;/td&gt;
&lt;td&gt;Our reading: server 1.32, 11 nodes, 9 on Amazon Linux 2; "Amazon Linux 2 compatibility" ERROR for 1.33.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;03:33–03:35&lt;/td&gt;
&lt;td&gt;13 EBS snapshots; the three Karpenter CRDs labelled for the &lt;code&gt;karpenter-crd&lt;/code&gt; release.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;03:36:38–03:59:07&lt;/td&gt;
&lt;td&gt;PR 1 to PR 4: Karpenter 0.37.0 → 0.37.8 → 1.0.14 → manifests to v1 → 1.14.3 (times per PR in the table above).&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;03:51&lt;/td&gt;
&lt;td&gt;PR 3 first plan: &lt;code&gt;5 to add, 0 to change, 5 to destroy&lt;/code&gt;. Not applied.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;03:54:53&lt;/td&gt;
&lt;td&gt;"disrupting nodeclaim(s) via replace"; at 03:55:18 all eight NodeClaims that existed before the apply &lt;code&gt;Drifted&lt;/code&gt;.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;04:00:53–04:01:59&lt;/td&gt;
&lt;td&gt;PR 5: &lt;code&gt;al2023@v20261112&lt;/code&gt;.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;04:52:43–04:56&lt;/td&gt;
&lt;td&gt;2 nodes still on AL2, &lt;code&gt;DisruptionBlocked&lt;/code&gt;; cordon and pod delete; all 12 nodes on AL2023.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;04:58:50&lt;/td&gt;
&lt;td&gt;Control-plane update to 1.33 starts.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;05:13:09–05:16:54&lt;/td&gt;
&lt;td&gt;27 pods not Running, &lt;code&gt;pull QPS exceeded&lt;/code&gt;, 11 target groups with no healthy target, Redis image &lt;code&gt;NotFound&lt;/code&gt;.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;05:20:22&lt;/td&gt;
&lt;td&gt;PR 8 apply refused: &lt;code&gt;Atleast one subnet in each AZ should have 2 free IPs&lt;/code&gt;.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;05:21&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;WARM_IP_TARGET=3&lt;/code&gt;, &lt;code&gt;MINIMUM_IP_TARGET=6&lt;/code&gt;; zone-b from 0 to 127 free.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;05:24:58&lt;/td&gt;
&lt;td&gt;Control-plane update to 1.34 starts (PR 9).&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;05:42&lt;/td&gt;
&lt;td&gt;Our script deletes both replicas of &lt;code&gt;api-a&lt;/code&gt;; script changed.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;05:43:54&lt;/td&gt;
&lt;td&gt;Control-plane update to 1.35 starts.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;06:02:47&lt;/td&gt;
&lt;td&gt;Control-plane update to 1.36 starts.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;06:24–06:26:38&lt;/td&gt;
&lt;td&gt;Prometheus &lt;code&gt;OOMKilled&lt;/code&gt;; PR 12 raises the limit to 3Gi.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;06:34:48&lt;/td&gt;
&lt;td&gt;Server &lt;code&gt;v1.36.5-eks-91f89a9&lt;/code&gt;; 10 nodes on &lt;code&gt;v1.36.5&lt;/code&gt;, all AL2023; 0 pods outside Running/Completed.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;07:04–07:08:06&lt;/td&gt;
&lt;td&gt;Snapshots deleted, unused; support type &lt;code&gt;STANDARD&lt;/code&gt;.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;08:07:59–08:16&lt;/td&gt;
&lt;td&gt;Audit run again: the five findings are absent; one new warn.&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  FAQ
&lt;/h2&gt;

&lt;h3&gt;
  
  
  karpenter version is not compatible with K8s version 1.32
&lt;/h3&gt;

&lt;p&gt;The controller's supported range ends below 1.32: 0.37.x allows Kubernetes 1.23–1.30 and 1.0.14 allows 1.25–1.31. Both log this once per start and keep creating nodes; we observed that on 1.32. Releases from 1.2 include 1.32. Version 1.14.3 words it &lt;code&gt;karpenter is not compatible with kubernetes version&lt;/code&gt;, so grep for both sentences.&lt;/p&gt;

&lt;h3&gt;
  
  
  Karpenter upgrade 0.37 to 1.0
&lt;/h3&gt;

&lt;p&gt;Go to the latest 0.37 patch with &lt;code&gt;webhook.enabled=true&lt;/code&gt; on the &lt;code&gt;karpenter&lt;/code&gt; and &lt;code&gt;karpenter-crd&lt;/code&gt; charts, then 1.0.x, then move manifests to v1 and wait until the CRDs' &lt;code&gt;storedVersions&lt;/code&gt; is &lt;code&gt;["v1"]&lt;/code&gt;. On Kubernetes 1.32 we skipped 1.1.x, whose range ends at 1.31 and whose source returns the version error. We did not run 1.1.x.&lt;/p&gt;

&lt;h3&gt;
  
  
  kubectl_manifest must be replaced api_version forces replacement
&lt;/h3&gt;

&lt;p&gt;Changing &lt;code&gt;karpenter.sh/v1beta1&lt;/code&gt; to &lt;code&gt;karpenter.sh/v1&lt;/code&gt; in a &lt;code&gt;kubectl_manifest&lt;/code&gt; planned &lt;code&gt;5 to add, 0 to change, 5 to destroy&lt;/code&gt; for us. Deleting a NodePool deletes its nodes. We ran &lt;code&gt;terraform state rm&lt;/code&gt; on the five addresses and added &lt;code&gt;import&lt;/code&gt; blocks; the plan became &lt;code&gt;5 to import, 0 to add, 5 to change, 0 to destroy&lt;/code&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  Atleast one subnet in each AZ should have 2 free IPs
&lt;/h3&gt;

&lt;p&gt;EKS returned this &lt;code&gt;InvalidRequestException&lt;/code&gt; from &lt;code&gt;UpdateClusterVersion&lt;/code&gt; when our /24 subnet in one zone had 0 free addresses: 21 network interfaces held 251 IPs under &lt;code&gt;WARM_ENI_TARGET=1&lt;/code&gt;. Setting &lt;code&gt;WARM_IP_TARGET=3&lt;/code&gt; and &lt;code&gt;MINIMUM_IP_TARGET=6&lt;/code&gt; on the aws-node DaemonSet left 127 free at the next read, and the next apply performed the update.&lt;/p&gt;

&lt;h3&gt;
  
  
  DisruptionBlocked Pod has "karpenter.sh/do-not-disrupt" annotation
&lt;/h3&gt;

&lt;p&gt;Karpenter excludes a node from drift replacement while a pod on it carries that annotation, unless the NodePool sets &lt;code&gt;terminationGracePeriod&lt;/code&gt; (Karpenter disruption docs). Ours set none, and two of our nodes stayed on AL2 for 50 minutes. We cordoned the node and deleted the annotated pod, one pod per workload at a time.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://benchlet.com/guides/eks-1-32-upgrade-karpenter-0-37-not-compatible-k8s-version-al2-al2023/" rel="noopener noreferrer"&gt;benchlet.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>kubernetes</category>
      <category>karpenter</category>
      <category>eks</category>
      <category>aws</category>
    </item>
    <item>
      <title>MinIO memory usage high on Kubernetes but never OOMKilled? Why the working set sits at 95% of the limit</title>
      <dc:creator>KYUNGHWAN KIM</dc:creator>
      <pubDate>Sun, 15 Nov 2026 16:00:22 +0000</pubDate>
      <link>https://dev.to/kkhdevs/minio-memory-usage-high-on-kubernetes-but-never-oomkilled-why-the-working-set-sits-at-95-of-the-2pi1</link>
      <guid>https://dev.to/kkhdevs/minio-memory-usage-high-on-kubernetes-but-never-oomkilled-why-the-working-set-sits-at-95-of-the-2pi1</guid>
      <description>&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fu3c61fbhamm0f84ewzxh.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fu3c61fbhamm0f84ewzxh.png" alt="One bar for the minio container's 512 MiB cgroup, filled to 1.3 MB below the limit: anon 396 MiB, page cache 83 MiB, slab 30 MiB, with a dashed line above the anon segment ending at 460.8 MiB, labelled &amp;quot;Go soft limit, which counts MinIO's own memory (anon) only, not cache or slab: 90 % = 460.8 MiB&amp;quot;, a header &amp;quot;minio-0 working set p95 / limit = 0.95 · OOMKilled: 0 in 30 days&amp;quot; and a footer &amp;quot;memory.current 510.8 of 512 MiB, oom_kill 0&amp;quot;." width="800" height="420"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The 512 MiB cgroup on 2026-11-15 at 13:49 UTC: full, with most of the top made of page cache the kernel can reclaim.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;If your MinIO memory usage is high on Kubernetes (&lt;code&gt;kubectl top&lt;/code&gt; shows &lt;code&gt;495Mi&lt;/code&gt; against a &lt;code&gt;512Mi&lt;/code&gt; limit, an audit reports &lt;code&gt;7d memory working set p95 / limit = 0.95&lt;/code&gt;) but you never see a MinIO pod OOMKilled (&lt;code&gt;container_oom_events_total&lt;/code&gt; is still &lt;code&gt;0&lt;/code&gt;), MinIO is probably not leaking: community releases read the container's own cgroup &lt;code&gt;memory.max&lt;/code&gt; and use 90 % of it as their budget (since PR #19839, May 2024, this includes the Go soft memory limit), so the memory fills the limit by design. To see whether yours is the same case, read the pod's cgroup (step 2 below): it is the same case when &lt;code&gt;memory.current&lt;/code&gt; is within a few MB of &lt;code&gt;memory.max&lt;/code&gt;, &lt;code&gt;anon&lt;/code&gt; stays under 90 % of &lt;code&gt;memory.max&lt;/code&gt;, and &lt;code&gt;memory.events&lt;/code&gt; shows &lt;code&gt;oom_kill 0&lt;/code&gt;. The cluster is a 2-node k3s cluster that I run (&lt;code&gt;server-1&lt;/code&gt; and &lt;code&gt;agent-1&lt;/code&gt;); MinIO runs as a single-replica StatefulSet &lt;code&gt;minio&lt;/code&gt;, pod &lt;code&gt;minio-0&lt;/code&gt;, one drive, image &lt;code&gt;minio/minio:RELEASE.2025-04-22T22-12-26Z&lt;/code&gt;. All times are UTC, 2026-11-15 unless a date is given.&lt;/p&gt;

&lt;h2&gt;
  
  
  What we saw: MinIO memory usage high on Kubernetes, 495Mi of 512Mi for 30 days, no MinIO pod OOMKilled
&lt;/h2&gt;

&lt;p&gt;The first signal was an automated audit of the cluster (kubegap, described under the checks below), run on 2026-11-14 at 16:25:29. Its problem 5, translated from Korean: "minio's memory is about to reach its limit." The reason it gave: a small rise in requests ends in OOMKilled, and with one replica the object store stops. The finding was &lt;code&gt;resources.memory-limit-near-p95&lt;/code&gt;, &lt;code&gt;pod minio-0: 7d memory working set p95 / limit = 0.95&lt;/code&gt;, with a KRR recommendation (KRR computes resource recommendations from Prometheus history) of about 582 MB for the memory request.&lt;/p&gt;

&lt;p&gt;The pod at 13:47:18 (names masked, pod IP and container id removed):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ kubectl -n apps get pod minio-0 -o wide; kubectl -n apps get pod minio-0 -o jsonpath='{.status.qosClass}{"\n"}{.status.containerStatuses[0].restartCount}{"\n"}{.status.containerStatuses[0].lastState}{"\n"}{.spec.containers[0].resources}{"\n"}'
NAME      READY   STATUS    RESTARTS      AGE    NODE
minio-0   1/1     Running   2 (36h ago)   128d   server-1
Burstable
2
{"terminated":{"exitCode":255,"finishedAt":"2026-11-14T01:12:30Z","reason":"Unknown","startedAt":"2026-07-11T00:26:47Z"}}
{"limits":{"cpu":"300m","memory":"512Mi"},"requests":{"cpu":"50m","memory":"128Mi"}}

$ kubectl top pod -n apps minio-0 --containers
POD       NAME    CPU(cores)   MEMORY(bytes)
minio-0   minio   30m          495Mi
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The last restart (exit 255, reason &lt;code&gt;Unknown&lt;/code&gt;) was the k3s restart of 2026-11-14, not memory. The pod is pinned to &lt;code&gt;server-1&lt;/code&gt; by &lt;code&gt;nodeSelector&lt;/code&gt; &lt;code&gt;storage: server-1&lt;/code&gt;; &lt;code&gt;server-1&lt;/code&gt; has 12 CPUs and 32207388Ki of memory, with 23638Mi (75 %) requested.&lt;/p&gt;

&lt;p&gt;Prometheus at 13:47:29, in MiB (two values where the current and the previous container each have a series):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;container_memory_working_set_bytes            now 495.1
  max_over_time [30d]                          498.1 / 505.9
  quantile_over_time(0.95) [7d]                494.5 / 486.8
container_memory_rss                           now 395.6
  max_over_time [30d]                          413.5 / 440.4
  quantile_over_time(0.95) [7d]                395.9 / 434.6
container_memory_cache                         now 82.3
container_memory_usage_bytes                   now 510.3
  max_over_time [30d]                          511.6 / 511.7
container_oom_events_total                     0
increase(container_oom_events_total[30d])      0 / 0
container_memory_failcnt                       0
kube_pod_container_status_last_terminated_reason   reason="Unknown" 1
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Daily maxima over 30 days (13:48:02, MiB):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;container_memory_rss (max per day)
  10-16=435 10-17=439 10-18=436 10-19=434 10-20=437 10-21=437 10-22=439 10-23=440 10-24=436 10-25=439
  10-26=438 10-27=439 10-28=438 10-29=439 10-30=440 10-31=438 11-01=440 11-02=440 11-03=438 11-04=437
  11-05=438 11-06=438 11-07=438 11-08=440 11-09=440 11-10=440 11-11=440 11-12=438 11-13=438 11-14=439 11-15=421
container_memory_working_set_bytes (max per day)
  10-16=503 10-17=503 10-18=500 10-19=502 10-20=502 10-21=503 10-22=502 10-23=502 10-24=502 10-25=503
  10-26=504 10-27=504 10-28=503 10-29=504 10-30=505 10-31=505 11-01=505 11-02=504 11-03=506 11-04=506
  11-05=506 11-06=504 11-07=502 11-08=492 11-09=494 11-10=496 11-11=491 11-12=489 11-13=491 11-14=491 11-15=499
container_memory_cache (max per day)
  between 55 and 70 every day, 152 on 11-14 (the restart day), 90 on 11-15
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;anon's daily maximum was 434–440 MiB every day for 30 days (421 on 11-15, after the restart of 11-14). A leak grows; this did not.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ff99kjwoqxe8s086vfjhz.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ff99kjwoqxe8s086vfjhz.png" alt="Prometheus graphs over 30 days for pod minio-0: container_memory_working_set_bytes flat at about 520 MB (about 500 MiB), container_memory_rss flat at about 450 MB, both dropping at the restart of Nov 14; at the right edge, after the change, the working-set line jumps above its old level, to about 650 MB, while the rss line cannot be told apart within the last few pixels; kube_pod_container_resource_limits flat at 536.87 MB (512 MiB) for 30 days, then a vertical step to 805.3 MB (768 MiB) at the right edge." width="800" height="647"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Prometheus, 30 days, captured after the change (values in decimal MB). Flat for a month at 512Mi; the restart of 11-14 and the 768Mi pod and its load test are at the right edge.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Inside the cgroup at 13:49:02. The recorded command was step 2's command with &lt;code&gt;sock&lt;/code&gt; and &lt;code&gt;kernel&lt;/code&gt; added to the &lt;code&gt;memory.stat&lt;/code&gt; filter and a check of environment variable names appended. Below is step 2's command and the part of that output it prints: the same lines without &lt;code&gt;sock 0&lt;/code&gt; (the kernel printed no &lt;code&gt;kernel&lt;/code&gt; line) and without the environment check.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ kubectl -n apps exec minio-0 -- sh -c 'echo max=$(cat /sys/fs/cgroup/memory.max) current=$(cat /sys/fs/cgroup/memory.current); cat /sys/fs/cgroup/memory.events; while read k v; do case $k in anon|file|active_file|inactive_file|slab) echo "$k $v";; esac; done &amp;lt; /sys/fs/cgroup/memory.stat'
max=536870912 current=535588864
low 0
high 0
max 0
oom 0
oom_kill 0
anon 415117312
file 86994944
inactive_file 30425088
active_file 56569856
slab 31441592
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;memory.current&lt;/code&gt; was 1.3 MB below &lt;code&gt;memory.max&lt;/code&gt; (512 MiB). Of that, anon (memory the process allocated) was 395.9 MiB, file (page cache) 83.0 MiB (53.9 active, 29.0 inactive) and slab 30.0 MiB. &lt;code&gt;oom_kill&lt;/code&gt; was 0. The same recording printed every environment variable name in the container that matched &lt;code&gt;GOMEMLIMIT&lt;/code&gt;, &lt;code&gt;GOGC&lt;/code&gt; or &lt;code&gt;MINIO_API*&lt;/code&gt;, and it printed none, so none of them was set. The only memory alert rules in the cluster were &lt;code&gt;KubeMemoryQuotaOvercommit&lt;/code&gt;, &lt;code&gt;NodeMemoryMajorPagesFaults&lt;/code&gt; and &lt;code&gt;NodeMemoryHighUtilization&lt;/code&gt;: nothing for an OOM kill or a container near its limit.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why the working set is near the memory limit but no OOM: MinIO sizes itself from 90 % of the cgroup limit
&lt;/h2&gt;

&lt;p&gt;MinIO reads its own limit. In the source at tag &lt;code&gt;RELEASE.2025-04-22T22-12-26Z&lt;/code&gt;, &lt;code&gt;cgroupMemLimit()&lt;/code&gt; reads &lt;code&gt;/sys/fs/cgroup/memory.max&lt;/code&gt; (with a cgroup v1 fallback) and &lt;code&gt;availableMemory()&lt;/code&gt; returns 90 % of it, &lt;code&gt;available = (limit * 9) / 10&lt;/code&gt; (&lt;code&gt;cmd/handler-api.go&lt;/code&gt; lines 62–106). At 512Mi that is 483,183,820 bytes, 460.8 MiB. MinIO uses that number three times:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Go soft memory limit.&lt;/strong&gt; At startup &lt;code&gt;setMaxResources&lt;/code&gt; calls &lt;code&gt;debug.SetMemoryLimit(int64(ctx.MemLimit))&lt;/code&gt; (&lt;code&gt;cmd/server-rlimit.go&lt;/code&gt; lines 84–86), and &lt;code&gt;MemLimit&lt;/code&gt; defaults to that 90 %. The Go documentation says the runtime then adjusts "the frequency of garbage collections and returning memory to the underlying system" to stay under it, and that the limit does not count kernel memory held for the process (&lt;code&gt;runtime/debug/garbage.go&lt;/code&gt; lines 181–197). Page cache and slab are outside it. This came in with MinIO PR #19839 (merged 2024-05-30).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Concurrent request pool.&lt;/strong&gt; &lt;code&gt;apiRequestsMaxPerNode = maxMem / (drives × (1 MiB + 32 KiB) + 2 × 1 MiB)&lt;/code&gt; (&lt;code&gt;cmd/handler-api.go&lt;/code&gt; lines 124–147), 3,178,496 bytes per request on one drive: 152 requests at 512Mi, 228 at 768Mi (computed from the source, not read from the server). Requests over the pool get HTTP 429 &lt;code&gt;TooManyRequests&lt;/code&gt; at once.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Erasure byte pool.&lt;/strong&gt; &lt;code&gt;newErasureServerPools&lt;/code&gt; sizes it at &lt;code&gt;min(1024, budget / 2 MiB)&lt;/code&gt; buffers of 2 MiB capacity (&lt;code&gt;cmd/erasure-server-pool.go&lt;/code&gt; lines 93–116): 230 at 512Mi, 345 at 768Mi. Each buffer is allocated with a 2 MiB capacity (&lt;code&gt;internal/bpool/bpool.go&lt;/code&gt; line 68), and a buffer that is handed back stays in the pool's channel until the process exits.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F54zynd4j2r7q1i70wrbn.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F54zynd4j2r7q1i70wrbn.png" alt="Two bars to scale. 512Mi: 460.8 MiB MinIO budget (Go soft limit, 152 requests, 230 × 2 MiB buffers) and a 51.2 MiB margin. 768Mi: 691.2 MiB budget (228 requests, 345 × 2 MiB buffers) and a 76.8 MiB margin." width="799" height="413"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;What MinIO does with the limit it is given, at 512Mi and at 768Mi. All three budgets grow with the limit; the margin stays 10 %.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The Kubernetes side explains why 95 % was not a kill. The documentation says memory limits "are enforced by the kernel with out of memory (OOM) kills" and "terminations only happen when the kernel detects memory pressure" (&lt;a href="https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/" rel="noopener noreferrer"&gt;Resource Management for Pods and Containers&lt;/a&gt;). The working set is usage minus &lt;code&gt;inactive_file&lt;/code&gt; (the kubelet excludes it "as it assumes that memory is reclaimable under pressure", &lt;a href="https://kubernetes.io/docs/concepts/scheduling-eviction/node-pressure-eviction/" rel="noopener noreferrer"&gt;Node-pressure eviction&lt;/a&gt;), so it "typically also includes some cached (file-backed) memory" (&lt;a href="https://kubernetes.io/docs/tasks/debug/debug-cluster/resource-metrics-pipeline/#memory" rel="noopener noreferrer"&gt;Resource metrics pipeline&lt;/a&gt;). Active page cache is counted in the working set but is still reclaimable. That is the difference between &lt;code&gt;container_memory_working_set_bytes&lt;/code&gt; and &lt;code&gt;container_memory_rss&lt;/code&gt;: on cgroup v2, cAdvisor's &lt;code&gt;container_memory_rss&lt;/code&gt; is the cgroup's &lt;code&gt;anon&lt;/code&gt;, not the process RSS, and the working set is usage minus &lt;code&gt;inactive_file&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;My reading (inference from the numbers above): at 512Mi MinIO had set the Go soft memory limit to 460.8 MiB, so anon stayed between 391.5 and 440.4 MiB over 30 days before the 11-14 restart (daily maxima 434–440 MiB; 255.9 MiB right after the restart), and the rest of the cgroup was page cache (83 MiB, which the kernel can reclaim) and slab (30 MiB). The 95 % working set was MinIO filling the limit it was given, not a leak. The risk was the margin: 512 − 460.8 = 51.2 MiB for everything outside the Go soft limit. Slab is neither anon nor page cache, and its 30 MiB counts against that margin.&lt;/p&gt;

&lt;h2&gt;
  
  
  What to check in your cluster now
&lt;/h2&gt;

&lt;p&gt;Steps 1 to 4 are read-only. Replace &lt;code&gt;&amp;lt;ns&amp;gt;&lt;/code&gt; and &lt;code&gt;&amp;lt;pod&amp;gt;&lt;/code&gt;, and &lt;code&gt;minio&lt;/code&gt; if your container has another name.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;1. Limit and QoS class.&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl -n &lt;span class="o"&gt;&amp;lt;&lt;/span&gt;ns&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; get pod &lt;span class="o"&gt;&amp;lt;&lt;/span&gt;pod&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; -o &lt;span class="nv"&gt;jsonpath&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;'{.status.qosClass}{"\n"}{range .spec.containers[*]}{.name}{" "}{.resources}{"\n"}{end}'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Ours before: Burstable, memory request 128Mi, limit 512Mi. MinIO sizes itself from that limit.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. What the memory inside the limit is made of (cgroup v2).&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl -n &lt;span class="o"&gt;&amp;lt;&lt;/span&gt;ns&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;exec&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt;pod&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; -- sh -c &lt;span class="s1"&gt;'echo max=$(cat /sys/fs/cgroup/memory.max) current=$(cat /sys/fs/cgroup/memory.current); cat /sys/fs/cgroup/memory.events; while read k v; do case $k in anon|file|active_file|inactive_file|slab) echo "$k $v";; esac; done &amp;lt; /sys/fs/cgroup/memory.stat'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The MinIO image has no &lt;code&gt;grep&lt;/code&gt;; the &lt;code&gt;while read&lt;/code&gt; loop does the filtering. Same problem as ours when &lt;code&gt;current&lt;/code&gt; is within a few MB of &lt;code&gt;max&lt;/code&gt;, &lt;code&gt;anon&lt;/code&gt; is under 90 % of &lt;code&gt;max&lt;/code&gt;, MinIO's own budget (ours: current 510.8 MiB of 512 MiB, anon 395.9 MiB against 460.8 MiB, file 83.0 MiB, slab 30.0 MiB), and &lt;code&gt;oom_kill&lt;/code&gt; is 0. &lt;code&gt;max&lt;/code&gt; in &lt;code&gt;memory.events&lt;/code&gt; counts the times the cgroup hit its limit and the kernel reclaimed; &lt;code&gt;oom_kill&lt;/code&gt; counts kills.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Flat or growing.&lt;/strong&gt; In Prometheus, graph both over 30 days with a 1-day step:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;max_over_time(container_memory_rss{namespace="&amp;lt;ns&amp;gt;",pod="&amp;lt;pod&amp;gt;",container="minio"}[1d]) / 2^20
max_over_time(container_memory_working_set_bytes{namespace="&amp;lt;ns&amp;gt;",pod="&amp;lt;pod&amp;gt;",container="minio"}[1d]) / 2^20
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Flat day after day means sized, not leaking. Ours: anon 434–440 MiB every day, working set 489–506 MiB, against 512 MiB.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. Has it ever been killed?&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;increase(container_oom_events_total{namespace="&amp;lt;ns&amp;gt;",pod="&amp;lt;pod&amp;gt;",container="minio"}[30d])
kube_pod_container_status_last_terminated_reason{namespace="&amp;lt;ns&amp;gt;",pod="&amp;lt;pod&amp;gt;"}
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Ours: 0, and the last terminated reason &lt;code&gt;Unknown&lt;/code&gt;, not &lt;code&gt;OOMKilled&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;5. How much MinIO gives itself.&lt;/strong&gt; 90 % of &lt;code&gt;memory.max&lt;/code&gt;. The request pool size is returned in the &lt;code&gt;X-RateLimit-Limit&lt;/code&gt; header of S3 calls, including an unauthenticated list on &lt;code&gt;/&lt;/code&gt;: &lt;code&gt;kubectl -n &amp;lt;ns&amp;gt; port-forward svc/&amp;lt;svc&amp;gt; 9000:9000 &amp;amp; sleep 2; curl -s -D - -o /dev/null http://127.0.0.1:9000/ | grep -i x-ratelimit&lt;/code&gt;. From the source it should read 152 at 512Mi on one drive. We did not run this check on our cluster, so we have no output to show. Computed values on one drive:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;limit&lt;/th&gt;
&lt;th&gt;budget (90 %)&lt;/th&gt;
&lt;th&gt;request pool (1 drive)&lt;/th&gt;
&lt;th&gt;byte pool&lt;/th&gt;
&lt;th&gt;margin (10 %)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;512Mi&lt;/td&gt;
&lt;td&gt;460.8 MiB&lt;/td&gt;
&lt;td&gt;152&lt;/td&gt;
&lt;td&gt;230 × 2 MiB&lt;/td&gt;
&lt;td&gt;51.2 MiB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;768Mi&lt;/td&gt;
&lt;td&gt;691.2 MiB&lt;/td&gt;
&lt;td&gt;228&lt;/td&gt;
&lt;td&gt;345 × 2 MiB&lt;/td&gt;
&lt;td&gt;76.8 MiB&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;With more drives the request pool is smaller (the divisor is drives × 1.03 MiB + 2 MiB). If all of 1 to 4 match ours, the limit is being used as designed; what you decide is how big the margin should be.&lt;/p&gt;

&lt;p&gt;To compare every container's working set with its memory limit on a schedule, &lt;a href="https://github.com/KyungHwanKim-devs/kubegap" rel="noopener noreferrer"&gt;kubegap&lt;/a&gt;, an open-source audit of Kubernetes cluster operations that I maintain, has a check for it: its &lt;code&gt;resources.memory-limit-near-p95&lt;/code&gt; check divides each container's 7-day p95 working set by its memory limit, warns above 0.8 and fails above 0.9. That check found this problem. It does not read &lt;code&gt;memory.stat&lt;/code&gt;, so the split of step 2 and the OOM history of step 4 stay manual, or go into alerts like the two in the next section.&lt;/p&gt;

&lt;h2&gt;
  
  
  What we changed, and what we have not
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Commit 1, 13:56:48&lt;/strong&gt;, in the manifests repository that Argo CD syncs: memory request 128Mi → 768Mi, limit 512Mi → 768Mi. 768Mi was the report's example value, above KRR's 582 MB. The same commit added a PrometheusRule &lt;code&gt;container-oom-rules&lt;/code&gt; with two alerts:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;alert&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ContainerOOMKilled&lt;/span&gt;
  &lt;span class="na"&gt;expr&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;|&lt;/span&gt;
    &lt;span class="s"&gt;increase(kube_pod_container_status_restarts_total[1h]) &amp;gt; 0&lt;/span&gt;
    &lt;span class="s"&gt;and on (namespace, pod, container)&lt;/span&gt;
    &lt;span class="s"&gt;kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} == 1&lt;/span&gt;
&lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;alert&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ContainerMemoryNearLimit&lt;/span&gt;
  &lt;span class="na"&gt;expr&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;|&lt;/span&gt;
    &lt;span class="s"&gt;max by (namespace, pod, container) (container_memory_rss{container!=""})&lt;/span&gt;
    &lt;span class="s"&gt;/ on (namespace, pod, container)&lt;/span&gt;
    &lt;span class="s"&gt;max by (namespace, pod, container) (kube_pod_container_resource_limits{resource="memory"}) &amp;gt; 0.9&lt;/span&gt;
  &lt;span class="na"&gt;for&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;30m&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Argo CD synced at 13:58:47 and the new pod was Ready at 13:59:07: about 20 seconds without MinIO, because there is one replica. &lt;strong&gt;The first commit left the pod Burstable.&lt;/strong&gt; Its comment said "memory request = limit (Guaranteed)", but the CPU request (50m) was not equal to the CPU limit (300m), and the Kubernetes &lt;a href="https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/" rel="noopener noreferrer"&gt;Pod QoS&lt;/a&gt; rules require request = limit for CPU as well as memory on every container.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Commit 2, 13:59:42&lt;/strong&gt;: CPU request 50m → 300m. The pod was replaced at 13:59:48 and Ready at 14:00:04 (about 16 seconds). At 14:00:12: QoS &lt;code&gt;Guaranteed&lt;/code&gt;, requests = limits = cpu 300m / memory 768Mi, cgroup &lt;code&gt;memory.max&lt;/code&gt; 805306368, &lt;code&gt;mc ready local&lt;/code&gt; answered "The cluster 'local' is ready", and the PrometheusRule existed. &lt;code&gt;memory.current&lt;/code&gt; was 233.8 MiB right after the restart; with no load, anon rose to 352.8 MiB in the next 7 minutes and to 381 MiB by 14:34.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The load test.&lt;/strong&gt; &lt;strong&gt;Three attempts did not reach MinIO, and their numbers are not results:&lt;/strong&gt;&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;14:00:24: the Job pod could not pull its image (&lt;code&gt;kubectl logs&lt;/code&gt;: "image can't be pulled"). The later attempts pinned the Job to &lt;code&gt;server-1&lt;/code&gt;, where the MinIO image is cached.&lt;/li&gt;
&lt;li&gt;14:07:53: &lt;code&gt;mc alias set&lt;/code&gt; read the password, which starts with &lt;code&gt;-&lt;/code&gt;, as a flag: &lt;code&gt;mc: &amp;lt;ERROR&amp;gt; Invalid command usage, flag provided but not defined&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;14:24:07: the alias was written to &lt;code&gt;~/.mc&lt;/code&gt;, but this image sets &lt;code&gt;MC_CONFIG_DIR&lt;/code&gt;, so &lt;code&gt;mc&lt;/code&gt; used local paths. 32 "uploads" finished in 0.2 s with 0 errors, and the MinIO pod's anon did not move.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Attempt 4 wrote the alias into &lt;code&gt;$MC_CONFIG_DIR&lt;/code&gt; and checked that &lt;code&gt;mc ls&lt;/code&gt; reached the server before starting. From 14:34:55 to 14:38:35 it uploaded 8 MiB objects with 32 parallel &lt;code&gt;mc cp&lt;/code&gt; per pod, in 1, 2 and then 4 pods (32, 64, 128 concurrent), checked 256 MiB / 32 objects per pod on the server, read them back with 32 parallel &lt;code&gt;mc cat&lt;/code&gt;, and removed the bucket. 0 errors at every stage.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Concurrent&lt;/th&gt;
&lt;th&gt;Upload time&lt;/th&gt;
&lt;th&gt;Read time&lt;/th&gt;
&lt;th&gt;Peak anon (MiB)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;idle, before&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;381.1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;32&lt;/td&gt;
&lt;td&gt;6.8 s&lt;/td&gt;
&lt;td&gt;1.6 s&lt;/td&gt;
&lt;td&gt;458.1 during the stage; 468.1 at 14:35:57, 15 s after its reads ended&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;64&lt;/td&gt;
&lt;td&gt;14.8 s&lt;/td&gt;
&lt;td&gt;7 s&lt;/td&gt;
&lt;td&gt;618.1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;128&lt;/td&gt;
&lt;td&gt;about 32 s&lt;/td&gt;
&lt;td&gt;about 18 s&lt;/td&gt;
&lt;td&gt;677.8 (14:37:35)&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;memory.current&lt;/code&gt; reached 767.6 MiB of 768 at 14:37:39. After the test &lt;code&gt;memory.events&lt;/code&gt; showed &lt;code&gt;max 72&lt;/code&gt; (the cgroup hit its limit 72 times and the kernel reclaimed page cache), &lt;code&gt;oom 0&lt;/code&gt;, &lt;code&gt;oom_kill 0&lt;/code&gt;, and the pod had 0 restarts.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F19g7p26hitd5mudnga7t.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F19g7p26hitd5mudnga7t.png" alt="Time series 14:34:54–14:39:43 of minio-0 anon and memory.current in MiB, 1-second samples, stages 32, 64 and 128 concurrent shaded; reference lines at 512 (old limit), 691.2 (Go soft limit) and 768 (new limit). anon rises from 381 to 678 at 128 concurrent and stays at 618 after the load; current touches 767.6." width="800" height="480"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;At 768Mi under 32, 64 and 128 concurrent 8 MiB requests: anon above the old 512 MiB limit from 64 concurrent, under MinIO's own 691.2 MiB soft limit, and not back down afterwards.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;What this test does and does not show:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;At 64 and 128 concurrent requests anon alone (618–678 MiB) was above the old 512Mi limit. &lt;strong&gt;We did not run the same load at 512Mi&lt;/strong&gt;, so whether the old pod would have been OOMKilled, or held back by its own budget (152 requests, 460.8 MiB soft limit), is not measured.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;MinIO fills the new limit too.&lt;/strong&gt; &lt;code&gt;memory.current&lt;/code&gt; reached 767.6 of 768 MiB, and &lt;strong&gt;anon stayed at 618 MiB after the load&lt;/strong&gt; (14:39:43; 627.6 MiB at 14:46:51), 237 MiB above where it started. That is 80 % of 768 MiB held at rest. That these are buffers kept by the byte pool is an inference from the source; MinIO does not export the pool size as a metric.&lt;/li&gt;
&lt;li&gt;The &lt;code&gt;ContainerMemoryNearLimit&lt;/code&gt; alert fires at 90 % of 768 MiB, 691 MiB, which is MinIO's own Go soft limit (inference: it will fire only when MinIO is past its own target).&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;kubegap after the change.&lt;/strong&gt; A full run collected at 14:40:19 passed &lt;code&gt;resources.memory-limit-near-p95&lt;/code&gt; for all 92 targets. The same ratio read from Prometheus at 14:45:52 was 0.78; the new pod only existed since 13:59, so that p95 is mostly the load-test hour. The same run reported three other things for MinIO:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;resources.cpu-throttling&lt;/code&gt; warn: "throttling ratio now 0.53, 24 h or the pod's life 0.02". &lt;strong&gt;This came from the timing: the run started 1.5 minutes after the load test ended&lt;/strong&gt;, so it measured the test, not the day.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;resources.krr-recommendation&lt;/code&gt; now fails on CPU, not memory: "CPU request over by 0.288 cores: CPU request 0.3 → 0.012 cores". The 300m CPU request set to make the pod Guaranteed is far above what MinIO uses. KRR's memory figure, 796 MB, is just under 768Mi (805.3 MB) and rated GOOD.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;resilience.n-plus-one&lt;/code&gt;: memory without &lt;code&gt;server-1&lt;/code&gt; moved from OK (30.7GB vs 31.0GB requested) to short (30.7GB vs 31.8GB), because the request grew by 640Mi.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Faky9n8wlhycme1cp5vri.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Faky9n8wlhycme1cp5vri.png" alt="Table of kubegap findings for minio, before (run 2026-11-14 16:25 UTC, limit 512Mi) and after (run 2026-11-15 14:40 UTC, limit 768Mi). resources.memory-limit-near-p95: fail, p95 / limit = 0.95, before; pass for all 92 targets, same ratio read at 14:45 = 0.78, after; each drawn as a bar against warn 0.8 and fail 0.9 marks. resources.cpu-throttling: not raised before; warn after, ratio now 0.53, 24 h 0.02. resources.krr-recommendation: fail both times; before on memory, request 128 → 582 MB; after on CPU, request over by 0.288 cores (0.3 → 0.012), with memory 768 → 796 MB rated GOOD. resilience.n-plus-one: fail both times, memory 30.7 GB vs 31.0 GB OK before, 30.7 GB vs 31.8 GB short after. resilience.single-eligible-node: warn, unchanged." width="800" height="547"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The check that failed before the change passes after it; the rerun started 1.5 minutes after the load test, which is where the new CPU warning comes from.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What we have not changed:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The CPU request stays at 300m.&lt;/strong&gt; It was raised to 300m only to make the pod Guaranteed, and it now exceeds what MinIO uses (the KRR result above); we left it as is, a choice for the owner.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;MINIO_API_REQUESTS_MAX&lt;/code&gt; and &lt;code&gt;MINIO_MEMLIMIT&lt;/code&gt; are not set.&lt;/strong&gt; They are the in-process way to keep MinIO below its limit. My judgement: at 768Mi the measured run had 0 OOM kills, and what a smaller request pool would cost in throughput is not measured.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The first load-test pod could not pull the MinIO image&lt;/strong&gt; (attempt 1). The running pod uses the copy cached on &lt;code&gt;server-1&lt;/code&gt;. Outside this problem; not changed.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;No second replica or distributed mode.&lt;/strong&gt; It needs more drives and is outside this problem. A restart of &lt;code&gt;minio-0&lt;/code&gt; still means 16–20 seconds without the object store, as both commits showed.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Where else this happened
&lt;/h2&gt;

&lt;p&gt;The Incident Library, a searchable collection of public post-incident reports that I maintain at benchlet.com, has no MinIO case and no case of a working set near the limit without a kill. The three cases below show what the two new alerts are for: &lt;code&gt;ContainerMemoryNearLimit&lt;/code&gt; warns while a container is still below its memory limit, and &lt;code&gt;ContainerOOMKilled&lt;/code&gt; reports a kill like theirs after it happens; each case also shows what the kill took down elsewhere.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;a href="https://gitlab.com/gitlab-com/gl-infra/production/-/issues/6907" rel="noopener noreferrer"&gt;GitLab, 2022-04-25&lt;/a&gt;: the Consul agent in every GitLab.com GKE pod had a 500MB memory limit, was killed again and again when its memory spiked, and the Rails pods lost replica discovery and sent every query to the primary database.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.blog/news-insights/company-news/github-availability-report-july-2020/" rel="noopener noreferrer"&gt;GitHub, 2020-07-13&lt;/a&gt;: a container not needed for production traffic exceeded its memory limit, pods were marked unavailable, and restarts failed because a DNS change had cut access to the image registry; 4 h 25 min degraded.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://www.buildkitestatus.com/incidents/3vngjcnbkycl" rel="noopener noreferrer"&gt;Buildkite, 2026-05-20&lt;/a&gt;: autoscaling raised the number of pods an EKS Prometheus tracked until Prometheus needed more memory than its host, and it was killed in a loop, so services stopped scaling.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;More: &lt;a href="https://benchlet.com/incidentlib/app?component=memory-limit&amp;amp;symptom=oom-killed" rel="noopener noreferrer"&gt;OOMKilled + container memory limit, 15 cases&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Timeline
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Time (UTC)&lt;/th&gt;
&lt;th&gt;Event&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;2026-10-16 to 11-14&lt;/td&gt;
&lt;td&gt;anon daily max 434–440 MiB every day against a 512Mi limit; 0 OOM kills.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2026-11-14 01:12:30&lt;/td&gt;
&lt;td&gt;k3s restart; &lt;code&gt;minio-0&lt;/code&gt; restarts (reason &lt;code&gt;Unknown&lt;/code&gt;, exit 255).&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2026-11-14 16:25:29&lt;/td&gt;
&lt;td&gt;kubegap full run: problem 5, &lt;code&gt;7d memory working set p95 / limit = 0.95&lt;/code&gt;.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;13:47:18–13:49:42&lt;/td&gt;
&lt;td&gt;State recorded: 495Mi of 512Mi, &lt;code&gt;memory.current&lt;/code&gt; 1.3 MB below &lt;code&gt;memory.max&lt;/code&gt;, &lt;code&gt;oom_kill&lt;/code&gt; 0, no OOM alert rule.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;13:56:48&lt;/td&gt;
&lt;td&gt;Commit 1: memory 768Mi request and limit, &lt;code&gt;container-oom-rules&lt;/code&gt;.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;13:58:47&lt;/td&gt;
&lt;td&gt;Argo CD sync; old pod replaced.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;13:59:07&lt;/td&gt;
&lt;td&gt;New pod Ready at 768Mi, still Burstable.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;13:59:42&lt;/td&gt;
&lt;td&gt;Commit 2: CPU request 50m → 300m.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;14:00:04&lt;/td&gt;
&lt;td&gt;Pod Ready; Guaranteed at 14:00:12.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;14:00:24&lt;/td&gt;
&lt;td&gt;Load test attempt 1: the Job pod's image can't be pulled.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;14:07:53&lt;/td&gt;
&lt;td&gt;Attempt 2: password read as an &lt;code&gt;mc&lt;/code&gt; flag.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;14:24:07&lt;/td&gt;
&lt;td&gt;Attempt 3: &lt;code&gt;mc&lt;/code&gt; writes to local paths; MinIO not reached.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;14:34:55&lt;/td&gt;
&lt;td&gt;Attempt 4 starts: 32, then 64, then 128 concurrent.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;14:37:35&lt;/td&gt;
&lt;td&gt;Peak anon 677.8 MiB; &lt;code&gt;memory.current&lt;/code&gt; 767.6 MiB at 14:37:39.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;14:38:35&lt;/td&gt;
&lt;td&gt;Load test ends; bucket removed; 0 restarts.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;14:39:43&lt;/td&gt;
&lt;td&gt;anon 618 MiB after the load; &lt;code&gt;memory.events&lt;/code&gt; max 72, &lt;code&gt;oom_kill&lt;/code&gt; 0.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;14:40:19&lt;/td&gt;
&lt;td&gt;kubegap rerun: &lt;code&gt;memory-limit-near-p95&lt;/code&gt; passes; CPU throttling warning from the test just before.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;14:45:52&lt;/td&gt;
&lt;td&gt;Same ratio from Prometheus: 0.78.&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  FAQ
&lt;/h2&gt;

&lt;h3&gt;
  
  
  minio pod OOMKilled
&lt;/h3&gt;

&lt;p&gt;Check first whether it was killed at all. &lt;code&gt;kube_pod_container_status_last_terminated_reason&lt;/code&gt; should read &lt;code&gt;OOMKilled&lt;/code&gt;, and &lt;code&gt;increase(container_oom_events_total[30d])&lt;/code&gt; should be above 0; &lt;code&gt;memory.events&lt;/code&gt; belongs to the running container's cgroup, so after a kill and restart it starts again at 0. Ours read &lt;code&gt;Unknown&lt;/code&gt; and 0 at 95 % of the limit. If it was killed, remember MinIO sizes itself from 90 % of &lt;code&gt;memory.max&lt;/code&gt;, so 10 % is left for page cache that cannot be reclaimed, slab and kernel memory.&lt;/p&gt;

&lt;h3&gt;
  
  
  working set near memory limit but no OOM
&lt;/h3&gt;

&lt;p&gt;The working set is usage minus &lt;code&gt;inactive_file&lt;/code&gt;, so active page cache counts in it, and the kernel can still reclaim that cache. The kernel kills only when it cannot reclaim enough. Our MinIO sat at 95 % working set for 30 days with &lt;code&gt;oom_kill&lt;/code&gt; 0, while the daily maximum of anon, the memory the process allocated, stayed at 434–440 MiB.&lt;/p&gt;

&lt;h3&gt;
  
  
  minio 512Mi memory limit
&lt;/h3&gt;

&lt;p&gt;At 512Mi MinIO gives itself 460.8 MiB: a Go soft limit, 152 concurrent requests on one drive, 230 two-MiB buffers. Ours ran 30 days unkilled with 51.2 MiB margin. At 768Mi, 64 concurrent 8 MiB uploads pushed anon to 618 MiB; that load was not run at 512Mi. A MinIO maintainer's answer to a 512Mi pod in &lt;a href="https://github.com/minio/minio/issues/20156" rel="noopener noreferrer"&gt;#20156&lt;/a&gt; was "set &lt;code&gt;--memlimit &amp;lt;pod_limit&amp;gt;&lt;/code&gt; for memory, such tiny memory is not what we optimize for"; at this tag a &lt;code&gt;--memlimit&lt;/code&gt; above 90 % of &lt;code&gt;memory.max&lt;/code&gt; is ignored, so only a lower value changes anything.&lt;/p&gt;

&lt;h3&gt;
  
  
  MINIO_API_REQUESTS_MAX
&lt;/h3&gt;

&lt;p&gt;It overrides MinIO's concurrent request pool, by default 90 % of the memory limit divided by drives × 1.03 MiB + 2 MiB: 152 at 512Mi, 228 at 768Mi on one drive. It is divided by the node count; requests over it get HTTP 429. The log line "Configured max API requests per node based on available memory" is printed only in distributed mode, so a single-drive MinIO never logs it; read the &lt;code&gt;X-RateLimit-Limit&lt;/code&gt; response header instead (step 5, not run on our cluster).&lt;/p&gt;

&lt;h3&gt;
  
  
  MINIO_MEMLIMIT
&lt;/h3&gt;

&lt;p&gt;It is the environment form of the hidden &lt;code&gt;--memlimit&lt;/code&gt; flag (help text: "set global memory limit per server via GOMEMLIMIT"), and it can only lower MinIO's budget. A value above 90 % of &lt;code&gt;memory.max&lt;/code&gt; logs "ignoring --memlimit value" and is capped. A lower value shrinks the request pool and the Go soft limit, but not the erasure byte pool, which reads the cgroup limit directly. We have not set it.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://benchlet.com/guides/minio-memory-usage-high-kubernetes-working-set-near-limit-no-oomkilled/" rel="noopener noreferrer"&gt;benchlet.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>kubernetes</category>
      <category>memory</category>
      <category>minio</category>
      <category>sre</category>
    </item>
    <item>
      <title>EKS "failed to assign an IP address to container" and awscni_no_available_ip_addresses: start-up race or IP exhaustion?</title>
      <dc:creator>KYUNGHWAN KIM</dc:creator>
      <pubDate>Sun, 15 Nov 2026 15:17:15 +0000</pubDate>
      <link>https://dev.to/kkhdevs/eks-failed-to-assign-an-ip-address-to-container-and-awscninoavailableipaddresses-start-up-pl1</link>
      <guid>https://dev.to/kkhdevs/eks-failed-to-assign-an-ip-address-to-container-and-awscninoavailableipaddresses-start-up-pl1</guid>
      <description>&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fcbkzr4g7m4a9s5865g9k.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fcbkzr4g7m4a9s5865g9k.png" alt="The failed IP assignment on node-a at 12:32:40.926 UTC on a zoomed time axis: the line &amp;quot;DataStore has no available IP/Prefix addresses&amp;quot;, eni-2's three IPs added to the datastore 10 ms later at 12:32:40.936, and the next request given an IP at 12:32:41.406."&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The one failure behind the counter, on 2026-11-07: 10 ms before the second ENI's IPs reached the datastore, 0.48 s before the next request got an IP.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;If a pod on your EKS node got a &lt;code&gt;FailedCreatePodSandBox&lt;/code&gt; event with &lt;code&gt;add cmd: failed to assign an IP address to container&lt;/code&gt;, ipamd logged &lt;code&gt;DataStore has no available IP/Prefix addresses&lt;/code&gt;, and the metric &lt;code&gt;awscni_no_available_ip_addresses&lt;/code&gt; is above zero, the node is not necessarily out of pod IPs, and the pod is not necessarily still stuck in ContainerCreating. On ours the counter stayed at 1 for eight days because of one failed request in the node's first seconds. With &lt;code&gt;WARM_ENI_TARGET=1&lt;/code&gt; the second ENI is attached only after the first pod takes an IP, and the fourth pod that needed an IP (host-network pods such as aws-node and kube-proxy take none) arrived 10 ms before that ENI's IPs were usable. The next request got an IP 0.48 s later. Below: the node log of that sequence, the VPC CNI source lines behind it, and five read-only checks with the values that separate the race from a real shortage. The cluster is a 9-node EKS cluster (Kubernetes 1.32, VPC CNI add-on v1.19.0-eksbuild.1 with no configuration values); the node is &lt;code&gt;node-a&lt;/code&gt;, a t8i.small; all times are UTC.&lt;/p&gt;

&lt;h2&gt;
  
  
  What we saw
&lt;/h2&gt;

&lt;p&gt;A full run of kubegap v0.3.0, an open-source audit of Kubernetes cluster operations that I maintain, on 2026-11-14 at 16:51 listed this among the problems after its top five (translated): "A system node that could not hand out a pod IP. On node-a (t8i.small) ipamd answered 'no available IP' once. IP use is now 6 of 9 and max-pods is 11, so pods can get stuck in ContainerCreating when many arrive at once. Severity high." The finding behind it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;vpccni.no-available-ip | target node-a | fail | high
  node node-a (t8i.small, pool system): ipamd answered 'no available IP' 1 times since it started;
  pod IPs 6/9 (0.667), ENI 3/3, max-pods 11
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The rule fired when the counter was above 0. Its explanation said any non-zero value "means a pod on this node has already stalled in ContainerCreating".&lt;/p&gt;

&lt;p&gt;The counter on the node on 2026-11-15 at 13:46:23, read with &lt;code&gt;curl 127.0.0.1:61678/metrics&lt;/code&gt; from inside the node (the apiserver pod proxy to port 61678 did not answer within 2 minutes because of the node security group):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight prometheus"&gt;&lt;code&gt;&lt;span class="n"&gt;awscni_no_available_ip_addresses&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
&lt;span class="n"&gt;awscni_assigned_ip_addresses&lt;/span&gt; &lt;span class="mi"&gt;6&lt;/span&gt;
&lt;span class="n"&gt;awscni_total_ip_addresses&lt;/span&gt; &lt;span class="mi"&gt;9&lt;/span&gt;
&lt;span class="n"&gt;awscni_eni_allocated&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt;
&lt;span class="n"&gt;awscni_eni_max&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt;
&lt;span class="n"&gt;awscni_total_ipv4_prefixes&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
&lt;span class="n"&gt;awscni_add_ip_req_count&lt;/span&gt; &lt;span class="mi"&gt;16&lt;/span&gt;
&lt;span class="n"&gt;awscni_force_removed_ips&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It had also been 1 in the snapshot of the kubegap run. kubegap read the metrics of 7 of the 9 nodes; the other 6 it read had 0.&lt;/p&gt;

&lt;p&gt;Nothing else showed a problem:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;kubectl get events -A --field-selector reason=FailedCreatePodSandBox&lt;/code&gt; at 13:45:20 returned nothing. Events are kept for about an hour, so this says nothing about the day the counter moved.&lt;/li&gt;
&lt;li&gt;The cluster's Prometheus does not scrape &lt;code&gt;awscni_*&lt;/code&gt;: &lt;code&gt;count(awscni_no_available_ip_addresses)&lt;/code&gt; returned no series.&lt;/li&gt;
&lt;li&gt;The ipamd log of the last 24 hours, warn and above, had no line.&lt;/li&gt;
&lt;li&gt;At 13:42:58 node-a ran 11 pods: 9 running, 2 completed CI jobs. Of the 9 running, 3 use host networking (aws-node, kube-proxy, node-exporter) and 6 have a pod IP (argocd-redis, ebs-csi-node, kafka-exporter, grafana, promtail and an app's PostgreSQL).&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The node was created on 2026-11-07 at 12:32:25 by Karpenter (on-demand, &lt;code&gt;system&lt;/code&gt; NodePool). The only "failed to assign" line in its &lt;code&gt;plugin.log&lt;/code&gt; is from that minute:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;2026-11-07T12:32:40.927Z error routed-eni-cni-plugin/cni.go:128       Failed to assign an IP address to container &amp;lt;sandbox 4&amp;gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Why &lt;code&gt;failed to assign an IP address to container&lt;/code&gt;: the fourth pod came before the second ENI's IPs
&lt;/h2&gt;

&lt;h3&gt;
  
  
  The node's capacity: 3 IPs per ENI
&lt;/h3&gt;

&lt;p&gt;A t8i.small has 3 network interfaces with 4 IPv4 addresses each (EC2 &lt;code&gt;DescribeInstanceTypes&lt;/code&gt;). Each ENI keeps one address for itself, so pods get 3 × (4 − 1) = 9 IPs, 3 per ENI. AWS's max-pods formula adds 2 for host-network pods: 9 + 2 = 11, the allocatable pods of node-a. The VPC CNI on this cluster runs with the defaults: &lt;code&gt;WARM_ENI_TARGET=1&lt;/code&gt;, &lt;code&gt;WARM_PREFIX_TARGET=1&lt;/code&gt;, &lt;code&gt;ENABLE_PREFIX_DELEGATION=false&lt;/code&gt;, no &lt;code&gt;WARM_IP_TARGET&lt;/code&gt;, no &lt;code&gt;MINIMUM_IP_TARGET&lt;/code&gt;, ipamd log at DEBUG in &lt;code&gt;/host/var/log/aws-routed-eni/ipamd.log&lt;/code&gt; inside the aws-node container, which is &lt;code&gt;/var/log/aws-routed-eni/ipamd.log&lt;/code&gt; on the node.&lt;/p&gt;

&lt;h3&gt;
  
  
  &lt;code&gt;DataStore has no available IP/Prefix addresses&lt;/code&gt;: the first 24 seconds of ipamd's log
&lt;/h3&gt;

&lt;p&gt;The rotated ipamd log on node-a still held the node's start. The file is JSON lines (&lt;code&gt;{"level":"info","ts":"…","caller":"…","msg":"…"}&lt;/code&gt;); the excerpts below show time, level, caller and message. The caller field does not match the v1.19.0 source lines cited in the next section (the error shows &lt;code&gt;data_store.go:607&lt;/code&gt;, the source has it at &lt;code&gt;data_store.go:734&lt;/code&gt;; the pool manager line shows &lt;code&gt;runtime/asm_amd64.s&lt;/code&gt;), so grep for the message text, not the caller. ipamd started 6 s after the node object was created, filled the primary ENI with 3 secondary IPs and logged its pool settings:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;2026-11-07T12:32:31.588Z info  aws-k8s-agent/main.go:42               Starting L-IPAMD   ...
2026-11-07T12:32:32.853Z info  ipamd/ipamd.go:931                     Trying to allocate 3 IP addresses on ENI eni-1
2026-11-07T12:32:33.297Z info  ipamd/ipamd.go:1121                    Added ENI(eni-1)'s IP/Prefix 192.0.2.11/32 to datastore
2026-11-07T12:32:33.297Z info  ipamd/ipamd.go:1121                    Added ENI(eni-1)'s IP/Prefix 192.0.2.12/32 to datastore
2026-11-07T12:32:33.297Z info  ipamd/ipamd.go:1121                    Added ENI(eni-1)'s IP/Prefix 192.0.2.13/32 to datastore
2026-11-07T12:32:33.297Z info  runtime/asm_amd64.s:1695               IP pool manager - max pods: 11, warm IP target: 0, warm prefix target: 1, warm ENI target: 1, minimum IP target: 0
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two pods took two of the three IPs. 1.7 s after the first one, ipamd began creating a second ENI:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;2026-11-07T12:32:34.052Z info  datastore/data_store.go:714            assignPodIPAddressUnsafe: Assign IP 192.0.2.11 to sandbox aws-cni/&amp;lt;sandbox 1&amp;gt;/eth0
2026-11-07T12:32:34.203Z info  datastore/data_store.go:714            assignPodIPAddressUnsafe: Assign IP 192.0.2.12 to sandbox aws-cni/&amp;lt;sandbox 2&amp;gt;/eth0
2026-11-07T12:32:35.798Z info  awsutils/awsutils.go:795               Trying to allocate 3 IP addresses on new ENI
2026-11-07T12:32:36.504Z info  awsutils/awsutils.go:941               Created a new ENI: eni-2
2026-11-07T12:32:37.800Z info  ipamd/ipamd.go:865                     Successfully created and attached a new ENI eni-2 to instance
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The third pod took the last IP of eni-1. The fourth request found none; 10 ms later eni-2's IPs were in the datastore:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;2026-11-07T12:32:40.738Z info  datastore/data_store.go:714            assignPodIPAddressUnsafe: Assign IP 192.0.2.13 to sandbox aws-cni/&amp;lt;sandbox 3&amp;gt;/eth0
2026-11-07T12:32:40.926Z error datastore/data_store.go:607            DataStore has no available IP/Prefix addresses
2026-11-07T12:32:40.927Z info  rpc/rpc.pb.go:863                      Send AddNetworkReply: IPv4Addr: , IPv6Addr: , DeviceNumber: -1, err: AssignPodIPv4Address: no available IP/Prefix addresses
2026-11-07T12:32:40.936Z info  ipamd/ipamd.go:886                     Found ENIs having 4 secondary IPs and 0 Prefixes
2026-11-07T12:32:40.936Z info  ipamd/ipamd.go:1121                    Added ENI(eni-2)'s IP/Prefix 192.0.2.15/32 to datastore
2026-11-07T12:32:40.936Z info  ipamd/ipamd.go:1121                    Added ENI(eni-2)'s IP/Prefix 192.0.2.16/32 to datastore
2026-11-07T12:32:40.936Z info  ipamd/ipamd.go:1121                    Added ENI(eni-2)'s IP/Prefix 192.0.2.17/32 to datastore
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The failed sandbox was torn down (&lt;code&gt;datastore: unknown pod&lt;/code&gt; at 12:32:40.947), a new one was created at 12:32:41.405, and it got the first IP of eni-2 at 12:32:41.406, 0.479 s after the failure:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;2026-11-07T12:32:41.406Z info  datastore/data_store.go:714            assignPodIPAddressUnsafe: Assign IP 192.0.2.15 to sandbox aws-cni/&amp;lt;sandbox 5&amp;gt;/eth0
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;From 12:32:45.937 to 12:32:51.031 a third ENI, eni-3, was created, attached and its 3 IPs added. From 12:32:51 the node had 9 IPs, and at 12:33:24 the last pod of the start-up got an IP.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fuxry0c7nbi11s8joi2gv.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fuxry0c7nbi11s8joi2gv.png" alt="Measured timeline of node-a from ipamd start, seconds 0 to 24, one lane per ENI and one lane of pod IP requests: eni-1's 3 IPs at +1.7 s, pods 1 to 3 taking them, eni-2 created and attached from +4.2 s to +6.2 s, the failed fourth request at +9.3 s, eni-2's IPs added 10 ms later, the next request served at +9.8 s, eni-3's IPs at +19.4 s, the sixth request at +22.6 s."&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;node-a on 2026-11-07, seconds since &lt;code&gt;Starting L-IPAMD&lt;/code&gt; at 12:32:31.588 UTC. The gap is between the attach of eni-2 and its IPs reaching the datastore.&lt;/em&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  Why the second ENI was being attached only then
&lt;/h3&gt;

&lt;p&gt;Each step below is in the amazon-vpc-cni-k8s source at tag v1.19.0:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;At start, &lt;code&gt;nodeInit&lt;/code&gt; calls &lt;code&gt;increaseDatastorePool&lt;/code&gt; once. One call either fills an existing ENI or attaches a new one, never both, so a new node starts with the primary ENI's 3 secondary IPs and no second ENI (&lt;code&gt;ipamd.go:565–567&lt;/code&gt;, :810–833).&lt;/li&gt;
&lt;li&gt;With &lt;code&gt;WARM_ENI_TARGET=1&lt;/code&gt; and 3 IPs per ENI, the pool counts as too low when fewer than 3 IPs are free (&lt;code&gt;ipamd.go:2178–2179&lt;/code&gt;). With 3 free IPs at start, 3 &amp;lt; 3 is false. The pool becomes low when the first pod takes an IP.&lt;/li&gt;
&lt;li&gt;The pool manager checks every 5 s (&lt;code&gt;ipamd.go:58&lt;/code&gt;, :656–665) and attaches one ENI per step. Our log matches: first pod at 12:32:34.052, new ENI started at 12:32:35.798.&lt;/li&gt;
&lt;li&gt;After the attach, ipamd waits until the ENI and its secondary IPs appear in instance metadata (&lt;code&gt;waitForENIAndIPsAttached&lt;/code&gt;, &lt;code&gt;awsutils.go:1742&lt;/code&gt;), sets up the ENI's network and only then adds the IPs to the datastore (&lt;code&gt;ipamd.go:844–893&lt;/code&gt;). On node-a the IPs reached the datastore 3.1 s after the attach (12:32:37.800 → 12:32:40.936). The node's own latency metric for &lt;code&gt;waitForENIAndIPsAttached&lt;/code&gt; totals 140 ms for both ENIs, so most of the 3.1 s came after the metadata wait; we did not trace which step took it.&lt;/li&gt;
&lt;li&gt;The gRPC handler serves each CNI ADD from the datastore without waiting for the pool manager (&lt;code&gt;rpc_handler.go:174&lt;/code&gt;). &lt;code&gt;AssignPodIPv4Address&lt;/code&gt; walks every ENI, finds no free IP, increments the counter, logs &lt;code&gt;DataStore has no available IP/Prefix addresses&lt;/code&gt; and returns the error (&lt;code&gt;data_store.go:680–735&lt;/code&gt;). The plugin then returns &lt;code&gt;add cmd: failed to assign an IP address to container&lt;/code&gt; to the runtime (&lt;code&gt;cni.go:185&lt;/code&gt;), and kubelet records &lt;code&gt;FailedCreatePodSandBox&lt;/code&gt; and retries.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;AWS's best-practices page describes the same order: "The IPAMD allocates additional ENI when a single Pod is scheduled and assigned a secondary IP address from the primary ENI." The repository's &lt;code&gt;docs/eni-and-ip-target.md&lt;/code&gt; says "Creating and attaching a new ENI to a node can take up to 10 seconds." In issue #2669, titled "DataStore has no available IP/Prefix addresses", a maintainer wrote that the log line is "not out of the ordinary if the node is scaling up and attaching new ENIs"; in issue #3255 a maintainer confirmed that a temporary &lt;code&gt;FailedCreatePodSandBox&lt;/code&gt; is expected while the pool is momentarily empty.&lt;/p&gt;

&lt;p&gt;So on a node with 3 IPs per ENI that receives more than 3 IP-using pods in its first seconds, one request can find the pool empty. Our conclusion (an inference from the log and the source): the single count is the fourth IP request on the node, made 9.3 s after ipamd started while the second ENI was 10 ms from usable, and it did not recur.&lt;/p&gt;

&lt;p&gt;We did not see the pod event for it: events had expired long before we looked. The event text above is what the plugin source returns, not something we captured.&lt;/p&gt;

&lt;h3&gt;
  
  
  What &lt;code&gt;awscni_no_available_ip_addresses&lt;/code&gt; counts
&lt;/h3&gt;

&lt;p&gt;&lt;code&gt;awscni_no_available_ip_addresses&lt;/code&gt; is a Prometheus counter that ipamd increments in &lt;code&gt;AssignPodIPv4Address&lt;/code&gt; just before it logs the error (&lt;code&gt;data_store.go:733–735&lt;/code&gt;, &lt;code&gt;prometheusmetrics.go:162–167&lt;/code&gt;). It is reset only when the aws-node container restarts, and it counts failed assignment calls, not pods: a pod whose IP request fails on two sandbox attempts adds 2. A value of 1 is history since aws-node started, not the current state. On node-a it was 1 at the kubegap snapshot, 172.3 hours after ipamd started, and still 1 the next day. A maintainer in issue #2649 lists five reasons the counter moves, among them "CNI waiting for new ENIs to be attached", and adds "we do not really delineate these cases today".&lt;/p&gt;

&lt;h2&gt;
  
  
  What to check in your cluster now
&lt;/h2&gt;

&lt;p&gt;All steps are read-only. Steps 3 and 4 run on the node (SSH or Session Manager), because the apiserver pod proxy to port 61678 usually times out when the node security group does not allow it; step 4 also needs root (&lt;code&gt;sudo&lt;/code&gt;; AWS-RunShellScript already runs as root), because the files in &lt;code&gt;/var/log/aws-routed-eni/&lt;/code&gt; are mode 0600, owned by root. Replace &lt;code&gt;&amp;lt;node&amp;gt;&lt;/code&gt;, &lt;code&gt;&amp;lt;type&amp;gt;&lt;/code&gt; and, in step 4, &lt;code&gt;&amp;lt;YYYY-MM-DDTHH:MM&amp;gt;&lt;/code&gt; (the minute of a failure from the first grep) with yours.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;1. How many pod IPs can the node hold, and how old is it?&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl get node &lt;span class="o"&gt;&amp;lt;&lt;/span&gt;node&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; -o &lt;span class="nv"&gt;jsonpath&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;'{.metadata.labels.node\.kubernetes\.io/instance-type}{" created="}{.metadata.creationTimestamp}{" max-pods="}{.status.allocatable.pods}{"\n"}'&lt;/span&gt;
aws ec2 describe-instance-types --instance-types &lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nb"&gt;type&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; --query &lt;span class="s1"&gt;'InstanceTypes[].[MaximumNetworkInterfaces,NetworkInfo.Ipv4AddressesPerInterface]'&lt;/span&gt; --output text
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Pod IPs = ENIs × (IPv4 per ENI − 1). Ours: t8i.small, 3 × (4 − 1) = 9, max-pods 11, created 8 days earlier. The smaller the IPs-per-ENI number, the fewer pods it takes to empty the first ENI.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Which warm settings does the VPC CNI run with?&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl -n kube-system get ds aws-node -o &lt;span class="nv"&gt;jsonpath&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;'{range .spec.template.spec.containers[?(@.name=="aws-node")].env[*]}{.name}={.value}{"\n"}{end}'&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; grep -E &lt;span class="s1"&gt;'WARM_|MINIMUM_IP|PREFIX_DELEGATION|LOG_FILE|LOGLEVEL'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Same setup as ours: &lt;code&gt;WARM_ENI_TARGET=1&lt;/code&gt;, &lt;code&gt;ENABLE_PREFIX_DELEGATION=false&lt;/code&gt;, no &lt;code&gt;WARM_IP_TARGET&lt;/code&gt; or &lt;code&gt;MINIMUM_IP_TARGET&lt;/code&gt;; &lt;code&gt;AWS_VPC_K8S_CNI_LOG_FILE&lt;/code&gt; tells step 4 where the ipamd log is (ours &lt;code&gt;/host/var/log/aws-routed-eni/ipamd.log&lt;/code&gt;, read on the node without &lt;code&gt;/host&lt;/code&gt;), and &lt;code&gt;AWS_VPC_K8S_CNI_LOGLEVEL=DEBUG&lt;/code&gt; makes it rotate within days. With these, the second ENI is attached only after the first pod takes an IP from the first one, so on a node with 3 IPs per ENI the fourth IP-using pod of the first seconds can find the pool empty.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. What do the counter and the pool say now? (on the node)&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl -s http://127.0.0.1:61678/metrics &lt;span class="o"&gt;|&lt;/span&gt; grep -E &lt;span class="s1"&gt;'^awscni_(no_available_ip_addresses|assigned_ip_addresses|total_ip_addresses|eni_allocated|eni_max) '&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Same problem as ours (history, not a shortage) when the counter is small and does not grow when you read it again later, &lt;code&gt;assigned&lt;/code&gt; is below &lt;code&gt;total&lt;/code&gt;, and &lt;code&gt;eni_allocated&lt;/code&gt; is below &lt;code&gt;eni_max&lt;/code&gt; or &lt;code&gt;total − assigned&lt;/code&gt; covers the pods still to come. Ours: counter 1 (and 1 about 21 hours earlier, in the kubegap snapshot), assigned 6 of 9, ENI 3 of 3.&lt;/p&gt;

&lt;p&gt;A real shortage looks different: the counter keeps rising, &lt;code&gt;assigned&lt;/code&gt; equals &lt;code&gt;total&lt;/code&gt;, and &lt;code&gt;kubectl get events -A --field-selector reason=FailedCreatePodSandBox&lt;/code&gt; shows &lt;code&gt;failed to assign an IP address to container&lt;/code&gt; for pods that stay in ContainerCreating; &lt;code&gt;eni_allocated&lt;/code&gt; then equals &lt;code&gt;eni_max&lt;/code&gt; when the node is full, or stays below it when the subnet has no free address (check the subnet's &lt;code&gt;AvailableIpAddressCount&lt;/code&gt;).&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. When did each failure happen? (on the node)&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;grep -h &lt;span class="s1"&gt;'Failed to assign an IP address'&lt;/span&gt; /var/log/aws-routed-eni/plugin.log*
&lt;span class="k"&gt;for&lt;/span&gt; f &lt;span class="k"&gt;in&lt;/span&gt; /var/log/aws-routed-eni/ipamd*.log&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt; grep -h &lt;span class="s1"&gt;'no available IP/Prefix'&lt;/span&gt; &lt;span class="s2"&gt;"$f"&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;done&lt;/span&gt;
zcat /var/log/aws-routed-eni/ipamd-*.log.gz 2&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;/dev/null &lt;span class="o"&gt;|&lt;/span&gt; grep -h &lt;span class="s1"&gt;'no available IP/Prefix'&lt;/span&gt;
zcat -f /var/log/aws-routed-eni/ipamd-*.log.gz /var/log/aws-routed-eni/ipamd.log &lt;span class="o"&gt;|&lt;/span&gt; grep &lt;span class="s1"&gt;'Starting L-IPAMD'&lt;/span&gt;   &lt;span class="c"&gt;# every start; the counter counts from the last one&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then read the minute around each failure for &lt;code&gt;Added ENI(…)'s IP/Prefix … to datastore&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;zcat -f /var/log/aws-routed-eni/ipamd-*.log.gz /var/log/aws-routed-eni/ipamd.log &lt;span class="o"&gt;|&lt;/span&gt; grep &lt;span class="s1"&gt;'"ts":"&amp;lt;YYYY-MM-DDTHH:MM&amp;gt;'&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; grep -v &lt;span class="s1"&gt;'"level":"debug"'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Start-up race when the failure is within seconds of &lt;code&gt;Starting L-IPAMD&lt;/code&gt; (or of the node joining), the IPs of a newly attached ENI are added to the datastore milliseconds to seconds after it, and the next CNI ADD gets an IP. Ours: ipamd started 12:32:31.588, failure 12:32:40.926, eni-2's IPs added 12:32:40.936, the next ADD got an IP at 12:32:41.406. The log rotates by size; node-a logs at DEBUG and its log rotated on 2026-11-14 at 00:15, 6.5 days after the node started, so when we read it on 11-15 the start was only in &lt;code&gt;ipamd-2026-11-14T00-15-03.555.log.gz&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;5. Can this node ever run out in steady state?&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl get pods -A --field-selector spec.nodeName=&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;node&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;,status.phase=Running -o json &lt;span class="o"&gt;|&lt;/span&gt; jq &lt;span class="s1"&gt;'[.items[] | select(.spec.hostNetwork == true)] | length'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;max-pods minus host-network pods is the most pods that can ever ask for an IP. Ours: 11 − 3 = 8 against 9 IPs, so the node cannot run out once its three ENIs are attached, apart from IPs held in the 30-second cooldown after a pod is deleted.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fobvtxiypy54a39hpx40k.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fobvtxiypy54a39hpx40k.png" alt="Two columns, start-up race and real shortage, comparing five signals: the counter (small and flat vs rising), assigned vs total IPs, allocated vs maximum ENIs, the time of the failure relative to ipamd start, and max-pods minus host-network pods against pod IPs. node-a's values are marked: counter 1, 6 of 9 IPs, 3 of 3 ENIs, failure 9.3 s after ipamd start, 8 against 9."&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The readings from steps 3 to 5 that separate the two cases, with node-a's values: counter and IPs read on 2026-11-15, failure timing from its 2026-11-07 log.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;If you do not want to repeat step 3, step 5 and the last-24-hours part of step 4 on every node by hand, &lt;a href="https://github.com/KyungHwanKim-devs/kubegap" rel="noopener noreferrer"&gt;kubegap&lt;/a&gt; does it: it reads the counter, the ipamd log of the last 24 hours and the aws-node start time from each node through a read-only SSM document, fails only on a "no available IP" line more than 120 s after an ipamd start, and reports a count older than that log as &lt;code&gt;vpccni.no-available-ip-unexplained&lt;/code&gt; with the max-pods check of step 5. As of 2026-11-15 no release has it: v0.3.0 (released 2026-11-14), whose report led to this article, fails on any non-zero counter, so install from &lt;code&gt;main&lt;/code&gt; at commit 0adb216 or later (pull request #210, rubric 0.1.96).&lt;/p&gt;

&lt;h2&gt;
  
  
  What we changed, and what we have not
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Nothing on the cluster
&lt;/h3&gt;

&lt;p&gt;No VPC CNI setting was changed. The evidence shows no shortage: one failed request 9.3 s after ipamd started, the next request served 0.48 s later, and no failure in the 8 days after. A VPC CNI setting applies to every node in the cluster, and I would not change all nine nodes to remove one retried sandbox on one of them.&lt;/p&gt;

&lt;h3&gt;
  
  
  The audit rule
&lt;/h3&gt;

&lt;p&gt;The rule that raised it was wrong for this case, so we fixed the rule (kubegap issue #209, opened 2026-11-15 14:04:26; pull request #210, merged 14:16:16 as commit 0adb216). It uses only what kubegap already collects; the node-side SSM document is unchanged.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;vpccni.no-available-ip&lt;/code&gt; (high) now fails only on a "no available IP" line in the last 24 hours of ipamd log that comes more than 120 s after an ipamd start. Lines within 120 s of a start are the start-up race and pass.&lt;/li&gt;
&lt;li&gt;A counter value that the 24-hour log cannot explain goes to a new check, &lt;code&gt;vpccni.no-available-ip-unexplained&lt;/code&gt; (warn, low). It says when ipamd started and whether max-pods minus host-network pods fits the IP capacity.&lt;/li&gt;
&lt;li&gt;Why 120 s: the race came 9.3 s after ipamd started on node-a. The pool manager runs every 5 s and waits for a new ENI's IPs in instance metadata with a backoff that starts at 100 ms, doubles and adds up to 15 % jitter; through the tenth try the sleeps add up to at most 58.8 s, so 120 s covers one tick and that wait with room for slow EC2 calls.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Re-evaluated on the same snapshot with the new code, without collecting anything again (14:16:29), condensed from the before/after comparison:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;vpccni.no-available-ip             node-a   fail/high  -&amp;gt;  pass
vpccni.no-available-ip-unexplained node-a   (new)      -&amp;gt;  warn/low
  ipamd answered 'no available IP' 1 times since it started at 2026-11-07T12:32:31Z
  (172.3 h before the snapshot) ... Pods that can need an IP:
  max-pods 11 - 3 hostNetwork = 8, against 9 pod IPs: fits
vpccni.prefix-delegation-off       cluster  fail       -&amp;gt;  warn
vpccni.node-ip-exhaustion          node-a   warn/high  (unchanged: 6/9 IPs in use, ENI 3/3)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fklky4tiskius0tsebodn.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fklky4tiskius0tsebodn.png" alt="The old and new  raw `vpccni.no-available-ip` endraw  rule as decision paths. Old: counter above 0, fail high. New: a &amp;quot;no available IP&amp;quot; line in the last 24 h of ipamd log more than 120 s after an ipamd start, fail high; within 120 s of a start, pass; a count older than the log, warn low with the max-pods fit. node-a's path is highlighted: count older than the log, 8 against 9, warn low."&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The kubegap rule before and after pull request #210, with node-a's path on the 2026-11-14 snapshot.&lt;/em&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  Not done
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Prefix delegation.&lt;/strong&gt; From the source, it would remove the first-seconds window: the primary ENI gets a /28 at &lt;code&gt;nodeInit&lt;/code&gt;. It would also need a max-pods change on this node type to use the extra addresses. Not tested.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;MINIMUM_IP_TARGET&lt;/code&gt;.&lt;/strong&gt; From the source, it would not remove the window by itself: &lt;code&gt;nodeInit&lt;/code&gt; still does one pool step, and further ENIs come from the 5-second loop. AWS describes it as a way "to front-load the assignment of multiple ENIs at instance launch"; we have not tested what it does on a t8i.small in its first seconds.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A different instance type for the system pool.&lt;/strong&gt; Not done, for the same reason as the settings: there is no shortage to fix.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;An alert on the counter.&lt;/strong&gt; The cluster's Prometheus does not scrape &lt;code&gt;awscni_*&lt;/code&gt; today. If we add it, I would alert on &lt;code&gt;increase(awscni_no_available_ip_addresses[10m]) &amp;gt; 0&lt;/code&gt; together with free IPs (&lt;code&gt;awscni_total_ip_addresses - awscni_assigned_ip_addresses&lt;/code&gt;) near zero, not on the raw value being above 0.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Where else this happened
&lt;/h2&gt;

&lt;p&gt;The Incident Library, our searchable collection of public post-incident reports on benchlet.com, has no case of a one-off failure on a new node or a false IP-exhaustion alarm. The cases under the same symptom and component are real exhaustion or real allocation outages, which is what a lasting failure looks like:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Neon, 2025-05-16: on VPC CNI v1.18.6 with the defaults (&lt;code&gt;WARM_ENI_TARGET=1&lt;/code&gt;, &lt;code&gt;WARM_IP_TARGET&lt;/code&gt; unset), nodes full on CPU and memory kept about 3.4k IPs no pod could use, the subnets ran out, and setting &lt;code&gt;WARM_IP_TARGET=1&lt;/code&gt; then failed pod starts for 30 s after each pod deletion.&lt;/li&gt;
&lt;li&gt;UK Ministry of Justice, 2023-07-21: new pods on six nodes stayed in ContainerCreating because the VPC CNI allocated no IPs; after about three hours the team found the live cluster lacked the prefix delegation setting its test cluster had, and the incident took 4 h 27 min.&lt;/li&gt;
&lt;li&gt;Adevinta, 2024-10-22: an EKS cluster ran out of IPv4 space; the on-call engineer found the aws-cni error about failing to assign an IP address, and the fix was custom networking on a secondary CIDR.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The other two cases under the same filter are CoW Protocol (an unpinned VPC CNI update during a capacity change) and a second UK Ministry of Justice incident (pods in ContainerCreating with the same error during a node drain): &lt;a href="https://benchlet.com/incidentlib/app?component=vpc-cni&amp;amp;symptom=no-available-ip" rel="noopener noreferrer"&gt;No available IP + VPC CNI, 5 cases&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Timeline
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Time (UTC)&lt;/th&gt;
&lt;th&gt;Event&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;2026-11-07 12:32:25&lt;/td&gt;
&lt;td&gt;node-a (t8i.small) created by Karpenter.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;12:32:31.588&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;Starting L-IPAMD&lt;/code&gt;.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;12:32:33.297&lt;/td&gt;
&lt;td&gt;eni-1's 3 secondary IPs in the datastore; pool manager: warm ENI target 1.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;12:32:34.052, 12:32:34.203&lt;/td&gt;
&lt;td&gt;Pods 1 and 2 get IPs from eni-1.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;12:32:35.798&lt;/td&gt;
&lt;td&gt;ipamd starts allocating a new ENI.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;12:32:37.800&lt;/td&gt;
&lt;td&gt;eni-2 created and attached.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;12:32:40.738&lt;/td&gt;
&lt;td&gt;Pod 3 gets the last IP of eni-1.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;12:32:40.926&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;DataStore has no available IP/Prefix addresses&lt;/code&gt;; counter to 1. plugin.log: &lt;code&gt;Failed to assign an IP address to container &amp;lt;sandbox 4&amp;gt;&lt;/code&gt; (12:32:40.927).&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;12:32:40.936&lt;/td&gt;
&lt;td&gt;eni-2's 3 IPs added to the datastore.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;12:32:41.406&lt;/td&gt;
&lt;td&gt;The new sandbox gets the first IP of eni-2.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;12:32:45.937–12:32:51.031&lt;/td&gt;
&lt;td&gt;eni-3 created, attached, IPs added; 9 IPs on the node.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;12:33:24&lt;/td&gt;
&lt;td&gt;Last pod of the start-up gets an IP.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2026-11-14 16:51&lt;/td&gt;
&lt;td&gt;kubegap v0.3.0 full run: &lt;code&gt;vpccni.no-available-ip&lt;/code&gt; fail/high on node-a.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2026-11-15 13:42:47&lt;/td&gt;
&lt;td&gt;9 nodes listed; node-a 8 days old, allocatable pods 11.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;13:43&lt;/td&gt;
&lt;td&gt;apiserver pod proxy to port 61678: no answer in 2 minutes.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;13:45:20&lt;/td&gt;
&lt;td&gt;No &lt;code&gt;FailedCreatePodSandBox&lt;/code&gt; events in the cluster.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;13:46:23&lt;/td&gt;
&lt;td&gt;On the node: counter 1, 6 of 9 IPs assigned, 3 of 3 ENIs.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;13:46:58&lt;/td&gt;
&lt;td&gt;ipamd and plugin logs read; one failure, on 11-07.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;13:47&lt;/td&gt;
&lt;td&gt;Prometheus has no &lt;code&gt;awscni_*&lt;/code&gt; series.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;14:04:26&lt;/td&gt;
&lt;td&gt;kubegap issue #209 opened.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;14:16:16&lt;/td&gt;
&lt;td&gt;Pull request #210 merged (0adb216, rubric 0.1.96).&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;14:16:29&lt;/td&gt;
&lt;td&gt;Same snapshot re-evaluated: node-a &lt;code&gt;vpccni.no-available-ip&lt;/code&gt; pass; &lt;code&gt;vpccni.no-available-ip-unexplained&lt;/code&gt; warn/low.&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  FAQ
&lt;/h2&gt;

&lt;h3&gt;
  
  
  pod stuck ContainerCreating failed to assign an IP address
&lt;/h3&gt;

&lt;p&gt;Check whether it lasts: &lt;code&gt;kubectl -n &amp;lt;namespace&amp;gt; get events --field-selector involvedObject.name=&amp;lt;pod&amp;gt;,reason=FailedCreatePodSandBox&lt;/code&gt; (events are kept about an hour), and the counter on the node. If the pod starts on the next sandbox attempt, the failure came within seconds of ipamd's start (kubegap uses 120 s) and &lt;code&gt;awscni_assigned_ip_addresses&lt;/code&gt; stays below &lt;code&gt;awscni_total_ip_addresses&lt;/code&gt;, it was the start-up race while a new ENI's IPs were not yet in the datastore. If the pod stays in ContainerCreating, assigned equals total and allocated ENIs equal the maximum, the node is out of IPs.&lt;/p&gt;

&lt;h3&gt;
  
  
  awscni_no_available_ip_addresses
&lt;/h3&gt;

&lt;p&gt;A Prometheus counter in ipamd, read on the node at &lt;code&gt;127.0.0.1:61678/metrics&lt;/code&gt;. It counts failed pod IP assignment calls since the aws-node container started and resets only when that container restarts. A non-zero value is history, not state. Alert on &lt;code&gt;increase(awscni_no_available_ip_addresses[10m]) &amp;gt; 0&lt;/code&gt; together with free IPs near zero, not on the value itself.&lt;/p&gt;

&lt;h3&gt;
  
  
  DataStore has no available IP/Prefix addresses
&lt;/h3&gt;

&lt;p&gt;ipamd logs it when a CNI ADD finds no free IP on any ENI in its datastore; the reply then carries &lt;code&gt;AssignPodIPv4Address: no available IP/Prefix addresses&lt;/code&gt;. Look at the lines just after it. If &lt;code&gt;Added ENI(…)'s IP/Prefix … to datastore&lt;/code&gt; for a new ENI follows within milliseconds and the next ADD gets an IP, it was the start-up race.&lt;/p&gt;

&lt;h3&gt;
  
  
  max pods t3.small eks
&lt;/h3&gt;

&lt;p&gt;Max pods is 11. AWS's formula is ENIs × (IPs per ENI − 1) + 2: a t3.small has 3 ENIs with 4 IPv4 each, so 9 pod IPs plus 2 for host-network pods such as aws-node and kube-proxy. The t8i.small has the same 11. At start-up the step that matters is 3 IPs per ENI: the fourth pod that needs an IP (host-network pods take none) needs the second ENI.&lt;/p&gt;

&lt;h3&gt;
  
  
  WARM_ENI_TARGET vs WARM_IP_TARGET vs MINIMUM_IP_TARGET
&lt;/h3&gt;

&lt;p&gt;&lt;code&gt;WARM_ENI_TARGET&lt;/code&gt; (default 1) keeps one ENI's IPs free; &lt;code&gt;WARM_IP_TARGET&lt;/code&gt; keeps a number of free IPs; &lt;code&gt;MINIMUM_IP_TARGET&lt;/code&gt; sets a floor per node. From the v1.19.0 source (inferred), none attaches a second ENI at node start: &lt;code&gt;nodeInit&lt;/code&gt; does one pool step, later ENIs come from the 5-second loop. Prefix delegation gives a /28 at start. Untested.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://benchlet.com/guides/eks-failed-to-assign-ip-address-awscni-no-available-ip-addresses-warm-eni/" rel="noopener noreferrer"&gt;benchlet.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>kubernetes</category>
      <category>aws</category>
      <category>eks</category>
      <category>networking</category>
    </item>
    <item>
      <title>Prometheus data lost after pod restart? How to set kube-prometheus-stack persistent storage (a storageSpec PVC) without losing the emptyDir data</title>
      <dc:creator>KYUNGHWAN KIM</dc:creator>
      <pubDate>Sun, 15 Nov 2026 01:39:19 +0000</pubDate>
      <link>https://dev.to/kkhdevs/prometheus-data-lost-after-pod-restart-how-to-set-kube-prometheus-stack-persistent-storage-a-ba6</link>
      <guid>https://dev.to/kkhdevs/prometheus-data-lost-after-pod-restart-how-to-set-kube-prometheus-stack-persistent-storage-a-ba6</guid>
      <description>&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fugwqs1kldclqmnw858rv.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fugwqs1kldclqmnw858rv.png" alt="Two panels: before, the Prometheus TSDB (22.7 GiB back to 2026-10-14) sits in an emptyDir inside the pod, so deleting the pod deletes it; after, the same data sits in a 50Gi PVC mounted at /prometheus and survives a deliberate pod delete." width="800" height="420"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Where kube-prometheus-stack kept 30 days of metrics before and after the move. Measured on our cluster on 2026-11-15.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;If Prometheus data was lost after a pod restart, or you are setting up kube-prometheus-stack persistent storage, the cause to check first is this: unless &lt;code&gt;prometheus.prometheusSpec.storageSpec&lt;/code&gt; is set, prometheus-operator stores the Prometheus TSDB on an &lt;code&gt;emptyDir&lt;/code&gt;, and an emptyDir is deleted together with the pod. The same cause is behind the operator log line &lt;code&gt;recreating StatefulSet because the update operation wasn't possible&lt;/code&gt; right after you add &lt;code&gt;storageSpec&lt;/code&gt;: if the old pod had an emptyDir, its TSDB was deleted with that pod. Our Prometheus pod had run that way for 349 days, and at the end its emptyDir held 22.7G covering the last 31.5 days of metrics. To check yours: if &lt;code&gt;kubectl get prometheus -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name} storage={.spec.storage}{"\n"}{end}'&lt;/code&gt; prints a line that ends in &lt;code&gt;storage=&lt;/code&gt; with nothing after it, that Prometheus keeps its TSDB on an emptyDir. To add persistent volume to Prometheus without losing data, create the PVC &lt;code&gt;prometheus-&amp;lt;name&amp;gt;-db-prometheus-&amp;lt;name&amp;gt;-0&lt;/code&gt; and copy the TSDB into its &lt;code&gt;prometheus-db/&lt;/code&gt; folder before adding &lt;code&gt;prometheus.prometheusSpec.storageSpec.volumeClaimTemplate&lt;/code&gt;, because the operator deletes the StatefulSet when you add it. In our move, the 30 days of blocks and WAL that we copied were kept, and a deliberate pod delete afterwards lost nothing stored. The samples the old pod wrote after the second copy (00:38:05) were never copied, and Prometheus's scrape of itself has a 2 min 6 s hole around the switch (the cause is our inference from the times, see "What it cost"). The cluster is a 2-node k3s cluster (&lt;code&gt;server-1&lt;/code&gt;, &lt;code&gt;agent-1&lt;/code&gt;) running kube-prometheus-stack 79.5.0 from Argo CD, prometheus-operator v0.86.2 and Prometheus v3.7.3; all times are UTC, 2026-11-15 unless a date is given.&lt;/p&gt;

&lt;h2&gt;
  
  
  What we saw: the Prometheus data volume was an emptyDir (&lt;code&gt;prometheus-kube-prometheus-stack-prometheus-db {}&lt;/code&gt;)
&lt;/h2&gt;

&lt;h3&gt;
  
  
  The finding
&lt;/h3&gt;

&lt;p&gt;A full run of kubegap v0.3.0 at 2026-11-14 16:25:29 (the tool is described in the check section below) listed as its problem 2, translated from Korean: "Prometheus data is on an emptyDir." The reason it gave: the moment the Prometheus pod is deleted and created again, 30 days of metrics are gone. Two findings carried it, both &lt;code&gt;fail&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;prometheus.storage-risk                 storage emptyDir (PVC None), PVC usage None, TSDB 22.3GiB, retention 30d (size cap 0B),
                                        actual oldest sample 31.2 days, failed compactions over 7 days 0, WAL corruptions 1, corruptions at startup 1
observability.stack-storage-emptydir    emptyDir volumes: config-out, prometheus-kube-prometheus-stack-prometheus-db, replicas 1
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  The StatefulSet and the Prometheus object
&lt;/h3&gt;

&lt;p&gt;At 00:31:25 we read the objects ourselves. The StatefulSet has no &lt;code&gt;volumeClaimTemplates&lt;/code&gt; (the first, empty line of the output), and the data volume is an emptyDir (&lt;code&gt;{}&lt;/code&gt;):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ kubectl -n monitoring get sts prometheus-kube-prometheus-stack-prometheus -o jsonpath='{.spec.volumeClaimTemplates}{"\n"}{range .spec.template.spec.volumes[*]}{.name}{" "}{.emptyDir}{.persistentVolumeClaim}{"\n"}{end}'

config
tls-assets
config-out {"medium":"Memory"}
prometheus-kube-prometheus-stack-prometheus-rulefiles-0
web-config
prometheus-kube-prometheus-stack-prometheus-db {}

$ kubectl -n monitoring get prometheus kube-prometheus-stack-prometheus -o jsonpath='retention={.spec.retention} retentionSize={.spec.retentionSize} storage={.spec.storage} replicas={.spec.replicas}'
retention=30d retentionSize= storage= replicas=1
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The container mounted it at &lt;code&gt;/prometheus&lt;/code&gt; with no &lt;code&gt;subPath&lt;/code&gt;. The pod was 349 days old, the &lt;code&gt;RESTARTS&lt;/code&gt; column showed 6 for the pod (it has two containers, &lt;code&gt;2/2&lt;/code&gt;, while step 4 below prints the count of the &lt;code&gt;prometheus&lt;/code&gt; container alone), and the pod itself had never been replaced (pod IP removed, node name masked):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;$ kubectl -n monitoring get pod prometheus-kube-prometheus-stack-prometheus-0 -o wide
NAME                                            READY   STATUS    RESTARTS      AGE    IP           NODE       NOMINATED NODE   READINESS GATES
prometheus-kube-prometheus-stack-prometheus-0   2/2     Running   6 &lt;span class="o"&gt;(&lt;/span&gt;23h ago&lt;span class="o"&gt;)&lt;/span&gt;   349d   &lt;span class="o"&gt;&amp;lt;&lt;/span&gt;pod IP&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;     server-1   &lt;span class="o"&gt;&amp;lt;&lt;/span&gt;none&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;           &lt;span class="o"&gt;&amp;lt;&lt;/span&gt;none&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;

&lt;span class="nv"&gt;created&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;2025-11-30T06:09:35Z
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  How much was in it, and on which disk
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;$ kubectl -n monitoring &lt;span class="nb"&gt;exec&lt;/span&gt; prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- sh -c &lt;span class="s1"&gt;'du -sh /prometheus; df -h /prometheus; ls /prometheus | wc -l'&lt;/span&gt;
22.7G   /prometheus
Filesystem                Size      Used Available Use% Mounted on
&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;node root LVM volume&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;
                        454.9G    249.7G    185.6G  57% /prometheus
23
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The 23 entries are 19 block directories plus &lt;code&gt;chunks_head&lt;/code&gt;, &lt;code&gt;lock&lt;/code&gt;, &lt;code&gt;queries.active&lt;/code&gt; and &lt;code&gt;wal&lt;/code&gt;. &lt;code&gt;df&lt;/code&gt; shows the node's root filesystem, because an emptyDir lives under the kubelet's directory on the node's disk. Prometheus shared that disk with everything else on &lt;code&gt;server-1&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Prometheus's own metrics at 00:31:36 (labels removed):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight prometheus"&gt;&lt;code&gt;&lt;span class="n"&gt;prometheus_tsdb_lowest_timestamp_seconds&lt;/span&gt;                  &lt;span class="mi"&gt;1791979200&lt;/span&gt;      &lt;span class="c1"&gt;# 2026-10-14T12:00:00Z&lt;/span&gt;
&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;time&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;prometheus_tsdb_lowest_timestamp_seconds&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mi"&gt;86400&lt;/span&gt; &lt;span class="mf"&gt;31.521971956019602&lt;/span&gt;
&lt;span class="n"&gt;prometheus_tsdb_head_series&lt;/span&gt;                               &lt;span class="mi"&gt;218919&lt;/span&gt;
&lt;span class="n"&gt;prometheus_tsdb_storage_blocks_bytes&lt;/span&gt;                      &lt;span class="mi"&gt;23835478583&lt;/span&gt;
&lt;span class="n"&gt;prometheus_tsdb_wal_storage_size_bytes&lt;/span&gt;                    &lt;span class="mi"&gt;393179565&lt;/span&gt;
&lt;span class="n"&gt;prometheus_tsdb_retention_limit_bytes&lt;/span&gt;                     &lt;span class="mi"&gt;0&lt;/span&gt;
&lt;span class="n"&gt;prometheus_tsdb_compactions_failed_total&lt;/span&gt;                  &lt;span class="mi"&gt;0&lt;/span&gt;
&lt;span class="n"&gt;prometheus_tsdb_wal_corruptions_total&lt;/span&gt;                     &lt;span class="mi"&gt;1&lt;/span&gt;
&lt;span class="n"&gt;prometheus_tsdb_blocks_loaded&lt;/span&gt;                             &lt;span class="mi"&gt;19&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;prometheus_tsdb_retention_limit_bytes 0&lt;/code&gt; means no size cap: only the 30-day time limit deleted data.&lt;/p&gt;

&lt;h3&gt;
  
  
  The restart the day before
&lt;/h3&gt;

&lt;p&gt;On 2026-11-14 the node rebooted; the Prometheus containers restarted at 01:12:30 (reason &lt;code&gt;Unknown&lt;/code&gt;). Two minutes later Prometheus logged, as collected by the kubegap log scan of that pod:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;2026-11-14T01:14:32.965Z level=WARN source=db.go:1019 msg="Encountered WAL read error, attempting repair" component=tsdb err="read records: c…
2026-11-14T01:14:32.965Z level=WARN source=wlog.go:413 msg="Starting corruption repair" component=tsdb segment=8483 offset=19763238
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;(The &lt;code&gt;err&lt;/code&gt; text is cut in the scan; the pod and its full log no longer exist.) This is the &lt;code&gt;WAL corruptions 1&lt;/code&gt; in the finding.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why: the prometheus operator emptyDir default puts the TSDB in the pod
&lt;/h2&gt;

&lt;h3&gt;
  
  
  No &lt;code&gt;storage&lt;/code&gt; means emptyDir, in the operator code
&lt;/h3&gt;

&lt;p&gt;kube-prometheus-stack 79.5.0 ships &lt;code&gt;storageSpec: {}&lt;/code&gt; in its &lt;code&gt;values.yaml&lt;/code&gt;, and our values file never set it (&lt;code&gt;git log -S storageSpec&lt;/code&gt; on that file is empty back to its first commit, 2025-11-11). With nothing there, the Prometheus object has no &lt;code&gt;spec.storage&lt;/code&gt;, and prometheus-operator v0.86.2 (&lt;code&gt;pkg/prometheus/server/statefulset.go&lt;/code&gt;, lines 91–130) does this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="n"&gt;storageSpec&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;cpf&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Storage&lt;/span&gt;
&lt;span class="k"&gt;switch&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="k"&gt;case&lt;/span&gt; &lt;span class="n"&gt;storageSpec&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="kc"&gt;nil&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;statefulset&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Spec&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Template&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Spec&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Volumes&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;statefulset&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Spec&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Template&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Spec&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Volumes&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;v1&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Volume&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="nc"&gt;Name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;prompkg&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;VolumeName&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="nc"&gt;VolumeSource&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;v1&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;VolumeSource&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="nc"&gt;EmptyDir&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="n"&gt;v1&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;EmptyDirVolumeSource&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;
&lt;span class="k"&gt;default&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="c1"&gt;// storageSpec.VolumeClaimTemplate&lt;/span&gt;
    &lt;span class="n"&gt;pvcTemplate&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;operator&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;MakeVolumeClaimTemplate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;storageSpec&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;VolumeClaimTemplate&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;pvcTemplate&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Name&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;""&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;pvcTemplate&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Name&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;prompkg&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;VolumeName&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;
    &lt;span class="n"&gt;statefulset&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Spec&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;VolumeClaimTemplates&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;statefulset&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Spec&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;VolumeClaimTemplates&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;pvcTemplate&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;VolumeName&lt;/code&gt; is &lt;code&gt;prometheus-&amp;lt;name&amp;gt;-db&lt;/code&gt; (&lt;code&gt;pkg/prometheus/common.go&lt;/code&gt;, lines 177–183), which is our &lt;code&gt;prometheus-kube-prometheus-stack-prometheus-db&lt;/code&gt;. The operator documentation says the same in one sentence: "By default, the operator configures Pods to store data on &lt;code&gt;emptyDir&lt;/code&gt; volumes which aren't persisted when the Pods are redeployed."&lt;/p&gt;

&lt;h3&gt;
  
  
  What an emptyDir survives, and when "Prometheus data lost after pod restart" is true
&lt;/h3&gt;

&lt;p&gt;The Kubernetes documentation for emptyDir: "When a Pod is removed from a node for any reason, the data in the &lt;code&gt;emptyDir&lt;/code&gt; is deleted permanently." and "A container crashing does &lt;em&gt;not&lt;/em&gt; remove a Pod from a node. The data in an &lt;code&gt;emptyDir&lt;/code&gt; volume is safe across container crashes." The data is deleted when a new pod is created; a restart of the Prometheus container inside the same pod keeps the emptyDir.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F4e1aky0nesaypous820k.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F4e1aky0nesaypous820k.png" alt="Two columns: events that restart the container in the same pod and keep the emptyDir (node reboot, marked as seen here on 11-14, OOMKill, liveness failure), and events that create a new pod and delete it (adding storageSpec, kubectl delete pod, a chart change to the pod template, drain, eviction, node replacement)." width="799" height="407"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Which everyday events keep an emptyDir and which delete it. Both columns happened on this Prometheus, one day apart. The node reboot is in the left column because on 2026-11-14 ours restarted the containers in the same pod and the emptyDir stayed.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Prometheus does not replicate its data. Its storage documentation (v3.7.3) says local storage "is not clustered or replicated" and "should be managed like any other single node database". With one replica and an emptyDir, one pod replacement removes all of it.&lt;/p&gt;

&lt;h3&gt;
  
  
  Adding &lt;code&gt;storageSpec&lt;/code&gt;: "recreating StatefulSet because the update operation wasn't possible"
&lt;/h3&gt;

&lt;p&gt;Kubernetes does not allow &lt;code&gt;volumeClaimTemplates&lt;/code&gt; to change on an existing StatefulSet, so when &lt;code&gt;storage&lt;/code&gt; appears on a running Prometheus the update is rejected with 422, and the operator (&lt;code&gt;pkg/prometheus/server/operator.go&lt;/code&gt;, lines 984–1003) deletes the StatefulSet and creates it again:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;ok&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="n"&gt;sErr&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;ErrStatus&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Code&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;422&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="n"&gt;sErr&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;ErrStatus&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Reason&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="n"&gt;metav1&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;StatusReasonInvalid&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;
    &lt;span class="n"&gt;logger&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Info&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"recreating StatefulSet because the update operation wasn't possible"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"reason"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;strings&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;failMsg&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;", "&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="n"&gt;propagationPolicy&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;metav1&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;DeletePropagationForeground&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;ssetClient&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Delete&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;sset&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;GetName&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;metav1&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;DeleteOptions&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nc"&gt;PropagationPolicy&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="n"&gt;propagationPolicy&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="kc"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;fmt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Errorf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"failed to delete StatefulSet to avoid forbidden action: %w"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="k"&gt;continue&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Foreground propagation deletes the pod before the StatefulSet is gone, and the emptyDir goes with the pod. We saw exactly this log line during the move (next sections). So adding &lt;code&gt;storageSpec&lt;/code&gt; and syncing, with nothing else, starts Prometheus on an empty PVC and deletes the old TSDB.&lt;/p&gt;

&lt;p&gt;A Red Hat blog post, as quoted in a search result summary (the page itself answered 404 when we fetched it), says "it is not possible to retain data that was written on the default EmptyDir-based installation". Our move contradicts that, and the reason is in how a StatefulSet names its PVCs: the PVC a StatefulSet pod uses is named &lt;code&gt;&amp;lt;volumeClaimTemplate name&amp;gt;-&amp;lt;pod name&amp;gt;&lt;/code&gt;, and a PVC that already exists under that name is used as it is. If you create it first and put the TSDB in it, the recreated pod opens the old data.&lt;/p&gt;

&lt;h3&gt;
  
  
  Where the files have to sit: &lt;code&gt;prometheus-db/&lt;/code&gt;
&lt;/h3&gt;

&lt;p&gt;The mount path is &lt;code&gt;/prometheus&lt;/code&gt; in both cases, but the subPath is not (&lt;code&gt;pkg/prometheus/common.go&lt;/code&gt;, lines 197–204):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="nc"&gt;SubPathForStorage&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;monitoringv1&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;StorageSpec&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="kt"&gt;string&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="kc"&gt;nil&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;DisableMountSubPath&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="s2"&gt;""&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="s2"&gt;"prometheus-db"&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Without &lt;code&gt;storage&lt;/code&gt; the emptyDir is mounted at its root; with a &lt;code&gt;volumeClaimTemplate&lt;/code&gt; the pod mounts the &lt;code&gt;prometheus-db&lt;/code&gt; directory of the PVC. A copy into the root of the PVC would be invisible to Prometheus.&lt;/p&gt;

&lt;h3&gt;
  
  
  What "Encountered WAL read error, attempting repair" dropped on 11-14
&lt;/h3&gt;

&lt;p&gt;In Prometheus v3.7.3 the repair that follows &lt;code&gt;Encountered WAL read error, attempting repair&lt;/code&gt; discards all data after the corruption point: it deletes every WAL segment newer than the corrupted one and rewrites the corrupted one only up to the offset (&lt;code&gt;tsdb/wlog/wlog.go&lt;/code&gt; line 399; the log lines at 413, 421 and 443). So the samples after segment 8483, offset 19763238, were lost, while the 19 blocks on disk were not touched. We did not measure how many samples that was.&lt;/p&gt;

&lt;h3&gt;
  
  
  No size cap on a shared disk
&lt;/h3&gt;

&lt;p&gt;With no size cap (the &lt;code&gt;prometheus_tsdb_retention_limit_bytes 0&lt;/code&gt; above) on a filesystem shared with the rest of &lt;code&gt;server-1&lt;/code&gt;, nothing would have stopped the TSDB from filling the node disk; that is why the same change sets &lt;code&gt;retentionSize&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  What to check in your cluster now
&lt;/h2&gt;

&lt;p&gt;Steps 1 to 4 are read-only and need kubectl and jq. The names are the kube-prometheus-stack defaults for release name &lt;code&gt;kube-prometheus-stack&lt;/code&gt; in namespace &lt;code&gt;monitoring&lt;/code&gt;; change them to yours.&lt;/p&gt;

&lt;h3&gt;
  
  
  1. Where does the TSDB live?
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl -n monitoring get prometheus -o &lt;span class="nv"&gt;jsonpath&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;'{range .items[*]}{.metadata.name}{" storage="}{.spec.storage}{" retentionSize="}{.spec.retentionSize}{"\n"}{end}'&lt;/span&gt;
kubectl -n monitoring get sts -l app.kubernetes.io/name=prometheus -o json &lt;span class="o"&gt;|&lt;/span&gt; jq -r &lt;span class="s1"&gt;'.items[] | .metadata.name, "  volumeClaimTemplates: \([.spec.volumeClaimTemplates[]?.metadata.name])", (.spec.template.spec.volumes[] | select(.name|test("-db$")) | "  \(.name): \(if .emptyDir then "emptyDir" else "not emptyDir" end)")'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Same problem if:&lt;/strong&gt; &lt;code&gt;storage=&lt;/code&gt; is empty, the &lt;code&gt;-db&lt;/code&gt; volume says &lt;code&gt;emptyDir&lt;/code&gt;, and &lt;code&gt;volumeClaimTemplates: []&lt;/code&gt;. Ours: &lt;code&gt;storage= retentionSize=&lt;/code&gt; and &lt;code&gt;prometheus-kube-prometheus-stack-prometheus-db: emptyDir&lt;/code&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. How much is in it?
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl -n monitoring &lt;span class="nb"&gt;exec&lt;/span&gt; prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- sh -c &lt;span class="s1"&gt;'du -sh /prometheus; df -h /prometheus'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Same problem if:&lt;/strong&gt; step 1 said &lt;code&gt;emptyDir&lt;/code&gt;; then everything &lt;code&gt;du&lt;/code&gt; prints is lost the next time the pod is deleted. &lt;code&gt;df&lt;/code&gt; shows which disk it shares, but it cannot tell an emptyDir from a local-path PVC: on k3s the local-path PVC we moved to sits on the same node disk as the emptyDir did. Ours: 22.7G, on a 454.9G root filesystem 57% used.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. How far back does it go, and is there a size cap?
&lt;/h3&gt;

&lt;p&gt;In the Prometheus UI or with &lt;code&gt;promtool query instant&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;(time() - prometheus_tsdb_lowest_timestamp_seconds) / 86400
prometheus_tsdb_retention_limit_bytes
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Same problem if:&lt;/strong&gt; the second value is &lt;code&gt;0&lt;/code&gt;. That means no &lt;code&gt;retentionSize&lt;/code&gt;: only time deletes data, and nothing stops the TSDB from filling the disk it shares with the node. Ours: 31.5 days and &lt;code&gt;0&lt;/code&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  4. Has the pod been replaced, and did a restart drop WAL samples?
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl -n monitoring get pod prometheus-kube-prometheus-stack-prometheus-0 -o &lt;span class="nv"&gt;jsonpath&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;'{.metadata.creationTimestamp}{"  restarts: "}{.status.containerStatuses[?(@.name=="prometheus")].restartCount}{"\n"}'&lt;/span&gt;
kubectl -n monitoring logs prometheus-kube-prometheus-stack-prometheus-0 -c prometheus &lt;span class="o"&gt;|&lt;/span&gt; grep -E &lt;span class="s2"&gt;"attempting repair|Starting corruption repair"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A container restart (an OOMKill, for example) keeps the emptyDir; ours on 2026-11-14 was a node reboot that restarted the containers in the same pod, and the emptyDir stayed. &lt;strong&gt;Same problem if:&lt;/strong&gt; step 1 said &lt;code&gt;emptyDir&lt;/code&gt;, whatever the values in this step are; they only show how close you have already been. With an emptyDir, the oldest sample cannot be older than the pod. If the creation time is close to your oldest sample, the pod was replaced then and everything before it was deleted with the old pod. If the creation time is older than your retention, every restart so far kept the data, and the next pod replacement deletes all of it. A repair line in the log means one of those restarts has already dropped the newest WAL samples. Ours: pod created 2025-11-30, last container restart at a node reboot on 2026-11-14, followed by &lt;code&gt;Encountered WAL read error, attempting repair&lt;/code&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  Doing this on every run
&lt;/h3&gt;

&lt;p&gt;This problem was found by &lt;a href="https://github.com/KyungHwanKim-devs/kubegap" rel="noopener noreferrer"&gt;kubegap&lt;/a&gt;, an open-source audit that reads the cluster your kubectl points at and lists operational risks with the evidence for each. kubegap repeats step 1, step 3 and the log check of step 4 on each run: its summary reads the storage kind (emptyDir, or the PVC template and its size), &lt;code&gt;retentionSize&lt;/code&gt;, the oldest sample in days, the WAL corruption count and the PVC usage ratio, and reports them as the findings &lt;code&gt;prometheus.storage-risk&lt;/code&gt;, &lt;code&gt;observability.stack-storage-emptydir&lt;/code&gt; and &lt;code&gt;logs.signatures.prometheus.wal-corrupt&lt;/code&gt;. It does not read the pod creation time or restart count of step 4. After the move, a full run at 00:44:32 no longer reported any of the three for Prometheus:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F5jeoue3nbj2jvm7s7cy2.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F5jeoue3nbj2jvm7s7cy2.png" alt="Table of three kubegap findings for Prometheus, fail or warn before the move and not reported after it, with the storage type, size cap, oldest sample and WAL corruption count from each run." width="799" height="407"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;kubegap full runs collected 2026-11-14 16:25 UTC (the report, v0.3.0) and 2026-11-15 00:44 UTC (after the move, one commit after v0.3.0: v0.3.0-1-g8c36a92).&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;One gap remains in that run: its PVC usage value (&lt;code&gt;pvc_used_ratio&lt;/code&gt;) is null, because the kubelet exports no &lt;code&gt;kubelet_volume_stats_*&lt;/code&gt; series for this PVC. At 00:50:02 only 2 of the cluster's 39 PVs had one. We did not investigate why the kubelet reports nothing for these local volumes.&lt;/p&gt;

&lt;h2&gt;
  
  
  What we changed, and what we have not
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Changed: kube-prometheus-stack persistent storage, or how to add persistent volume to Prometheus without losing data
&lt;/h3&gt;

&lt;p&gt;Prometheus is installed by the Argo CD Application &lt;code&gt;monitoring&lt;/code&gt; from the chart values in the manifests repository, with automated &lt;code&gt;prune: true&lt;/code&gt; and &lt;code&gt;selfHeal: true&lt;/code&gt;, so the change had to go through Git. We filled the PVC before the commit, because the sync would delete the pod (see "Adding &lt;code&gt;storageSpec&lt;/code&gt;" under Why) and with it the only copy of 31 days of metrics.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fi1xfptec8aohkczl2fj2.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fi1xfptec8aohkczl2fj2.png" alt="Diagram of node server-1: a copy pod reads the old emptyDir from the host and writes it into prometheus-db/ on a PVC created by hand under the name the StatefulSet expects; the new Prometheus pod then mounts that folder at /prometheus." width="800" height="338"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Where the files went so the recreated pod found them. One copy pod on the same node mounted the old emptyDir from the host, read-only, and the new PVC.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;1. The PVC and a copy pod (00:32:52).&lt;/strong&gt; Applied with kubectl for the move only, not kept in Git (&lt;code&gt;server-1&lt;/code&gt; and the pod uid are masked):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;PersistentVolumeClaim&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0&lt;/span&gt;
  &lt;span class="na"&gt;namespace&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;monitoring&lt;/span&gt;
&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;storageClassName&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;local-path-server-1&lt;/span&gt;
  &lt;span class="na"&gt;accessModes&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s"&gt;ReadWriteOnce&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
  &lt;span class="na"&gt;resources&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;storage&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;50Gi&lt;/span&gt;
&lt;span class="nn"&gt;---
&lt;/span&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Pod&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;prometheus-data-copy&lt;/span&gt;
  &lt;span class="na"&gt;namespace&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;monitoring&lt;/span&gt;
&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;nodeSelector&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;kubernetes.io/hostname&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;server-1&lt;/span&gt;
  &lt;span class="na"&gt;restartPolicy&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Never&lt;/span&gt;
  &lt;span class="na"&gt;containers&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;copy&lt;/span&gt;
      &lt;span class="na"&gt;image&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;alpine:3.20&lt;/span&gt;
      &lt;span class="na"&gt;command&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"sh"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"-c"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"apk add --no-cache rsync &amp;gt;/dev/null &amp;amp;&amp;amp; mkdir -p /dst/prometheus-db &amp;amp;&amp;amp; sleep infinity"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
      &lt;span class="na"&gt;volumeMounts&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="s"&gt;name:&lt;/span&gt; &lt;span class="s"&gt;src&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;mountPath:&lt;/span&gt; &lt;span class="s"&gt;/src&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;readOnly:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
        &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="s"&gt;name:&lt;/span&gt; &lt;span class="s"&gt;dst&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;mountPath:&lt;/span&gt; &lt;span class="s"&gt;/dst&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="na"&gt;volumes&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;src&lt;/span&gt;
      &lt;span class="na"&gt;hostPath&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="na"&gt;path&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;/var/lib/kubelet/pods/&amp;lt;pod uid&amp;gt;/volumes/kubernetes.io~empty-dir/prometheus-kube-prometheus-stack-prometheus-db&lt;/span&gt;
        &lt;span class="na"&gt;type&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Directory&lt;/span&gt;
    &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;dst&lt;/span&gt;
      &lt;span class="na"&gt;persistentVolumeClaim&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="na"&gt;claimName&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The pod uid comes from &lt;code&gt;kubectl -n monitoring get pod prometheus-kube-prometheus-stack-prometheus-0 -o jsonpath='{.metadata.uid}'&lt;/code&gt;. The StorageClass &lt;code&gt;local-path-server-1&lt;/code&gt; is a k3s local-path StorageClass: it uses the &lt;code&gt;rancher.io/local-path&lt;/code&gt; provisioner, keeps its volumes on the same node disk as the emptyDir, and has binding mode &lt;code&gt;WaitForFirstConsumer&lt;/code&gt;, so the PVC was bound only once a pod used it (here the copy pod); the volume was provisioned at 00:33:01.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. First copy while Prometheus kept running (00:33:20–00:37:53).&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;$ kubectl -n monitoring &lt;span class="nb"&gt;exec&lt;/span&gt; prometheus-data-copy -- rsync -a --delete --stats --exclude=lock /src/ /dst/prometheus-db/
Number of files: 158 &lt;span class="o"&gt;(&lt;/span&gt;reg: 116, dir: 42&lt;span class="o"&gt;)&lt;/span&gt;
Number of regular files transferred: 116
Total transferred file size: 24,396,762,944 bytes
sent 24,414,426,021 bytes  received 2,562 bytes  89,266,649.30 bytes/sec
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;4 min 33 s at 89 MB/s. The second run used the same command.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Second copy right before the switch (00:38:04–00:38:05).&lt;/strong&gt; Only the WAL and head chunks had changed:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Number of files: 158 (reg: 116, dir: 42)
Total transferred file size: 108,133,626 bytes
22.7G   /dst/prometheus-db
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;4. The values change (pushed 00:38:05, Argo CD refresh requested 00:38:08).&lt;/strong&gt; Comment lines omitted:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight diff"&gt;&lt;code&gt; prometheus:
   prometheusSpec:
     retention: 30d
&lt;span class="gi"&gt;+    retentionSize: 40GB&lt;/span&gt;
&lt;span class="gi"&gt;+    storageSpec:&lt;/span&gt;
&lt;span class="gi"&gt;+      volumeClaimTemplate:&lt;/span&gt;
&lt;span class="gi"&gt;+        spec:&lt;/span&gt;
&lt;span class="gi"&gt;+          storageClassName: local-path-server-1&lt;/span&gt;
&lt;span class="gi"&gt;+          accessModes: ["ReadWriteOnce"]&lt;/span&gt;
&lt;span class="gi"&gt;+          resources:&lt;/span&gt;
&lt;span class="gi"&gt;+            requests:&lt;/span&gt;
&lt;span class="gi"&gt;+              storage: 50Gi&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;5. The operator recreated the StatefulSet (00:38:26)&lt;/strong&gt;, as the source says it would:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;ts=2026-11-15T00:38:26.821426997Z level=info caller=/workspace/pkg/prometheus/server/operator.go:996 msg="recreating StatefulSet because the update operation wasn't possible" component=prometheus-controller key=monitoring/kube-prometheus-stack-prometheus statefulset=prometheus-kube-prometheus-stack-prometheus shard=0 reason="Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'revisionHistoryLimit', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden"
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The old pod's containers were stopped (&lt;code&gt;Killing&lt;/code&gt;) the same second, and the new pod was created on &lt;code&gt;server-1&lt;/code&gt; at 00:38:30. Its data mount and the new argument:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="nl"&gt;"dbMount"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;"mountPath"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"/prometheus"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"name"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"prometheus-kube-prometheus-stack-prometheus-db"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"subPath"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"prometheus-db"&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="err"&gt;...&lt;/span&gt; &lt;span class="s2"&gt;"--storage.tsdb.retention.time=30d"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="s2"&gt;"--storage.tsdb.retention.size=40GB"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="s2"&gt;"--storage.tsdb.path=/prometheus"&lt;/span&gt; &lt;span class="err"&gt;...&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;6. Prometheus opened the copied TSDB.&lt;/strong&gt; From the new pod's startup log (19 &lt;code&gt;Found healthy block&lt;/code&gt; lines, the first shown):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;00:38:32.659Z msg="Found healthy block" component=tsdb mint=1791979200038 maxt=1792173600000 ulid=01M53F5PAQ7C40DZXPAKF6RB13
00:38:33.786Z msg="Replaying WAL, this may take a while" component=tsdb
00:39:21.785Z msg="WAL replay completed" component=tsdb checkpoint_replay_duration=2.889331339s wal_replay_duration=45.109577526s ... total_replay_duration=48.613000449s
00:39:22.360Z msg="TSDB started"
00:39:22.867Z msg="Server is ready to receive web requests."
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;mint&lt;/code&gt; of the first block is the old lowest timestamp, 1791979200. The startup probe failed once with 503 during the replay (00:39:15); the pod was Ready at 00:39:32. At 00:39:55: &lt;code&gt;du -sh /prometheus&lt;/code&gt; 22.7G, lowest timestamp 1791979200 (2026-10-14T12:00:00Z), 19 blocks, &lt;code&gt;prometheus_tsdb_retention_limit_bytes&lt;/code&gt; 42,949,672,960, &lt;code&gt;prometheus_tsdb_wal_corruptions_total&lt;/code&gt; 0. The copy pod was deleted at 00:40:09.&lt;/p&gt;

&lt;h3&gt;
  
  
  The proof: deleting the pod on purpose
&lt;/h3&gt;

&lt;p&gt;To check that a new pod keeps the data, we deleted the pod at 00:41:41 and compared the same values before and after. Before, at 00:41:40: lowest timestamp 1791979200, 19 blocks, 22.7G.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;$ kubectl -n monitoring delete pod prometheus-kube-prometheus-stack-prometheus-0   &lt;span class="c"&gt;# 00:41:41&lt;/span&gt;
pod &lt;span class="s2"&gt;"prometheus-kube-prometheus-stack-prometheus-0"&lt;/span&gt; deleted from monitoring namespace
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The new pod was created at 00:41:43 on the same PVC (Bound, 11 min old), replayed the WAL in 54.4 s, logged &lt;code&gt;TSDB started&lt;/code&gt; at 00:42:44 and was ready by 00:43:03. At 00:43:57 all five values were the same as at 00:39:55.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fbqe74kvzv1b3w2gudhm0.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fbqe74kvzv1b3w2gudhm0.png" alt="Prometheus graph of prometheus_tsdb_head_series over 5 weeks after both pod replacements; the line still starts around Oct 14." width="800" height="426"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Prometheus at 00:44:17, after both pod replacements: the 5-week graph still starts around Oct 14. The colour changes at the right end because the &lt;code&gt;instance&lt;/code&gt; label (the pod IP) changed with each new pod.&lt;/em&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  What it cost
&lt;/h3&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fe3lwimuf8hcj5tmnyh8t.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fe3lwimuf8hcj5tmnyh8t.png" alt="Timeline of 00:33 to 00:44 UTC: two rsync copies, the Git push and the StatefulSet recreation, then two gaps in Prometheus's own 30-second samples, one around the switch and one around the deliberate pod delete." width="800" height="349"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;What the move cost, measured from Prometheus scraping itself every 30 s: two holes in new samples; the stored 30 days stayed.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The self-scrape series &lt;code&gt;up{job="kube-prometheus-stack-prometheus"}&lt;/code&gt; has no samples from 00:37:45 to 00:39:51 (2 min 6 s) and from 00:41:21 to 00:42:51 (1 min 30 s). The second hole is the deliberate restart: nothing was scraped while the pod was down, and nothing stored was lost. For the first hole, the inference from the times above: the last sample before the second copy was scraped at 00:37:45 and is in the copy; the next one, due about 30 s later, was written after the copy finished at 00:38:05, while the old pod ran until 00:38:26, so it stayed in the old emptyDir and was deleted with it; after that nothing scraped until the new pod was up. A third rsync between the push and the recreate would have narrowed it; we did not try.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why 50Gi and &lt;code&gt;retentionSize: 40GB&lt;/code&gt;
&lt;/h3&gt;

&lt;p&gt;&lt;code&gt;40GB&lt;/code&gt; in &lt;code&gt;retentionSize&lt;/code&gt; is 40 GiB: Prometheus documents the units as "Based on powers-of-2, so 1KB is 1024B", the operator's &lt;code&gt;ByteSize&lt;/code&gt; type says the same, and the metric shows 42,949,672,960 bytes. That is 80% of the 50Gi request, within the Prometheus documentation's advice "setting the retention size to, at most, 80-85% of your allocated Prometheus disk space". With both limits set, whichever triggers first is used; the current 22.7G is well below either.&lt;/p&gt;

&lt;p&gt;On this StorageClass the 50Gi is not a limit. The local-path-provisioner README: "No support for the volume capacity limit currently. The capacity limit will be ignored for now." So &lt;code&gt;retentionSize&lt;/code&gt; is the only cap actually enforced, and beyond it only the free space on the node disk limits the TSDB. One more limit from the Prometheus docs: "Only the persistent blocks are deleted to honor this retention although WAL and m-mapped chunks are counted in the total size", so a growing WAL is counted but not shrunk by the cap.&lt;/p&gt;

&lt;h3&gt;
  
  
  Not changed, and why
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Alertmanager and Grafana are still on emptyDir.&lt;/strong&gt; kubegap reports &lt;code&gt;observability.stack-storage-emptydir&lt;/code&gt; for both. They were not part of problem 2 in the kubegap report; we expect the same copy-then-switch method to work for them but have not tried it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The PVC is on the same disk of the same node as the emptyDir was.&lt;/strong&gt; The data now survives a pod replacement, not the loss of the disk or of &lt;code&gt;server-1&lt;/code&gt;. Copies outside the node and the cluster are problem 1 of the same kubegap report and are not covered here. The StorageClass also has reclaim policy &lt;code&gt;Delete&lt;/code&gt;; the StatefulSet's &lt;code&gt;persistentVolumeClaimRetentionPolicy&lt;/code&gt; is &lt;code&gt;Retain&lt;/code&gt; for both deletion and scale-down, so deleting the StatefulSet keeps the PVC, but deleting the PVC by hand deletes the data.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;No volume-filling alert for the new PVC.&lt;/strong&gt; There is no usage metric for it (the kubelet exports none), so neither a &lt;code&gt;KubePersistentVolumeFillingUp&lt;/code&gt; alert nor kubegap sees it fill. &lt;code&gt;retentionSize&lt;/code&gt; is the cap that is enforced; the existing node disk alert rules were not changed.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Prometheus memory and CPU.&lt;/strong&gt; Memory was 1081Mi of the 1536Mi limit before the move, and the startup probe allows 60 × 15 s = 15 minutes; WAL replays here took 45 s and 54 s. CPU throttling and the memory limit are other items of the same kubegap report.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Where else this happened
&lt;/h2&gt;

&lt;p&gt;The Incident Library, a searchable index of public postmortems on benchlet.com (links below), has no case of a Prometheus TSDB deleted because it ran on an emptyDir. The closest cases are Prometheus restarts that went wrong at the WAL replay or the probes. This move restarts Prometheus twice and replays the WAL each time (45 s and 54 s here, under a startup probe that allows 15 minutes), so if your WAL is larger, check the startup probe before the switch:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;UK Ministry of Justice, 2024-04-15: a planned change restarted Prometheus, the WAL reload took longer than the startup probe allowed, and the platform had no monitoring for 3 h 21 min until the probe was raised to 30 minutes.&lt;/li&gt;
&lt;li&gt;Saifeddine Rajhi, 2024-06-14 (personal blog): after a kube-prometheus-stack upgrade Prometheus was OOMKilled in a loop while replaying its WAL, which needed more than 50Gi of memory against about 30Gi in steady state.&lt;/li&gt;
&lt;li&gt;UK Ministry of Justice, 2023-11-01: Prometheus on the live cluster restarted several times after readiness probe failures, leaving gaps in the metrics.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;More: &lt;a href="https://benchlet.com/incidentlib/app?component=prometheus&amp;amp;symptom=monitoring-down" rel="noopener noreferrer"&gt;Prometheus + monitoring down, 8 cases&lt;/a&gt;. Cases where data on a pod or node was gone after a change, the outcome this move avoided, are in &lt;a href="https://benchlet.com/incidentlib/app?symptom=data-loss&amp;amp;domain=kubernetes" rel="noopener noreferrer"&gt;data loss on Kubernetes, 5 cases&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Timeline
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Time (UTC)&lt;/th&gt;
&lt;th&gt;Event&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;2025-11-11 07:49:16&lt;/td&gt;
&lt;td&gt;First commit of the chart values file; it never contains &lt;code&gt;storageSpec&lt;/code&gt;.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2025-11-30 06:09:35&lt;/td&gt;
&lt;td&gt;The Prometheus pod is created on &lt;code&gt;server-1&lt;/code&gt; with its TSDB on an emptyDir.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2026-11-14 01:12:30&lt;/td&gt;
&lt;td&gt;Node reboot; the Prometheus containers restart in the same pod (reason &lt;code&gt;Unknown&lt;/code&gt;). The emptyDir is kept.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2026-11-14 01:14:32&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;Encountered WAL read error, attempting repair&lt;/code&gt;, &lt;code&gt;Starting corruption repair&lt;/code&gt; segment 8483; samples after the corruption point are discarded.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2026-11-14 16:25:29&lt;/td&gt;
&lt;td&gt;kubegap v0.3.0 full run reports problem 2: Prometheus data on an emptyDir, 22.3GiB, oldest sample 31.2 days, no size cap.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2026-11-15 00:31:25–00:31:36&lt;/td&gt;
&lt;td&gt;Our checks: emptyDir, 22.7G, oldest sample 2026-10-14T12:00:00Z, no size cap.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;00:32:52&lt;/td&gt;
&lt;td&gt;PVC &lt;code&gt;prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0&lt;/code&gt; (50Gi) and copy pod created.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;00:33:01&lt;/td&gt;
&lt;td&gt;The PVC's volume is provisioned on &lt;code&gt;server-1&lt;/code&gt;.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;00:33:20–00:37:53&lt;/td&gt;
&lt;td&gt;First rsync while Prometheus runs: 24,396,762,944 bytes, 4 min 33 s.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;00:37:45&lt;/td&gt;
&lt;td&gt;Last self-scrape sample of the old pod.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;00:38:04–00:38:05&lt;/td&gt;
&lt;td&gt;Second rsync: 108,133,626 bytes changed; 22.7G in the PVC.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;00:38:05&lt;/td&gt;
&lt;td&gt;Values commit with &lt;code&gt;retentionSize: 40GB&lt;/code&gt; and &lt;code&gt;storageSpec.volumeClaimTemplate&lt;/code&gt; pushed; Argo CD refresh at 00:38:08.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;00:38:26&lt;/td&gt;
&lt;td&gt;Operator: &lt;code&gt;recreating StatefulSet because the update operation wasn't possible&lt;/code&gt;; old pod's containers stopped.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;00:38:30&lt;/td&gt;
&lt;td&gt;New pod created on &lt;code&gt;server-1&lt;/code&gt;, PVC mounted with &lt;code&gt;subPath: prometheus-db&lt;/code&gt;.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;00:39:15&lt;/td&gt;
&lt;td&gt;Startup probe fails once with 503 during WAL replay.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;00:39:22&lt;/td&gt;
&lt;td&gt;19 healthy blocks, WAL replay 45.1 s, &lt;code&gt;TSDB started&lt;/code&gt;, ready to receive web requests.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;00:39:32&lt;/td&gt;
&lt;td&gt;Pod Ready.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;00:39:51&lt;/td&gt;
&lt;td&gt;First self-scrape sample of the new pod (hole of 2 min 6 s).&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;00:39:55&lt;/td&gt;
&lt;td&gt;Size, oldest sample and block count unchanged; size cap 40 GiB.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;00:40:09&lt;/td&gt;
&lt;td&gt;Copy pod deleted.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;00:41:41&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;kubectl delete pod prometheus-kube-prometheus-stack-prometheus-0&lt;/code&gt;; new pod created 00:41:43 on the same PVC.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;00:42:44&lt;/td&gt;
&lt;td&gt;WAL replay 54.4 s, &lt;code&gt;TSDB started&lt;/code&gt;; first self-scrape sample at 00:42:51 (hole of 1 min 30 s); ready by 00:43:03.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;00:43:57&lt;/td&gt;
&lt;td&gt;Size, oldest sample and block count unchanged.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;00:44:17&lt;/td&gt;
&lt;td&gt;5-week graph of &lt;code&gt;prometheus_tsdb_head_series&lt;/code&gt; still starts around Oct 14.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;00:44:32&lt;/td&gt;
&lt;td&gt;kubegap full run (one commit after v0.3.0, v0.3.0-1-g8c36a92): the three Prometheus findings are gone; emptyDir still reported for Alertmanager and Grafana.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;00:50:02&lt;/td&gt;
&lt;td&gt;No &lt;code&gt;kubelet_volume_stats_*&lt;/code&gt; series for the new PVC; 2 of 39 PVs have one.&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  FAQ
&lt;/h2&gt;

&lt;h3&gt;
  
  
  prometheus data lost after pod restart
&lt;/h3&gt;

&lt;p&gt;It depends on what restarted, because with no &lt;code&gt;storageSpec&lt;/code&gt; the prometheus operator emptyDir default stores the TSDB in a volume that is deleted with the pod, not when a container restarts. A container restart (OOMKill, liveness failure) keeps the emptyDir, so the blocks stay; ours was a node reboot that restarted the containers in the same pod, and the emptyDir stayed. Even then, a WAL repair after an unclean stop can still drop the newest samples. A new pod gets an empty emptyDir: drain, eviction, &lt;code&gt;kubectl delete pod&lt;/code&gt;, a chart change to the pod template, or adding &lt;code&gt;storageSpec&lt;/code&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  kube-prometheus-stack persistent storage
&lt;/h3&gt;

&lt;p&gt;Set &lt;code&gt;prometheus.prometheusSpec.storageSpec.volumeClaimTemplate&lt;/code&gt; with a StorageClass (ours, &lt;code&gt;local-path-server-1&lt;/code&gt;, is a k3s local-path StorageClass with the &lt;code&gt;rancher.io/local-path&lt;/code&gt; provisioner), access mode and size; the chart default is &lt;code&gt;storageSpec: {}&lt;/code&gt;, which leaves the operator's emptyDir. Add &lt;code&gt;retentionSize&lt;/code&gt; below the volume size (Prometheus docs: at most 80–85%). On a running Prometheus this change makes the operator recreate the StatefulSet, so copy existing data first.&lt;/p&gt;

&lt;h3&gt;
  
  
  prometheus operator add persistent volume without losing data
&lt;/h3&gt;

&lt;p&gt;Create the PVC the new StatefulSet will ask for, &lt;code&gt;prometheus-&amp;lt;name&amp;gt;-db-prometheus-&amp;lt;name&amp;gt;-0&lt;/code&gt;, with the StorageClass and size you will put in &lt;code&gt;storageSpec&lt;/code&gt;. Copy the old emptyDir from the node into &lt;code&gt;prometheus-db/&lt;/code&gt; on it, once while running and once just before the switch. Then commit &lt;code&gt;storageSpec&lt;/code&gt;. Our 30 days survived the switch and a deliberate pod delete. The cost was 2 min 6 s of samples.&lt;/p&gt;

&lt;h3&gt;
  
  
  Encountered WAL read error, attempting repair
&lt;/h3&gt;

&lt;p&gt;Prometheus hit a WAL read error while loading the head at startup. In v3.7.3 the repair deletes every WAL segment newer than the corrupted one and cuts that segment at the corruption offset, so the samples after that point are gone; blocks are not touched. &lt;code&gt;prometheus_tsdb_wal_corruptions_total&lt;/code&gt; counts it. Ours followed a node reboot.&lt;/p&gt;

&lt;h3&gt;
  
  
  retentionSize GB or GiB
&lt;/h3&gt;

&lt;p&gt;GiB. Both Prometheus and prometheus-operator define the units as powers of 2, so 1KB is 1024B and our &lt;code&gt;retentionSize: 40GB&lt;/code&gt; shows up as &lt;code&gt;prometheus_tsdb_retention_limit_bytes&lt;/code&gt; 42,949,672,960. The cap counts WAL and m-mapped head chunks, but only blocks are deleted to honour it, so leave headroom below the volume.&lt;/p&gt;

&lt;h3&gt;
  
  
  local-path provisioner PVC size not enforced
&lt;/h3&gt;

&lt;p&gt;Not enforced: the local-path-provisioner README says "The capacity limit will be ignored for now." Our &lt;code&gt;storage: 50Gi&lt;/code&gt; does not stop the TSDB from growing, so &lt;code&gt;retentionSize&lt;/code&gt; (ours &lt;code&gt;40GB&lt;/code&gt;) is the only cap, and beyond it only the free space on the node disk limits the TSDB.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://benchlet.com/guides/prometheus-emptydir-data-loss-kube-prometheus-stack-storagespec-pvc/" rel="noopener noreferrer"&gt;benchlet.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>kubernetes</category>
      <category>prometheus</category>
      <category>monitoring</category>
      <category>sre</category>
    </item>
    <item>
      <title>Will Argo CD auto-prune delete your namespace? How to prevent it with Prune=false and Delete=false</title>
      <dc:creator>KYUNGHWAN KIM</dc:creator>
      <pubDate>Sat, 14 Nov 2026 21:06:37 +0000</pubDate>
      <link>https://dev.to/kkhdevs/will-argo-cd-auto-prune-delete-your-namespace-how-to-prevent-it-with-prunefalse-and-deletefalse-1g5h</link>
      <guid>https://dev.to/kkhdevs/will-argo-cd-auto-prune-delete-your-namespace-how-to-prevent-it-with-prunefalse-and-deletefalse-1g5h</guid>
      <description>&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F2dtu45ydz500ru0azz0y.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F2dtu45ydz500ru0azz0y.png" alt="A kustomization.yaml with the line &amp;quot;- namespace.yaml&amp;quot; struck out, an arrow labelled &amp;quot;2 min 48 s, automated sync, prune: true&amp;quot;, and the Namespace prune-test-a marked DELETED together with a ConfigMap that was still in Git and a Secret that had been made by hand. Below: &amp;quot;What stops it, on the Namespace: argocd.argoproj.io/sync-options: Prune=false,Delete=false&amp;quot;. Measured with a throwaway test Application on Argo CD v3.2.0; no production Namespace was deleted." width="800" height="420"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;If you searched "argocd deleted namespace", "argocd prune deleted my namespace" or "argocd prevent namespace deletion", run &lt;code&gt;kubectl logs -n argocd sts/argocd-application-controller | grep "Adding resource result"&lt;/code&gt;. A Namespace that Argo CD has just deleted shows &lt;code&gt;Pruned&lt;/code&gt; / &lt;code&gt;pruned&lt;/code&gt;; one that it skipped shows &lt;code&gt;PruneSkipped&lt;/code&gt; / &lt;code&gt;ignored (no prune)&lt;/code&gt;. The cause of the first: an Application with automated &lt;code&gt;prune: true&lt;/code&gt; tracks its own Namespace (or a CRD), and that object's manifest dropped out of what Git renders, so the next automated sync deleted it with everything inside. The difference between the two results is one annotation on the object, &lt;code&gt;argocd.argoproj.io/sync-options: Prune=false,Delete=false&lt;/code&gt;: &lt;code&gt;Prune=false&lt;/code&gt; stops the prune, and &lt;code&gt;Delete=false&lt;/code&gt; covers the other path, where an ApplicationSet deletes the Application and &lt;code&gt;resources-finalizer.argocd.argoproj.io&lt;/code&gt; deletes the resources it tracks. On our cluster an audit found 8 production Namespaces and 23 CRDs exposed this way. Nothing was deleted in production: we reproduced both deletion paths with throwaway test Applications, then fixed the manifests. Below are the raw output of that test, the Argo CD source lines that decide which annotation blocks which path, why the &lt;code&gt;auto-sync will wipe out all resources&lt;/code&gt; protection (&lt;code&gt;allowEmpty&lt;/code&gt;) does not cover one removed manifest, read-only checks with the value that means you have the same problem, and what we changed and did not change, including an &lt;code&gt;argocd_app_info{sync_status="OutOfSync"}&lt;/code&gt; alert. The cluster is a 2-node k3s cluster (k3s v1.33.5+k3s1) running Argo CD v3.2.0, with the manifests repository on GitHub; all times are UTC, 2026-11-14.&lt;/p&gt;

&lt;h2&gt;
  
  
  What we saw: &lt;code&gt;Pruned&lt;/code&gt; / &lt;code&gt;pruned&lt;/code&gt; versus &lt;code&gt;PruneSkipped&lt;/code&gt; / &lt;code&gt;ignored (no prune)&lt;/code&gt;
&lt;/h2&gt;

&lt;h3&gt;
  
  
  The finding
&lt;/h3&gt;

&lt;p&gt;The finding came from a full run of kubegap v0.3.0, an open-source audit of Kubernetes cluster operations, taken at 16:25:29. Its problem 3, translated from Korean: "Argo CD apps with prune enabled manage a Namespace and CRDs." The reason it gave: if one manifest drops out of Git or a path changes, Argo CD can delete a production namespace with the PVCs and Secrets in it, or delete the Istio and Gateway API CRDs and with them every object of those kinds. The report counted 9 of 11 production Applications with automated &lt;code&gt;prune: true&lt;/code&gt; that track a Namespace or CRD: 8 track one Namespace each, and &lt;code&gt;istio&lt;/code&gt; tracks 23 CRDs. All 9 have &lt;code&gt;selfHeal&lt;/code&gt; and the finalizer &lt;code&gt;resources-finalizer.argocd.argoproj.io&lt;/code&gt;. The report had not opened the repository; it inferred that the Git directories contain a Namespace manifest.&lt;/p&gt;

&lt;p&gt;At 19:11 we ran the check ourselves (names masked, CRD lines shortened):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;$ kubectl get applications.argoproj.io -A -o json &lt;span class="o"&gt;|&lt;/span&gt; jq -r &lt;span class="s1"&gt;'.items[] | select(.spec.syncPolicy.automated.prune==true) | .metadata.name as $n | .status.resources[]? | select(.kind=="Namespace" or .kind=="CustomResourceDefinition") | $n+" "+.kind+"/"+.name'&lt;/span&gt;
prod-a Namespace/prod-a
prod-j Namespace/prod-j
prod-k Namespace/prod-k
prod-c Namespace/prod-c
prod-g Namespace/prod-g
istio CustomResourceDefinition/authorizationpolicies.security.istio.io
istio CustomResourceDefinition/backendtlspolicies.gateway.networking.k8s.io
istio CustomResourceDefinition/destinationrules.networking.istio.io
istio CustomResourceDefinition/envoyfilters.networking.istio.io
... 19 more CustomResourceDefinition lines &lt;span class="k"&gt;for&lt;/span&gt; istio
prod-i Namespace/prod-i
prod-l Namespace/prod-l
prod-h Namespace/prod-h
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fv007g86ecuxq9jrzv9rh.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fv007g86ecuxq9jrzv9rh.png" alt="Three bars for the 8 production Namespaces. 16 PVCs: 8 declared in Git and tracked by Argo CD, 8 made by StatefulSet volumeClaimTemplates and not tracked. 44 Secrets: 26 owned by an ExternalSecret, 18 with no owner. 16 volumes: 15 with reclaim policy Retain, 1 with Delete." width="800" height="327"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;What the 8 production Namespaces held before the fix. Half of the PVCs and 18 of the 44 Secrets exist only in the cluster.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The rest of what we read at 19:11:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The cluster has 22 Applications. 21 have automated &lt;code&gt;prune: true&lt;/code&gt;. 15 are generated by one ApplicationSet, &lt;code&gt;prod-apps&lt;/code&gt;, and 7 are tracked by an Application named &lt;code&gt;root&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;None of the 8 Namespaces had an &lt;code&gt;argocd.argoproj.io/sync-options&lt;/code&gt; annotation. Each had the tracking annotation &lt;code&gt;argocd.argoproj.io/tracking-id: &amp;lt;app&amp;gt;:/Namespace:/&amp;lt;name&amp;gt;&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;The inference of the report was right: the manifests repository has a Namespace manifest in each of the 8 directories, 7 as a &lt;code&gt;namespace.yaml&lt;/code&gt; listed in &lt;code&gt;kustomization.yaml&lt;/code&gt; and one inside a larger manifest file.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;prod-apps&lt;/code&gt; uses a git directory generator on &lt;code&gt;prod/*&lt;/code&gt;. Its template has &lt;code&gt;prune: true&lt;/code&gt;, &lt;code&gt;selfHeal: true&lt;/code&gt;, &lt;code&gt;CreateNamespace=true&lt;/code&gt;, no &lt;code&gt;finalizers&lt;/code&gt;, and the ApplicationSet has no &lt;code&gt;spec.syncPolicy&lt;/code&gt;. The generated Applications carry &lt;code&gt;resources-finalizer.argocd.argoproj.io&lt;/code&gt; all the same, and they have no &lt;code&gt;destination.namespace&lt;/code&gt;; the namespace comes from &lt;code&gt;namespace:&lt;/code&gt; in each kustomization.&lt;/li&gt;
&lt;li&gt;The 8 namespaces held 51 pods, 16 PVCs and 44 Secrets. Counted later, at 19:38: 8 of the 16 PVCs are not tracked by Argo CD (they come from StatefulSet &lt;code&gt;volumeClaimTemplates&lt;/code&gt;), 26 of the 44 Secrets are owned by an ExternalSecret and 18 have no owner. Of the 16 PersistentVolumes bound in those namespaces, 15 have reclaim policy &lt;code&gt;Retain&lt;/code&gt; and 1 has &lt;code&gt;Delete&lt;/code&gt; (a 5Gi backup volume).&lt;/li&gt;
&lt;li&gt;The 23 CRDs come from two sources: the Istio base chart 1.30.8, whose CRDs carried &lt;code&gt;helm.sh/resource-policy: keep&lt;/code&gt; and no sync-options, and Gateway API v1.5.1 from a kustomize remote, whose CRDs carried neither. In the listing taken at 19:11:07, 15 CRDs show &lt;code&gt;keep&lt;/code&gt; and 8 show no annotation. Custom objects existed under 7 of the 23 kinds: gatewayclasses 2, gateways.gateway.networking.k8s.io 1, httproutes 3, destinationrules 1, peerauthentications 1, telemetries 1, virtualservices 1.&lt;/li&gt;
&lt;li&gt;Prometheus had no &lt;code&gt;argocd_app_info&lt;/code&gt; series, no rule mentioning &lt;code&gt;argocd_&lt;/code&gt;, and there was no ServiceMonitor for Argo CD.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  The test
&lt;/h3&gt;

&lt;p&gt;We did not remove anything from a production directory to find out. We added three throwaway directories under &lt;code&gt;prod/&lt;/code&gt;, each a kustomization with a Namespace and one ConfigMap: &lt;code&gt;prune-test-a&lt;/code&gt; and &lt;code&gt;prune-test-c&lt;/code&gt; without any annotation, &lt;code&gt;prune-test-b&lt;/code&gt; with &lt;code&gt;argocd.argoproj.io/sync-options: Prune=false,Delete=false&lt;/code&gt; on the Namespace. The commit was pushed at 19:13:54. The ApplicationSet created the three Applications at 19:14:36, and all three were Synced and Healthy by 19:14:47, each tracking &lt;code&gt;ConfigMap/tracked-by-git&lt;/code&gt; and its Namespace, each with the finalizer. At 19:14:45 we created a Secret named &lt;code&gt;not-in-git&lt;/code&gt; by hand in each of the three namespaces, as a stand-in for objects that exist only in the cluster.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Step 1: the Namespace manifest leaves Git.&lt;/strong&gt; At 19:15:03 we pushed a commit that removes &lt;code&gt;namespace.yaml&lt;/code&gt;, and its line in &lt;code&gt;kustomization.yaml&lt;/code&gt;, from &lt;code&gt;prune-test-a&lt;/code&gt; and &lt;code&gt;prune-test-b&lt;/code&gt;. The application controller log (commit replaced by &lt;code&gt;&amp;lt;commit&amp;gt;&lt;/code&gt;):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;prune-test-a
19:17:51  Initiated automated sync to [&amp;lt;commit&amp;gt;]
19:17:51  Adding resource result, status: 'Pruned', phase: 'Succeeded', message: 'pruned'
19:18:00  Adding resource result, status: 'SyncFailed', phase: 'Failed', message: 'namespaces "prune-test-a" not found'

prune-test-b
19:18:10  Initiated automated sync to [&amp;lt;commit&amp;gt;]
19:18:10  Adding resource result, status: 'PruneSkipped', phase: 'Succeeded', message: 'ignored (no prune)'
19:18:11  Sync operation to &amp;lt;commit&amp;gt; succeeded
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;At 19:18:40:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight console"&gt;&lt;code&gt;$ kubectl get configmap,secret -n prune-test-a
No resources found &lt;span class="k"&gt;in&lt;/span&gt; prune-test-a namespace.
$ kubectl get configmap,secret -n prune-test-b
configmap/istio-ca-crl         1     4m3s
configmap/istio-ca-root-cert   1     4m3s
configmap/kube-root-ca.crt     1     4m3s
configmap/tracked-by-git       1     4m3s
secret/not-in-git   Opaque   1     3m56s
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F311e3jjbtt5crcdljkg6.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F311e3jjbtt5crcdljkg6.png" alt="Timeline from 19:14 to 19:24 UTC with one bar per test Namespace. prune-test-a (no annotation) ends at 19:17:51, pruned. prune-test-c (no annotation) ends at 19:19:17, deleted with its Application. prune-test-b (Prune=false,Delete=false) runs through both commits, with PruneSkipped at 19:18:10 and &amp;quot;Application deleted, Namespace kept&amp;quot; at 19:19:17, until it was deleted by hand at 19:23:57." width="799" height="393"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The whole test in one picture: commit 1 removed the Namespace manifest from a and b, commit 2 removed all three directories. Only the annotated Namespace was still there afterwards.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Namespace &lt;code&gt;prune-test-a&lt;/code&gt; got its &lt;code&gt;deletionTimestamp&lt;/code&gt; at 19:17:51, 2 min 48 s after the push, and was gone at the 19:18:05 sample. No webhook is configured (inferred from the controller log line "comparison expired"); the controller picked the commit up on its own refresh interval. The Secret we had made by hand went with the namespace.&lt;/p&gt;

&lt;p&gt;The Application &lt;code&gt;prune-test-a&lt;/code&gt; did not recover. It still had a ConfigMap to apply into a namespace that no longer existed (status at 19:18:40, trimmed):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="nl"&gt;"sync"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"OutOfSync"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="nl"&gt;"health"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Missing"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="nl"&gt;"message"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"one or more objects failed to apply, reason: namespaces \"prune-test-a\" not found. Retrying attempt #3 at 7:18PM."&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It retried 5 times until 19:23:10. The namespace was not recreated although the Application has &lt;code&gt;CreateNamespace=true&lt;/code&gt;. My reading is that this is because the generated Applications have no &lt;code&gt;destination.namespace&lt;/code&gt; (inference; we did not test it separately).&lt;/p&gt;

&lt;p&gt;&lt;code&gt;prune-test-b&lt;/code&gt; at 19:18:40: operation &lt;code&gt;Succeeded&lt;/code&gt;, application &lt;code&gt;OutOfSync&lt;/code&gt; / &lt;code&gt;Healthy&lt;/code&gt;, and in &lt;code&gt;status.resources&lt;/code&gt; the Namespace is &lt;code&gt;OutOfSync&lt;/code&gt; with &lt;code&gt;requiresPruning: true&lt;/code&gt;. The Namespace was &lt;code&gt;Active&lt;/code&gt; with the same uid and the same creationTimestamp (19:14:37), and both &lt;code&gt;tracked-by-git&lt;/code&gt; and &lt;code&gt;not-in-git&lt;/code&gt; were present.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Step 2: the directories are removed.&lt;/strong&gt; At 19:18:55 we pushed a commit that removes all three directories. The result is in the next section, with the code that explains it: &lt;code&gt;prune-test-c&lt;/code&gt; was deleted with its Application, and &lt;code&gt;prune-test-b&lt;/code&gt; stayed.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why
&lt;/h2&gt;

&lt;h3&gt;
  
  
  A tracked Namespace is an ordinary resource to prune
&lt;/h3&gt;

&lt;p&gt;An Application tracks a Namespace when a Namespace manifest is part of what its source renders. Ours had one in every directory, so each Application listed its own Namespace in &lt;code&gt;status.resources&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;The Argo CD documentation for automated sync says: "By default (and as a safety mechanism), automated sync will not delete resources when Argo CD detects the resource is no longer defined in Git." With &lt;code&gt;prune: true&lt;/code&gt; in the automated policy, it does. The only built-in protection left is this one: "automated sync with prune have a protection from any automation/human errors when there are no target resources." In the source that check, which reports &lt;code&gt;auto-sync will wipe out all resources&lt;/code&gt;, fires only when every resource of the Application needs pruning and &lt;code&gt;allowEmpty&lt;/code&gt; is false. If one manifest drops out and the rest still render, it does not fire. That is the case of &lt;code&gt;prune-test-a&lt;/code&gt;: the ConfigMap stayed in Git, the Namespace manifest left, and the Namespace was pruned.&lt;/p&gt;

&lt;p&gt;What is deleted together with a Namespace or a CRD is decided by Kubernetes, not by Argo CD. The Kubernetes documentation on deleting a namespace: "This deletes &lt;em&gt;everything&lt;/em&gt; under the namespace!" On deleting a CRD: "the server will uninstall the RESTful API endpoint and delete all custom objects stored in it." Everything under the namespace includes objects Argo CD never tracked, which is what happened to the hand-made Secret in &lt;code&gt;prune-test-a&lt;/code&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  Prune is stopped by one annotation only
&lt;/h3&gt;

&lt;p&gt;This is &lt;code&gt;pruneObject&lt;/code&gt; in gitops-engine at the commit that Argo CD v3.2.0 pins (&lt;code&gt;pkg/sync/sync_context.go&lt;/code&gt;):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sc&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;syncContext&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="nf"&gt;pruneObject&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;liveObj&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;unstructured&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Unstructured&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;prune&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;dryRun&lt;/span&gt; &lt;span class="kt"&gt;bool&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;common&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;ResultCode&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;string&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="n"&gt;prune&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;common&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;ResultCodePruneSkipped&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"ignored (requires pruning)"&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;resourceutil&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;HasAnnotationOption&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;liveObj&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;common&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;AnnotationSyncOptions&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;common&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;SyncOptionDisablePrune&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;common&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;ResultCodePruneSkipped&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"ignored (no prune)"&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;dryRun&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;common&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;ResultCodePruned&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"pruned (dry run)"&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;common&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;ResultCodePruned&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"pruned"&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It returns without deleting in two cases: the sync ran without prune (&lt;code&gt;ignored (requires pruning)&lt;/code&gt;), or the live object has &lt;code&gt;Prune=false&lt;/code&gt; in &lt;code&gt;argocd.argoproj.io/sync-options&lt;/code&gt; (&lt;code&gt;ignored (no prune)&lt;/code&gt;, result code &lt;code&gt;PruneSkipped&lt;/code&gt;). &lt;code&gt;helm.sh/resource-policy&lt;/code&gt; is not checked there, and neither is the kind of the object.&lt;/p&gt;

&lt;h3&gt;
  
  
  Deleting the Application is a second path, with different exemptions
&lt;/h3&gt;

&lt;p&gt;The Argo CD documentation on the finalizer: "When deleting an Application with this finalizer, the Argo CD application controller will perform a cascading delete of the Application's resources." For ApplicationSets, the documentation says generated Applications get the finalizer "if &lt;code&gt;.syncPolicy.preserveResourcesOnDeletion&lt;/code&gt; is set to false", and that &lt;code&gt;preserveResourcesOnDeletion: true&lt;/code&gt; "prevents the finalizer from being added to the Application". The source (&lt;code&gt;applicationset/utils/utils.go&lt;/code&gt; lines 272–280) adds the finalizer when the template has none and &lt;code&gt;syncPolicy&lt;/code&gt; is absent or &lt;code&gt;preserveResourcesOnDeletion&lt;/code&gt; is false. Our ApplicationSet had no &lt;code&gt;syncPolicy&lt;/code&gt;, which is why every generated Application carried the finalizer without our having written it anywhere.&lt;/p&gt;

&lt;p&gt;With a git directory generator, the Application is deleted when the generator no longer produces an Application of that name: the directory was removed, renamed or moved out of the pattern. What the cascade then deletes is decided by &lt;code&gt;shouldBeDeleted&lt;/code&gt; (&lt;code&gt;controller/appcontroller.go&lt;/code&gt; lines 1165–1170 in v3.2.0):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="c1"&gt;// shouldBeDeleted returns whether a given resource obj should be deleted on cascade delete of application app&lt;/span&gt;
&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ctrl&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="nc"&gt;ApplicationController&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="nf"&gt;shouldBeDeleted&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;app&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;appv1&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Application&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;obj&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;unstructured&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Unstructured&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="kt"&gt;bool&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="n"&gt;kube&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;IsCRD&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;obj&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="nf"&gt;isSelfReferencedApp&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;app&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;kube&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;GetObjectRef&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;obj&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;
        &lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="n"&gt;resourceutil&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;HasAnnotationOption&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;obj&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;synccommon&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;AnnotationSyncOptions&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;synccommon&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;SyncOptionDisableDeletion&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;
        &lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="n"&gt;resourceutil&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;HasAnnotationOption&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;obj&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;helm&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;ResourcePolicyAnnotation&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;helm&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;ResourcePolicyKeep&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It returns false for a CRD, for the Application's self-reference, for an object with &lt;code&gt;Delete=false&lt;/code&gt;, and for an object with &lt;code&gt;helm.sh/resource-policy: keep&lt;/code&gt;. &lt;code&gt;Prune=false&lt;/code&gt; is not in this function.&lt;/p&gt;

&lt;p&gt;Step 2 of the test, at 19:19:17 (ApplicationSet controller first, then the application controller):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;19:19:17  prune-test-c  Deleted application
19:19:17  prune-test-a  Deleted application
19:19:17  prune-test-b  Deleted application
19:19:17  prune-test-c  Deleting application's resources with Foreground propagation policy
19:19:17  prune-test-c  2 objects remaining for deletion
19:19:17  prune-test-b  Deleting application's resources with Foreground propagation policy
19:19:17  prune-test-b  1 objects remaining for deletion
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;At 19:23:48 (the &lt;code&gt;prune-test-a&lt;/code&gt; lines left out; that namespace had been deleted in step 1):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;$ kubectl get applications -n argocd &lt;span class="o"&gt;|&lt;/span&gt; grep prune-test
&lt;span class="o"&gt;(&lt;/span&gt;none&lt;span class="o"&gt;)&lt;/span&gt;
$ kubectl get ns &lt;span class="o"&gt;|&lt;/span&gt; grep prune-test
prune-test-b          Active   9m11s
$ kubectl get configmap,secret -n prune-test-c
No resources found &lt;span class="k"&gt;in&lt;/span&gt; prune-test-c namespace.
$ kubectl get configmap,secret -n prune-test-b
configmap/istio-ca-crl         1     9m11s
configmap/istio-ca-root-cert   1     9m11s
configmap/kube-root-ca.crt     1     9m11s
secret/not-in-git   Opaque   1     9m4s
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Namespace &lt;code&gt;prune-test-c&lt;/code&gt; got its &lt;code&gt;deletionTimestamp&lt;/code&gt; at 19:19:17 and was gone at the 19:19:27 sample, with its hand-made Secret. For &lt;code&gt;prune-test-b&lt;/code&gt; the controller counted 1 object for deletion, not 2: the Namespace kept the same uid, and the Secret &lt;code&gt;not-in-git&lt;/code&gt; was still in it. The ConfigMap &lt;code&gt;tracked-by-git&lt;/code&gt;, which was tracked and had no annotation, was deleted by the cascade. &lt;code&gt;Delete=false&lt;/code&gt; on the Namespace protects the Namespace and whatever Argo CD does not track inside it; it does not protect the other tracked resources of the Application. The ApplicationSet controller logged &lt;code&gt;Deleted application&lt;/code&gt; for &lt;code&gt;prune-test-a&lt;/code&gt; at 19:19:17 as well, but the Application stayed until 19:23:10: its finalizer ran only after the sync operation failed (5 retries), and it then deleted 0 resources.&lt;/p&gt;

&lt;h3&gt;
  
  
  The two paths against the four protections
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Automated prune (manifest leaves Git)&lt;/th&gt;
&lt;th&gt;Cascade (Application deleted with the finalizer)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;Prune=false&lt;/code&gt; on the object&lt;/td&gt;
&lt;td&gt;Skipped: &lt;code&gt;PruneSkipped&lt;/code&gt; / &lt;code&gt;ignored (no prune)&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Not checked: deleted&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;Delete=false&lt;/code&gt; on the object&lt;/td&gt;
&lt;td&gt;Not checked: pruned&lt;/td&gt;
&lt;td&gt;Skipped&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;helm.sh/resource-policy: keep&lt;/code&gt; on the object&lt;/td&gt;
&lt;td&gt;Not checked: pruned&lt;/td&gt;
&lt;td&gt;Skipped&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;The object is a CRD&lt;/td&gt;
&lt;td&gt;No exemption: pruned unless it has &lt;code&gt;Prune=false&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Skipped, with or without any annotation&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The table is read from the two functions above. Our test confirmed the Namespace cases with both options set together (skipped on both paths) and with neither (deleted on both paths). The conclusions that follow are inferences from the source: a CRD can be deleted by Argo CD through prune and not through the cascade; &lt;code&gt;keep&lt;/code&gt; does not stop prune; a Namespace needs &lt;code&gt;Prune=false&lt;/code&gt; against prune and &lt;code&gt;Delete=false&lt;/code&gt; against the cascade.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F1ct1x47r0s8gwk1854uh.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F1ct1x47r0s8gwk1854uh.png" alt="Diagram of two paths. Path 1: namespace.yaml leaves what Git renders, the application controller prunes, the Namespace is deleted with everything in it; stopped by Prune=false, while Delete=false, helm.sh/resource-policy: keep and the allow-empty check are not read. Path 2: the directory is removed, the ApplicationSet controller deletes the Application, the resources finalizer cascades over every tracked resource; stopped by Delete=false or helm.sh/resource-policy: keep, CRDs are always skipped, and Prune=false is not read. A Namespace needs Prune=false,Delete=false to survive both." width="799" height="521"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The two functions of the previous sections as one picture: each path reads its own option and ignores the other.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;One thing I had wrong before reading the source: I took the &lt;code&gt;helm.sh/resource-policy: keep&lt;/code&gt; on the Istio CRDs for what protects them when the Application is deleted. CRDs are skipped by the cascade with or without any annotation, so &lt;code&gt;keep&lt;/code&gt; adds nothing for a CRD on that path, and it does nothing on the prune path.&lt;/p&gt;

&lt;h2&gt;
  
  
  What to check in your cluster now
&lt;/h2&gt;

&lt;p&gt;Checks 1 to 4 are read-only and need kubectl and jq. Check 5 changes Git and the cluster.&lt;/p&gt;

&lt;h3&gt;
  
  
  1. Applications with automated prune that track a Namespace or a CRD, and whether that object is protected
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl get applications.argoproj.io -A -o json &lt;span class="o"&gt;|&lt;/span&gt; jq -r &lt;span class="s1"&gt;'
  .items[] | select(.spec.syncPolicy.automated.prune==true) | .metadata.name as $app |
  .status.resources[]? | select(.kind=="Namespace" or .kind=="CustomResourceDefinition") |
  "\($app) \(.kind) \(.name)"'&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt;
&lt;span class="k"&gt;while&lt;/span&gt; &lt;span class="nb"&gt;read&lt;/span&gt; app kind name&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt;
  &lt;span class="nv"&gt;opt&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$(&lt;/span&gt;kubectl get &lt;span class="s2"&gt;"$kind"&lt;/span&gt; &lt;span class="s2"&gt;"$name"&lt;/span&gt; -o json &lt;span class="o"&gt;|&lt;/span&gt; jq -r &lt;span class="s1"&gt;'.metadata.annotations["argocd.argoproj.io/sync-options"] // "-"'&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;
  &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"$app $kind/$name sync-options=$opt"&lt;/span&gt;
&lt;span class="k"&gt;done&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Same problem if:&lt;/strong&gt; a line ends in &lt;code&gt;sync-options=-&lt;/code&gt;, or the value has no &lt;code&gt;Prune=false&lt;/code&gt;. (&lt;code&gt;Prune=confirm&lt;/code&gt; makes the sync wait for a confirmation; we did not test it.) That Application will delete that Namespace or CRD on the next automated sync after its manifest drops out of the rendered output. Before the fix we had 31 such lines: 8 Namespaces in 8 Applications and 23 CRDs in one. A Namespace line whose value has &lt;code&gt;Prune=false&lt;/code&gt; but not &lt;code&gt;Delete=false&lt;/code&gt; is still open to check 2 (CRDs are not: the cascade skips them).&lt;/p&gt;

&lt;p&gt;No output at all means no Application with automated prune tracks a Namespace or CRD.&lt;/p&gt;

&lt;p&gt;If you do not want to repeat this by hand after every change, &lt;a href="https://github.com/KyungHwanKim-devs/kubegap" rel="noopener noreferrer"&gt;kubegap&lt;/a&gt; does it. kubegap is an open-source audit that reads the cluster your kubectl points at and writes a report of what will hurt in operation, with the evidence for each item. Its run at 16:25:29 is where this problem was found: it listed the 9 Applications. With the change we made to it afterwards ("kubegap after the fix" below) it opens each Namespace and CRD an Application tracks and lists, by name, the ones automated prune would delete and the Namespaces that would be deleted with their Application. It reads the annotations the way the installed Argo CD version does, and an object that is already protected is not listed. That covers check 1 and the Namespace part of check 2.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. Applications whose tracked resources are deleted when the Application is deleted
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl get applications.argoproj.io -A -o json &lt;span class="o"&gt;|&lt;/span&gt; jq -r &lt;span class="s1"&gt;'
  .items[] | [.metadata.name,
    ((.metadata.finalizers // []) | join(",")),
    ((.metadata.ownerReferences // []) | map(.kind+"/"+.name) | join(","))] | @tsv'&lt;/span&gt;
kubectl get applicationsets.argoproj.io -A -o json &lt;span class="o"&gt;|&lt;/span&gt; jq -r &lt;span class="s1"&gt;'
  .items[] | [.metadata.name, (.spec.syncPolicy | tostring)] | @tsv'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Same problem if:&lt;/strong&gt; an Application has &lt;code&gt;resources-finalizer.argocd.argoproj.io&lt;/code&gt; and is owned by an ApplicationSet whose &lt;code&gt;syncPolicy&lt;/code&gt; is &lt;code&gt;null&lt;/code&gt; (or has neither &lt;code&gt;preserveResourcesOnDeletion: true&lt;/code&gt; nor an &lt;code&gt;applicationsSync&lt;/code&gt; policy of &lt;code&gt;create-only&lt;/code&gt; or &lt;code&gt;create-update&lt;/code&gt;). When the generator stops producing that Application (for a git directory generator: the directory is renamed, moved or removed), the ApplicationSet controller deletes the Application and, because of the finalizer, the application controller deletes every resource the Application tracks. A Namespace it tracks goes too unless it carries &lt;code&gt;Delete=false&lt;/code&gt; (or &lt;code&gt;helm.sh/resource-policy: keep&lt;/code&gt;). CRDs are skipped by this path. Ours printed &lt;code&gt;prod-apps   null&lt;/code&gt; at 19:39, after the fix; the ApplicationSet was not changed.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. What is deleted with a Namespace and cannot be recreated from Git
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;NS&lt;/span&gt;&lt;span class="o"&gt;=&amp;lt;&lt;/span&gt;namespace&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="nv"&gt;APP&lt;/span&gt;&lt;span class="o"&gt;=&amp;lt;&lt;/span&gt;application&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;
kubectl get pvc -n &lt;span class="s2"&gt;"$NS"&lt;/span&gt; -o json &lt;span class="o"&gt;|&lt;/span&gt; jq -r &lt;span class="s1"&gt;'.items[].metadata.name'&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; sort &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; /tmp/pvc-all
kubectl get application &lt;span class="s2"&gt;"$APP"&lt;/span&gt; -n argocd -o json &lt;span class="o"&gt;|&lt;/span&gt;
  jq -r &lt;span class="s1"&gt;'.status.resources[] | select(.kind=="PersistentVolumeClaim") | .name'&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; sort &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; /tmp/pvc-tracked
comm -23 /tmp/pvc-all /tmp/pvc-tracked           &lt;span class="c"&gt;# PVCs Argo CD does not track&lt;/span&gt;
kubectl get secret -n &lt;span class="s2"&gt;"$NS"&lt;/span&gt; -o json &lt;span class="o"&gt;|&lt;/span&gt;
  jq -r &lt;span class="s1"&gt;'.items[] | select(.metadata.ownerReferences==null) | .metadata.name'&lt;/span&gt;   &lt;span class="c"&gt;# Secrets nothing would recreate&lt;/span&gt;
kubectl get pv -o json &lt;span class="o"&gt;|&lt;/span&gt; jq -r --arg ns &lt;span class="s2"&gt;"$NS"&lt;/span&gt; &lt;span class="s1"&gt;'
  .items[] | select(.spec.claimRef.namespace==$ns) |
  "\(.spec.claimRef.name) \(.spec.persistentVolumeReclaimPolicy)"'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Same problem if:&lt;/strong&gt; the first list is not empty (PVCs made from a StatefulSet's &lt;code&gt;volumeClaimTemplates&lt;/code&gt; are the usual case), or the second list has Secrets you created by hand, or a PV shows &lt;code&gt;Delete&lt;/code&gt;. A PVC is deleted with its Namespace; with reclaim policy &lt;code&gt;Delete&lt;/code&gt; the volume's data is deleted too, with &lt;code&gt;Retain&lt;/code&gt; the PersistentVolume stays and has to be re-bound by hand. Ours: 16 PVCs in the eight namespaces, 8 of them not tracked; 44 Secrets, 18 without an owner; 15 volumes &lt;code&gt;Retain&lt;/code&gt;, 1 &lt;code&gt;Delete&lt;/code&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  4. Would anyone notice an Application that stays OutOfSync?
&lt;/h3&gt;

&lt;p&gt;&lt;code&gt;Prune=false&lt;/code&gt; turns "the Namespace is deleted" into "the Application stays OutOfSync". That only helps if someone is told.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl get prometheusrules -A -o json &lt;span class="o"&gt;|&lt;/span&gt; jq -r &lt;span class="s1"&gt;'
  .items[] | (.metadata.namespace+"/"+.metadata.name) as $r | .spec.groups[].rules[] |
  select((.expr|tostring)|test("argocd_app_info")) | "\($r) \(.alert // .record)"'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;and in Prometheus: &lt;code&gt;count by (sync_status) (argocd_app_info)&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Same problem if:&lt;/strong&gt; the first command prints nothing, or the query returns no series (the application controller's metrics port, 8082, is not scraped). Both were true on our cluster before the fix.&lt;/p&gt;

&lt;h3&gt;
  
  
  5. Test it with a throwaway Application (changes Git and the cluster)
&lt;/h3&gt;

&lt;p&gt;In the repository your Applications read, add three directories, each with a Namespace manifest and one ConfigMap; give only &lt;code&gt;b&lt;/code&gt; the annotation:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Namespace&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;prune-test-b&lt;/span&gt;
  &lt;span class="na"&gt;annotations&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;argocd.argoproj.io/sync-options&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Prune=false,Delete=false&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;After the three Applications are Synced, create one object by hand in each namespace (&lt;code&gt;kubectl create secret generic not-in-git --from-literal=note=x -n prune-test-a&lt;/code&gt;), then:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Remove &lt;code&gt;namespace.yaml&lt;/code&gt; (and its line in &lt;code&gt;kustomization.yaml&lt;/code&gt;) from &lt;code&gt;a&lt;/code&gt; and &lt;code&gt;b&lt;/code&gt;, push, wait for the automated sync.
&lt;code&gt;kubectl get ns | grep prune-test&lt;/code&gt; and &lt;code&gt;kubectl get application prune-test-b -n argocd -o json | jq '.status.operationState.syncResult.resources'&lt;/code&gt;; the &lt;code&gt;Pruned&lt;/code&gt; result of &lt;code&gt;prune-test-a&lt;/code&gt; is in the controller log (&lt;code&gt;kubectl logs -n argocd sts/argocd-application-controller | grep "Adding resource result"&lt;/code&gt;), not in the Application status&lt;/li&gt;
&lt;li&gt;Remove all three directories, push, wait for the Applications to disappear.
&lt;code&gt;kubectl get ns | grep prune-test&lt;/code&gt; and &lt;code&gt;kubectl get secret -n prune-test-b&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;&lt;code&gt;kubectl delete ns prune-test-b&lt;/code&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;&lt;strong&gt;Protected if:&lt;/strong&gt; after 1, &lt;code&gt;prune-test-a&lt;/code&gt; is gone and &lt;code&gt;prune-test-b&lt;/code&gt; is &lt;code&gt;Active&lt;/code&gt; with the result &lt;code&gt;PruneSkipped&lt;/code&gt; / &lt;code&gt;ignored (no prune)&lt;/code&gt;; after 2, &lt;code&gt;prune-test-c&lt;/code&gt; is gone and &lt;code&gt;prune-test-b&lt;/code&gt; is still &lt;code&gt;Active&lt;/code&gt; with the hand-made Secret in it.&lt;/p&gt;

&lt;h2&gt;
  
  
  What we changed, and what we have not
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Changed
&lt;/h3&gt;

&lt;p&gt;Three commits, pushed at 19:24:03. (The comment in the second one first gave the CRD split as 13 and 10; a follow-up commit at 20:27:55 corrected it to the 15 and 8 shown below.)&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;1. &lt;code&gt;Prune=false,Delete=false&lt;/code&gt; on the 8 Namespace manifests.&lt;/strong&gt; The manifests stay in Git. The report offered two ways: take the Namespace manifest out of Git and rely on &lt;code&gt;CreateNamespace=true&lt;/code&gt;, or keep it and annotate. I kept it, because taking the manifest out is the very change that triggers the prune, and it would have to be preceded by the annotation in any case.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Namespace&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;prod-a&lt;/span&gt;
  &lt;span class="na"&gt;annotations&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;argocd.argoproj.io/sync-options&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Prune=false,Delete=false&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;2. &lt;code&gt;prune: false&lt;/code&gt; in the automated policy of Application &lt;code&gt;istio&lt;/code&gt;.&lt;/strong&gt; The Istio base chart 1.30.8 has no value that adds annotations to its CRDs; its CRD-related values are &lt;code&gt;base.excludedCRDs&lt;/code&gt;, &lt;code&gt;base.enableCRDTemplates&lt;/code&gt; and &lt;code&gt;base.enableIstioConfigCRDs&lt;/code&gt;. Automated prune is the only path we found in the source by which an automated sync deletes a CRD (a manual sync with force or replace was not tested), so turning prune off for the one Application that tracks them closes that path.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="c1"&gt;# argocd-apps/istio.yaml
&lt;/span&gt;&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;syncPolicy&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;automated&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="c1"&gt;# prune is off: this Application owns 23 CRDs (15 from the istio base chart, 8 Gateway API). A CRD that drops out
&lt;/span&gt;      &lt;span class="c1"&gt;# of the render would be pruned together with every object of that kind, and the base chart has no value to put
&lt;/span&gt;      &lt;span class="c1"&gt;# Prune=false on its CRDs. After a chart upgrade, delete resources the new version no longer renders by hand.
&lt;/span&gt;      &lt;span class="na"&gt;prune&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;
      &lt;span class="na"&gt;selfHeal&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;3. A scrape of the Argo CD application controller and an alert.&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;monitoring.coreos.com/v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ServiceMonitor&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;argocd-metrics&lt;/span&gt;
  &lt;span class="na"&gt;namespace&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;monitoring&lt;/span&gt;
  &lt;span class="na"&gt;labels&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;release&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;kube-prometheus-stack&lt;/span&gt;
&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;namespaceSelector&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;matchNames&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s"&gt;argocd&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
  &lt;span class="na"&gt;selector&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;matchLabels&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;app.kubernetes.io/name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;argocd-metrics&lt;/span&gt;
  &lt;span class="na"&gt;endpoints&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;port&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;metrics&lt;/span&gt;
&lt;span class="nn"&gt;---
&lt;/span&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;monitoring.coreos.com/v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;PrometheusRule&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;argocd-rules&lt;/span&gt;
  &lt;span class="na"&gt;namespace&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;monitoring&lt;/span&gt;
  &lt;span class="na"&gt;labels&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;release&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;kube-prometheus-stack&lt;/span&gt;
&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;groups&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;argocd.rules&lt;/span&gt;
      &lt;span class="na"&gt;rules&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;alert&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ArgoAppOutOfSync&lt;/span&gt;
          &lt;span class="na"&gt;expr&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;argocd_app_info{sync_status="OutOfSync", name!="root"} == 1&lt;/span&gt;
          &lt;span class="na"&gt;for&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;30m&lt;/span&gt;
          &lt;span class="na"&gt;labels&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="na"&gt;severity&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;warning&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;root&lt;/code&gt; is excluded because it is permanently OutOfSync for an unrelated reason: one Application is defined both in &lt;code&gt;argocd-apps/&lt;/code&gt; and by the ApplicationSet. That is not fixed here.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;State after.&lt;/strong&gt; At 19:31 all 8 live Namespaces carried &lt;code&gt;Prune=false,Delete=false&lt;/code&gt; and were &lt;code&gt;Active&lt;/code&gt; with their original creationTimestamp; &lt;code&gt;istio&lt;/code&gt; had &lt;code&gt;automated: {prune: false, selfHeal: true}&lt;/code&gt;; the 8 Applications and &lt;code&gt;istio&lt;/code&gt; were Synced. At 19:39:18 check 1 printed 8 lines, each ending &lt;code&gt;sync-options=Prune=false,Delete=false&lt;/code&gt;, and no CRD line. At 19:37:38, after the &lt;code&gt;monitoring&lt;/code&gt; Application had synced (its first Helm hook pod waited about 9 minutes on an image pull), &lt;code&gt;up{job="argocd-metrics"}&lt;/code&gt; was 1, &lt;code&gt;count by (sync_status) (argocd_app_info)&lt;/code&gt; returned Synced 21 and OutOfSync 1 (&lt;code&gt;root&lt;/code&gt;), and the rule &lt;code&gt;ArgoAppOutOfSync&lt;/code&gt; was loaded and inactive.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The alert, tested.&lt;/strong&gt; At 19:43:12 we pushed &lt;code&gt;prune-test-b&lt;/code&gt; again, with the annotated Namespace, and at 19:44:43 removed its &lt;code&gt;namespace.yaml&lt;/code&gt;. The alert &lt;code&gt;ArgoAppOutOfSync{name="prune-test-b"}&lt;/code&gt; became pending at 19:48:49 and fired at 20:18:49, 30 minutes later. Alertmanager shows it active from 20:18:49 with receiver &lt;code&gt;discord&lt;/code&gt;. At 20:19:02 the Application was &lt;code&gt;OutOfSync&lt;/code&gt; / &lt;code&gt;Healthy&lt;/code&gt;, the Namespace result was &lt;code&gt;PruneSkipped&lt;/code&gt; / &lt;code&gt;ignored (no prune)&lt;/code&gt;, and the Namespace was &lt;code&gt;Active&lt;/code&gt;, 34 minutes old. We then removed the directory and deleted the Namespace by hand.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fuauczh9dk0eyx6zyvf02.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fuauczh9dk0eyx6zyvf02.png" alt="Prometheus Alerts page filtered to ArgoAppOutOfSync: rule group argocd.rules, expression argocd_app_info{name!=&amp;quot;root&amp;quot;,sync_status=&amp;quot;OutOfSync&amp;quot;} == 1, for: 30m, severity warning, one alert in state FIRING with labels name=&amp;quot;prune-test-b&amp;quot;, sync_status=&amp;quot;OutOfSync&amp;quot;, health_status=&amp;quot;Healthy&amp;quot;." width="799" height="571"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Prometheus, 20:22 UTC: the alert firing for the test Application, which had been OutOfSync for more than 30 minutes because its annotated Namespace was &lt;code&gt;PruneSkipped&lt;/code&gt;. Project and repository names are replaced.&lt;/em&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  What it costs
&lt;/h3&gt;

&lt;p&gt;The Argo CD documentation says of &lt;code&gt;Prune=false&lt;/code&gt;: "The app will be out of sync if Argo CD expects a resource to be pruned." We saw two consequences.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The Application stays &lt;code&gt;OutOfSync&lt;/code&gt; until someone puts the manifest back or deletes the object by hand. A Namespace you do want removed now has to be deleted by hand, as we did with &lt;code&gt;prune-test-b&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;The controller keeps starting sync operations. In the first test &lt;code&gt;prune-test-b&lt;/code&gt; started 5 sync operations in 28 s (19:18:10, :12, :14, :20, :38), and the controller logged growing waits between them ("retrying in 1.25s", "5.65s", "17.66s", "53.60s"). In the alert test it logged "Initiated automated sync" 26 times for the Application between 19:44:20 (the initial sync) and 20:17:16.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  kubegap after the fix
&lt;/h3&gt;

&lt;p&gt;A full kubegap v0.3.0 run at 19:37:43, after our fix: its list &lt;code&gt;prune_with_namespace_or_crd&lt;/code&gt; went from 9 Applications to 8 (&lt;code&gt;istio&lt;/code&gt; was gone), and the finding &lt;code&gt;argocd.prune-manages-namespace-or-crd.cluster&lt;/code&gt; was still &lt;code&gt;fail&lt;/code&gt; for the 8 Applications whose Namespace now carried &lt;code&gt;Prune=false,Delete=false&lt;/code&gt;. Version 0.3.0 counted the kinds an Application with prune tracks and did not read the annotation (&lt;code&gt;lib/summaries/argocdctl.py&lt;/code&gt; lines 173–174 and 289). The check's own suggestion is that annotation, and a problem that has been fixed has to be gone from the next run, so this was a defect of the check. We filed it as kubegap issue 207 and fixed it in pull request 208: the summary now opens each tracked Namespace and CRD and judges it as the installed Argo CD version does, &lt;code&gt;Prune=false&lt;/code&gt; against prune, &lt;code&gt;Delete=false&lt;/code&gt; or &lt;code&gt;helm.sh/resource-policy: keep&lt;/code&gt; against the cascade, and no CRD on the cascade path.&lt;/p&gt;

&lt;p&gt;A full run with that change at 22:47:56 lists no object for prune and no Namespace for the cascade, and the finding is gone. The same code on the snapshot taken at 16:25:29, before our fix, lists 9 Applications, 31 objects for prune (the 31 lines of our check at 19:11) and 8 Namespaces for the cascade. Until a kubegap release contains pull request 208, version 0.3.0 keeps reporting an annotated Namespace; confirm it with check 1 above.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Feqdi2hom2nvbvdxbveiv.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Feqdi2hom2nvbvdxbveiv.png" alt="Three tiles. 16:25, kubegap v0.3.0: 9 Applications listed, FAIL. After the cluster fix at 19:24, 19:37, kubegap v0.3.0: 8 Applications listed, STILL FAIL, because v0.3.0 did not read the annotation. After the check was fixed in pull request 208, 22:47: 0 objects that would be deleted, FINDING GONE; on the 16:25 snapshot the same code lists 31 objects." width="799" height="293"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The same kubegap check in three runs: before the fix, after the cluster was fixed, and after the check itself was fixed.&lt;/em&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  Not changed, and what stays exposed
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;preserveResourcesOnDeletion&lt;/code&gt; on the ApplicationSet.&lt;/strong&gt; Deleting a directory still deletes that Application's tracked Deployments, Services and Git-declared PVCs. Only the Namespace and what Argo CD does not track in it survive, as the &lt;code&gt;prune-test-b&lt;/code&gt; result shows.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;Prune=confirm&lt;/code&gt; / &lt;code&gt;Delete=confirm&lt;/code&gt;.&lt;/strong&gt; Not adopted and not tested; with these values Argo CD waits for a confirmation before it prunes or deletes the object.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The &lt;code&gt;root&lt;/code&gt; Application's own prune.&lt;/strong&gt; Unchanged; &lt;code&gt;root&lt;/code&gt; still has automated &lt;code&gt;prune: true&lt;/code&gt;. It tracks Applications, and the 19:11 check printed no Namespace or CRD line for it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The other Applications.&lt;/strong&gt; 20 of the 22 Applications still have automated &lt;code&gt;prune: true&lt;/code&gt;. The annotation protects the 8 Namespaces, not the Deployments, Services or PVCs those Applications declare.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Limits
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;A manual sync with force or replace was not tested. A record in the Incident Library, a public collection of incident write-ups (IL-0058, from GitHub issue argoproj/argo-cd#28630, which we did not open; it is the third case under "Public cases where Argo CD deleted resources" below), reports that &lt;code&gt;Prune=false&lt;/code&gt; did not stop a force sync from deleting and recreating a Namespace. Our fix does not cover that path.&lt;/li&gt;
&lt;li&gt;Pruning of a CRD was not reproduced. What we say about CRDs is from the Argo CD source and the Kubernetes documentation.&lt;/li&gt;
&lt;li&gt;Whether &lt;code&gt;argocd app sync --prune&lt;/code&gt; on v3.2.0 exits with &lt;code&gt;resources require pruning&lt;/code&gt; on a &lt;code&gt;Prune=false&lt;/code&gt; resource was not tested.&lt;/li&gt;
&lt;li&gt;No Argo CD web UI screen was captured, so the result texts in this article are from the Application status and the controller log, not from the UI panel.&lt;/li&gt;
&lt;li&gt;Whether the Discord message arrived was not looked at. What is confirmed is that Alertmanager holds the alert as active for that receiver.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Public cases where Argo CD deleted resources
&lt;/h2&gt;

&lt;p&gt;The Incident Library is a searchable collection of public post-incident reports. Each case is classified by symptom, component and cause, so you can look up who else had the failure you are looking at and what they changed afterwards. Filtered to the component Argo CD and the symptom "resources deleted" it has three cases, each of them Argo CD deleting resources after a configuration change. The summaries below are the library's records; of the three linked sources we opened the InfluxData post. None of them is a Namespace manifest dropping out of a kustomize render; the triggers differ.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;InfluxData, 2022-06-24:&lt;/strong&gt; a pull request gave a new Argo CD Application the name of the existing production one; Argo CD replaced it and deleted the InfluxDB Cloud production environment. Service was back in just under six hours and no data was lost. Afterwards the team put &lt;code&gt;Prune=false&lt;/code&gt; on its stateful resources and on the Namespaces Argo CD manages, set every volume to retain its data, and turned on &lt;code&gt;FailOnSharedResource=true&lt;/code&gt; (&lt;a href="https://www.influxdata.com/blog/deleting-production-steps-how-fix-it/" rel="noopener noreferrer"&gt;their write-up&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Skyscanner, 2021-08-25:&lt;/strong&gt; a templating change that dropped its &lt;code&gt;{{ }}&lt;/code&gt; delimiters made Argo CD delete all 478 services in every region (&lt;a href="https://medium.com/@SkyscannerEng/how-a-couple-of-characters-brought-down-our-site-356ccaf1fbc3" rel="noopener noreferrer"&gt;their write-up&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;One Argo CD user, GitHub issue, 2026-07-08:&lt;/strong&gt; on Argo CD v3.3.9, namespaces declared by several Applications were deleted by a manual sync with force, and about 97 Applications were stuck Missing and OutOfSync; &lt;code&gt;Prune=false&lt;/code&gt; did not stop the force replace (&lt;a href="https://github.com/argoproj/argo-cd/issues/28630" rel="noopener noreferrer"&gt;argoproj/argo-cd#28630&lt;/a&gt;).&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;These are the set &lt;a href="https://benchlet.com/incidentlib/app?component=argocd&amp;amp;symptom=resources-deleted" rel="noopener noreferrer"&gt;Argo CD + resources deleted, 3 cases&lt;/a&gt;. The same outcome from other tools and from commands run by hand, for example a Crossplane change and a namespace deleted on the wrong kubeconfig context, is in &lt;a href="https://benchlet.com/incidentlib/app?symptom=resources-deleted" rel="noopener noreferrer"&gt;resources deleted, 18 cases&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Timeline
&lt;/h2&gt;

&lt;p&gt;All times UTC, 2026-11-14.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Time&lt;/th&gt;
&lt;th&gt;Event&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;16:25:29&lt;/td&gt;
&lt;td&gt;kubegap v0.3.0 full run reports problem 3: 9 Applications with automated prune track a Namespace or CRDs.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;19:11&lt;/td&gt;
&lt;td&gt;Check run by hand: 31 lines (8 Namespaces, 23 CRDs). No sync-options annotation on any of the 8 Namespaces. No Argo CD metrics in Prometheus.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;19:13:54&lt;/td&gt;
&lt;td&gt;Test: three throwaway directories pushed (&lt;code&gt;prune-test-a&lt;/code&gt;, &lt;code&gt;-b&lt;/code&gt;, &lt;code&gt;-c&lt;/code&gt;; only &lt;code&gt;b&lt;/code&gt; annotated).&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;19:14:36–47&lt;/td&gt;
&lt;td&gt;ApplicationSet creates the three Applications; all Synced / Healthy.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;19:14:45&lt;/td&gt;
&lt;td&gt;Secret &lt;code&gt;not-in-git&lt;/code&gt; created by hand in each test namespace.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;19:15:03&lt;/td&gt;
&lt;td&gt;Commit removes &lt;code&gt;namespace.yaml&lt;/code&gt; from &lt;code&gt;prune-test-a&lt;/code&gt; and &lt;code&gt;prune-test-b&lt;/code&gt;.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;19:17:51&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;prune-test-a&lt;/code&gt;: automated sync, Namespace &lt;code&gt;Pruned&lt;/code&gt; / &lt;code&gt;pruned&lt;/code&gt;. Gone at the 19:18:05 sample.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;19:18:00&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;prune-test-a&lt;/code&gt;: ConfigMap fails with &lt;code&gt;namespaces "prune-test-a" not found&lt;/code&gt;; retries until 19:23:10.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;19:18:10&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;prune-test-b&lt;/code&gt;: automated sync, Namespace &lt;code&gt;PruneSkipped&lt;/code&gt; / &lt;code&gt;ignored (no prune)&lt;/code&gt;; Application OutOfSync / Healthy.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;19:18:10–38&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;prune-test-b&lt;/code&gt; starts 5 sync operations in 28 s.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;19:18:55&lt;/td&gt;
&lt;td&gt;Commit removes all three directories.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;19:19:17&lt;/td&gt;
&lt;td&gt;ApplicationSet deletes the Applications; cascade deletes Namespace &lt;code&gt;prune-test-c&lt;/code&gt;. &lt;code&gt;prune-test-b&lt;/code&gt; stays.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;19:23:10&lt;/td&gt;
&lt;td&gt;Application &lt;code&gt;prune-test-a&lt;/code&gt; deleted after its sync operation failed (5 retries).&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;19:23:48&lt;/td&gt;
&lt;td&gt;Only Namespace &lt;code&gt;prune-test-b&lt;/code&gt; is left, &lt;code&gt;Active&lt;/code&gt;, with &lt;code&gt;not-in-git&lt;/code&gt;.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;19:23:57&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;prune-test-b&lt;/code&gt; deleted by hand.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;19:24:03&lt;/td&gt;
&lt;td&gt;Fix pushed: annotation on 8 Namespace manifests, &lt;code&gt;prune: false&lt;/code&gt; on &lt;code&gt;istio&lt;/code&gt;, ServiceMonitor and alert rule.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;19:31&lt;/td&gt;
&lt;td&gt;8 live Namespaces carry &lt;code&gt;Prune=false,Delete=false&lt;/code&gt;; &lt;code&gt;istio&lt;/code&gt; has prune off; all 9 Synced.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;19:37:38&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;argocd_app_info&lt;/code&gt; scraped: Synced 21, OutOfSync 1 (&lt;code&gt;root&lt;/code&gt;). Rule &lt;code&gt;ArgoAppOutOfSync&lt;/code&gt; loaded.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;19:37:43&lt;/td&gt;
&lt;td&gt;kubegap v0.3.0 rerun: 9 Applications become 8; the finding is still &lt;code&gt;fail&lt;/code&gt; for the 8 annotated Namespaces.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;19:39:18&lt;/td&gt;
&lt;td&gt;Check 1 prints 8 lines, all &lt;code&gt;sync-options=Prune=false,Delete=false&lt;/code&gt;, no CRD line.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;19:43:12&lt;/td&gt;
&lt;td&gt;Alert test: &lt;code&gt;prune-test-b&lt;/code&gt; pushed again.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;19:44:43&lt;/td&gt;
&lt;td&gt;Alert test: its &lt;code&gt;namespace.yaml&lt;/code&gt; removed.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;19:48:49&lt;/td&gt;
&lt;td&gt;Alert &lt;code&gt;ArgoAppOutOfSync{name="prune-test-b"}&lt;/code&gt; pending.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;20:18:49&lt;/td&gt;
&lt;td&gt;Alert firing; active in Alertmanager for receiver &lt;code&gt;discord&lt;/code&gt;.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;20:22:55&lt;/td&gt;
&lt;td&gt;Prometheus Alerts page captured. After this the test directory was removed and the Namespace deleted by hand.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;20:27:55&lt;/td&gt;
&lt;td&gt;Follow-up commit corrects the CRD counts in the &lt;code&gt;istio&lt;/code&gt; comment (15 and 8).&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;22:47:56&lt;/td&gt;
&lt;td&gt;kubegap rerun with the fix of its check (pull request 208): no object listed, the finding is gone.&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  FAQ
&lt;/h2&gt;

&lt;h3&gt;
  
  
  argocd deleted namespace / argocd prune deleted my namespace: argocd prevent namespace deletion
&lt;/h3&gt;

&lt;p&gt;Put &lt;code&gt;argocd.argoproj.io/sync-options: Prune=false,Delete=false&lt;/code&gt; on the Namespace manifest. &lt;code&gt;Prune=false&lt;/code&gt; makes an automated sync report &lt;code&gt;PruneSkipped&lt;/code&gt; / &lt;code&gt;ignored (no prune)&lt;/code&gt; when the manifest leaves Git. &lt;code&gt;Delete=false&lt;/code&gt; keeps the Namespace when its Application is deleted with the resources finalizer. In our test the annotated Namespace survived both, and the two unannotated ones were deleted.&lt;/p&gt;

&lt;h3&gt;
  
  
  argocd applicationset deleted application and resources
&lt;/h3&gt;

&lt;p&gt;The ApplicationSet controller adds &lt;code&gt;resources-finalizer.argocd.argoproj.io&lt;/code&gt; to generated Applications unless &lt;code&gt;preserveResourcesOnDeletion&lt;/code&gt; is true. When a git directory disappears, it deletes the Application, and the application controller then deletes every tracked resource because of the finalizer. &lt;code&gt;Delete=false&lt;/code&gt; or &lt;code&gt;helm.sh/resource-policy: keep&lt;/code&gt; on an object exempts that object; CRDs are always skipped. In our test only the annotated Namespace remained.&lt;/p&gt;

&lt;h3&gt;
  
  
  argocd CRD deleted prune
&lt;/h3&gt;

&lt;p&gt;In Argo CD v3.2.0, prune skips an object only when it carries &lt;code&gt;Prune=false&lt;/code&gt;; &lt;code&gt;helm.sh/resource-policy: keep&lt;/code&gt; is not read on that path. The cascade on Application deletion skips CRDs, and prune does not. If a chart cannot annotate its CRDs, set &lt;code&gt;prune: false&lt;/code&gt; on that Application. We did not reproduce a CRD prune; this is from the source.&lt;/p&gt;

&lt;h3&gt;
  
  
  auto-sync will wipe out all resources
&lt;/h3&gt;

&lt;p&gt;That message is the allow-empty protection: automated sync with prune refuses to run when every resource of the Application would be pruned and &lt;code&gt;allowEmpty&lt;/code&gt; is not set. It does not fire when one manifest is missing and the rest still render. In our test a Namespace was pruned while its ConfigMap stayed in Git.&lt;/p&gt;

&lt;h3&gt;
  
  
  argocd_app_info{sync_status="OutOfSync"} alert
&lt;/h3&gt;

&lt;p&gt;Scrape the application controller's metrics port (8082) and alert on &lt;code&gt;argocd_app_info{sync_status="OutOfSync"} == 1&lt;/code&gt; with &lt;code&gt;for: 30m&lt;/code&gt;. A resource that &lt;code&gt;Prune=false&lt;/code&gt; keeps from being pruned leaves its Application OutOfSync, so this alert is what tells a person that a deletion was skipped. In our test the alert became pending at 19:48:49 and fired at 20:18:49.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://benchlet.com/guides/argocd-auto-prune-namespace-crd-prune-false-delete-false/" rel="noopener noreferrer"&gt;benchlet.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>kubernetes</category>
      <category>argocd</category>
      <category>gitops</category>
      <category>sre</category>
    </item>
    <item>
      <title>Pods stuck Pending after node failure: why didn't Kubernetes reschedule them to the other node?</title>
      <dc:creator>KYUNGHWAN KIM</dc:creator>
      <pubDate>Fri, 13 Nov 2026 04:08:47 +0000</pubDate>
      <link>https://dev.to/kkhdevs/pods-stuck-pending-after-node-failure-why-didnt-kubernetes-reschedule-them-to-the-other-node-5688</link>
      <guid>https://dev.to/kkhdevs/pods-stuck-pending-after-node-failure-why-didnt-kubernetes-reschedule-them-to-the-other-node-5688</guid>
      <description>&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fta4b6qiju37bw0epbhn6.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fta4b6qiju37bw0epbhn6.png" alt="Scheduler message &amp;quot;0/2 nodes are available: 1 node(s) didn't match Pod's node affinity/selector, 1 node(s) were unschedulable.&amp;quot; above the sentence &amp;quot;15 app Deployments could start on only one of our two Kubernetes nodes.&amp;quot; and the line &amp;quot;The message is from a pod pinned on purpose, during our drain test. The 15 apps had the same kind of rule; after the fix they were back on the other node in 2 min 22 s.&amp;quot;" width="800" height="420"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;If your pods are not rescheduled when a node goes down and stay stuck Pending after node failure, run &lt;code&gt;kubectl describe pod&lt;/code&gt; and read the &lt;code&gt;FailedScheduling&lt;/code&gt; event. After a drain it reads &lt;code&gt;0/2 nodes are available: 1 node(s) didn't match Pod's node affinity/selector, 1 node(s) were unschedulable.&lt;/code&gt;; after a node dies, the second item is &lt;code&gt;1 node(s) had untolerated taint {node.kubernetes.io/unreachable: }&lt;/code&gt; instead (&lt;code&gt;1 node(s) had untolerated taint(s)&lt;/code&gt; on Kubernetes 1.35 and later). Either way, the pod has been recreated, and no node satisfies its own placement rules. On our cluster 15 stateless app Deployments carried a required node affinity, &lt;code&gt;role NotIn [observability]&lt;/code&gt;, and the &lt;code&gt;role=observability&lt;/code&gt; label is on the only other node, so each of them could run on exactly one node. Nine of them also had topology spread rules that kept two defaults: &lt;code&gt;nodeTaintsPolicy&lt;/code&gt; (Ignore), which can leave the second replica Pending with &lt;code&gt;didn't match pod topology spread constraints&lt;/code&gt; once a node is gone, and &lt;code&gt;matchLabelKeys&lt;/code&gt; (empty), which during an ordinary rolling update moved seven single-replica apps to the other node. We changed the rule from &lt;code&gt;requiredDuringSchedulingIgnoredDuringExecution&lt;/code&gt; to &lt;code&gt;preferredDuringSchedulingIgnoredDuringExecution&lt;/code&gt; with weight 100 and added &lt;code&gt;nodeTaintsPolicy: Honor&lt;/code&gt; and &lt;code&gt;matchLabelKeys: [pod-template-hash]&lt;/code&gt; to the spread rules; below are the scheduler output, the Kubernetes documents that explain it, read-only checks with the value that means you have the same problem, and the drain test that confirmed the fix. The cluster is a 2-node k3s cluster (k3s v1.33.5+k3s1, one server and one agent); all times are UTC, 2026-11-13 unless a date is given.&lt;/p&gt;

&lt;h2&gt;
  
  
  What we saw: &lt;code&gt;0/2 nodes are available: 1 node(s) didn't match Pod's node affinity/selector&lt;/code&gt;
&lt;/h2&gt;

&lt;p&gt;The two nodes, both Ready, no taints:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Node&lt;/th&gt;
&lt;th&gt;k3s role&lt;/th&gt;
&lt;th&gt;CPU&lt;/th&gt;
&lt;th&gt;Memory&lt;/th&gt;
&lt;th&gt;GPUs&lt;/th&gt;
&lt;th&gt;Labels&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;server-1&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;the only server: control plane, embedded etcd&lt;/td&gt;
&lt;td&gt;12&lt;/td&gt;
&lt;td&gt;32207388Ki&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;role=observability&lt;/code&gt;, &lt;code&gt;storage=server-1&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;agent-1&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;agent&lt;/td&gt;
&lt;td&gt;6&lt;/td&gt;
&lt;td&gt;32802564Ki&lt;/td&gt;
&lt;td&gt;4 &lt;code&gt;nvidia.com/gpu&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;none of these&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The finding came from a full run of kubegap, an open-source audit of Kubernetes cluster operations, on 2026-11-11 at 05:08:33. Its problem 5, translated from Korean: "The two nodes cannot stand in for each other. The apps are tied to agent-1, and the databases, the registry and the observability stack to server-1, by labels and local-path volumes. server-1 is the only k3s server and etcd, so losing server-1 also stops the API server." It counted 18 workloads that could run only on agent-1 and 24 only on server-1, CoreDNS with 1 replica on server-1 and no PodDisruptionBudget, and both ingress-nginx replicas on server-1.&lt;/p&gt;

&lt;p&gt;On 2026-11-13 at 02:25 we listed the node rules of every Deployment and StatefulSet. Fifteen running app Deployments, in four production namespaces, had the same required rule (excerpt, names masked):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;$ kubectl get deploy,sts -A -o json &lt;span class="o"&gt;|&lt;/span&gt; jq …   &lt;span class="c"&gt;# a variant of check 1's jq program that also prints replicas and PVC claims; check 1 prints the rule as nodeSelector=… required=…&lt;/span&gt;
Deployment   prod-f/signal-f     1  role NotIn observability
Deployment   prod-f/web-f        1  role NotIn observability
Deployment   prod-b/api-b        2  role NotIn observability
Deployment   prod-b/admin-b      1  role NotIn observability
...
Deployment   prod-d/api-d        2  role NotIn observability
Deployment   prod-e/web-e        1  role NotIn observability
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The same rule written as a label selector:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ kubectl get nodes -l 'role notin (observability)'
NAME      STATUS   ROLES    AGE    VERSION
agent-1   Ready    &amp;lt;none&amp;gt;   347d   v1.33.5+k3s1
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;One node. Nine of the 15 Deployments also had this spread rule, with no &lt;code&gt;nodeTaintsPolicy&lt;/code&gt; and no &lt;code&gt;matchLabelKeys&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;topologySpreadConstraints&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;maxSkew&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
    &lt;span class="na"&gt;topologyKey&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;kubernetes.io/hostname&lt;/span&gt;
    &lt;span class="na"&gt;whenUnsatisfiable&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;DoNotSchedule&lt;/span&gt;
    &lt;span class="na"&gt;labelSelector&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;matchLabels&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="na"&gt;app&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;api-d&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Three of them ran 2 replicas at 02:27, and all six of those pods were on &lt;code&gt;agent-1&lt;/code&gt;: &lt;code&gt;api-b&lt;/code&gt;, &lt;code&gt;api-d&lt;/code&gt; and an admin app that its HorizontalPodAutoscaler later scaled to 1. At 02:35 there were 38 running pods (DaemonSets excluded) on &lt;code&gt;agent-1&lt;/code&gt; and 59 on &lt;code&gt;server-1&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;We fixed the rules first (next section) and then drained &lt;code&gt;agent-1&lt;/code&gt; at 02:51:48 to test the fix. So the scheduler never wrote the message at the top of this page for the 15 app Deployments. The scheduler wrote it during the drain for pods that are tied to &lt;code&gt;agent-1&lt;/code&gt; on purpose: a TURN server on hostNetwork, a GPU embedding service and a vector database, each with a &lt;code&gt;kubernetes.io/hostname&lt;/code&gt; selector. With the old rule the scheduler would have given the 15 apps' pods the same &lt;code&gt;didn't match Pod's node affinity/selector&lt;/code&gt; reason for &lt;code&gt;server-1&lt;/code&gt;, because the rule matches only &lt;code&gt;agent-1&lt;/code&gt; (inference from the rule and the selector output above; we did not run the drain before the fix).&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fboxc0qik8ruodgz7fqm4.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fboxc0qik8ruodgz7fqm4.png" alt="Terminal output of kubectl get pods with status.phase=Pending and the PodScheduled message for eight masked pods: four &amp;quot;didn't match PersistentVolume's node affinity&amp;quot;, three &amp;quot;didn't match Pod's node affinity/selector&amp;quot;, one &amp;quot;Insufficient nvidia.com/gpu&amp;quot;, each with &amp;quot;1 node(s) were unschedulable&amp;quot;." width="800" height="581"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;During the drain at 02:54 UTC: the eight pods that had nowhere else to go, each with the scheduler's reason. No other pod was Pending.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;At 02:54:12 these eight were the only Pending pods: four pods with local-path volumes on &lt;code&gt;agent-1&lt;/code&gt; (&lt;code&gt;didn't match PersistentVolume's node affinity&lt;/code&gt;), the three pods above with hostname selectors, and the second GPU workload (&lt;code&gt;Insufficient nvidia.com/gpu&lt;/code&gt;; &lt;code&gt;server-1&lt;/code&gt; has no GPU). &lt;code&gt;server-1&lt;/code&gt; was running 89 non-DaemonSet pods and &lt;code&gt;agent-1&lt;/code&gt; none. One of the eight lines, unmasked except for the pod name:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;0/2 nodes are available: 1 node(s) didn't match Pod's node affinity/selector, 1 node(s) were unschedulable. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fqfaoggd5cgulxpoae85y.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fqfaoggd5cgulxpoae85y.png" alt="Prometheus graph, 02:35 to 03:10 UTC, of running non-DaemonSet pods per node: agent-1 falls from about 36 to 0 at 02:52 and climbs back after 02:55; server-1 rises to 89 and comes back down." width="799" height="571"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Prometheus, running pods per node (DaemonSets excluded), 02:35–03:10 UTC. agent-1 drops to 0 at the drain and refills after uncordon and the rollout restarts.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Why
&lt;/h2&gt;

&lt;h3&gt;
  
  
  A required &lt;code&gt;NotIn&lt;/code&gt; rule is satisfied every day and fails on the day the node is lost
&lt;/h3&gt;

&lt;p&gt;The Kubernetes documentation for node affinity:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;requiredDuringSchedulingIgnoredDuringExecution&lt;/code&gt;: "The scheduler can't schedule the Pod unless the rule is met."&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;preferredDuringSchedulingIgnoredDuringExecution&lt;/code&gt;: "If a matching node is not available, the scheduler still schedules the Pod." Each preferred term has a weight from 1 to 100.&lt;/li&gt;
&lt;li&gt;"NotIn and DoesNotExist allow you to define node anti-affinity behavior."&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;code&gt;role NotIn [observability]&lt;/code&gt; reads as "anywhere except the observability node". With two nodes, it means "only &lt;code&gt;agent-1&lt;/code&gt;". &lt;code&gt;IgnoredDuringExecution&lt;/code&gt; means running pods are never checked again, so the rule does nothing while &lt;code&gt;agent-1&lt;/code&gt; is up. It is evaluated only when a pod is scheduled, so it fails only when a pod has to be scheduled while &lt;code&gt;agent-1&lt;/code&gt; cannot take it: when &lt;code&gt;agent-1&lt;/code&gt; has failed, is drained or cordoned, or has no room left.&lt;/p&gt;

&lt;p&gt;The scheduler runs its filters in a fixed order (NodeUnschedulable, NodeName, TaintToleration, NodeAffinity, …, PodTopologySpread, from &lt;code&gt;default_plugins.go&lt;/code&gt; in the Kubernetes source), and a node is rejected by the first filter that fails. A drained node therefore reports &lt;code&gt;were unschedulable&lt;/code&gt;. On the surviving node only the node affinity failure is reported; a topology spread failure on the same node is never evaluated, so it does not appear in the message.&lt;/p&gt;

&lt;p&gt;For a node that dies instead of being drained, the second item is a taint, not &lt;code&gt;were unschedulable&lt;/code&gt;. Assembled from the Kubernetes source, not observed by us: up to Kubernetes 1.34 the line reads &lt;code&gt;1 node(s) had untolerated taint {node.kubernetes.io/unreachable: }&lt;/code&gt;, and from 1.35 it reads &lt;code&gt;1 node(s) had untolerated taint(s)&lt;/code&gt; without the key.&lt;/p&gt;

&lt;h3&gt;
  
  
  &lt;code&gt;nodeTaintsPolicy&lt;/code&gt; defaults to Ignore
&lt;/h3&gt;

&lt;p&gt;The Kubernetes documentation for topology spread: "If this value is null, the behavior is equivalent to the Ignore policy." With Honor, "nodes without taints, along with tainted nodes for which the incoming pod has a toleration, are included." The field is GA in 1.33. The Kubernetes blog describes the failure: "This may lead to a node with untolerated taint as the only candidate for spreading, and as a result, the pod will stuck in Pending if it doesn't tolerate the taint." and "For backwards compatibility, &lt;code&gt;nodeTaintsPolicy&lt;/code&gt; defaults to &lt;code&gt;Ignore&lt;/code&gt;."&lt;/p&gt;

&lt;p&gt;What this means on two nodes once the node rule is preferred, as we reasoned it before the test (not reproduced; we added Honor before draining): with Ignore, an unreachable or cordoned &lt;code&gt;agent-1&lt;/code&gt; still counts as a spreading domain, and it holds 0 matching pods, because the spread count skips pods that are terminating (&lt;code&gt;countPodsMatchSelector&lt;/code&gt; in the Kubernetes source). The first replacement replica goes to &lt;code&gt;server-1&lt;/code&gt;. The second would make the skew 2 − 0 = 2, more than &lt;code&gt;maxSkew: 1&lt;/code&gt;, so with &lt;code&gt;DoNotSchedule&lt;/code&gt; it stays Pending with &lt;code&gt;didn't match pod topology spread constraints&lt;/code&gt;. A 2-replica Deployment would run at half its replicas for as long as &lt;code&gt;agent-1&lt;/code&gt; is gone.&lt;/p&gt;

&lt;h3&gt;
  
  
  Without &lt;code&gt;matchLabelKeys&lt;/code&gt;, every rolling update counts the old pod
&lt;/h3&gt;

&lt;p&gt;This part we observed. Our first fix changed the node rule to preferred and added Honor. Argo CD had applied it by 02:39:23, which started a rolling update of all 15 Deployments. The new pod of seven single-replica apps, and one new replica each of &lt;code&gt;api-b&lt;/code&gt; and &lt;code&gt;api-d&lt;/code&gt;, went to &lt;code&gt;server-1&lt;/code&gt;, although the new rule prefers &lt;code&gt;agent-1&lt;/code&gt; with weight 100.&lt;/p&gt;

&lt;p&gt;The reason (inference from the spread rule and the documentation below): the &lt;code&gt;labelSelector&lt;/code&gt; matches pods of every revision, so during the update the old pod on &lt;code&gt;agent-1&lt;/code&gt; is still counted. A new pod on &lt;code&gt;agent-1&lt;/code&gt; would make the skew 2 − 0 = 2; on &lt;code&gt;server-1&lt;/code&gt; it makes 1 − 1 = 0. &lt;code&gt;DoNotSchedule&lt;/code&gt; is a filter and the preferred node affinity is only a score, so &lt;code&gt;agent-1&lt;/code&gt; is removed before scores are compared and the weight of 100 never applies to it.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fet1abm9cyhnyuvxsipf2.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fet1abm9cyhnyuvxsipf2.png" alt="Terminal table of pods after the first rollout with NODE and IMAGE PULL ON server-1 columns: seven single-replica pods in red on server-1 with pull times from 2m25.179s to 5m49.087s and sizes from 55 MB to 844 MB, and one replica each of api-b and api-d on server-1 with pulls of 6m13.859s and 4m0.693s." width="800" height="560"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;After the first fix, an ordinary rollout put seven single-replica apps on server-1. Their images had never been there: 2 min 25 s to 5 min 49 s to pull.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Those images had never been pulled on &lt;code&gt;server-1&lt;/code&gt;. There were nine pulls on &lt;code&gt;server-1&lt;/code&gt; from the in-cluster registry between 02:41:44 and 02:45:35, from 2m25.179s (843,669,709 bytes) to 6m13.859s (835,296,653 bytes); a 94,655,960-byte image took 5m49.087s.&lt;/p&gt;

&lt;p&gt;The Kubernetes documentation for &lt;code&gt;matchLabelKeys&lt;/code&gt;: with &lt;code&gt;pod-template-hash&lt;/code&gt;, "which is added automatically by the Deployment controller", the rule can "distinguish between different revisions in a single Deployment". The field is beta and enabled by default since 1.27. With &lt;code&gt;matchLabelKeys: [pod-template-hash]&lt;/code&gt; the old revision's pod is not counted, so a single-replica app's new pod can go back to the preferred node.&lt;/p&gt;

&lt;h2&gt;
  
  
  What to check in your cluster now
&lt;/h2&gt;

&lt;p&gt;Checks 1 to 4 only read. They need &lt;code&gt;kubectl&lt;/code&gt; and &lt;code&gt;jq&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;1. Workloads whose required rules leave exactly one node.&lt;/strong&gt; A required rule (&lt;code&gt;nodeSelector&lt;/code&gt; or &lt;code&gt;requiredDuringSchedulingIgnoredDuringExecution&lt;/code&gt;) is checked against labels that exist today. List every required node rule with the first command, then rewrite each one as a selector for &lt;code&gt;kubectl get nodes -l&lt;/code&gt; (&lt;code&gt;key=value&lt;/code&gt; for &lt;code&gt;nodeSelector&lt;/code&gt;, &lt;code&gt;key in (a,b)&lt;/code&gt; for &lt;code&gt;In&lt;/code&gt;, &lt;code&gt;key notin (a)&lt;/code&gt; for &lt;code&gt;NotIn&lt;/code&gt;, &lt;code&gt;key&lt;/code&gt; for &lt;code&gt;Exists&lt;/code&gt;, &lt;code&gt;!key&lt;/code&gt; for &lt;code&gt;DoesNotExist&lt;/code&gt;, a comma for &lt;code&gt;AND&lt;/code&gt;, one command per &lt;code&gt;OR&lt;/code&gt; term) and count the nodes it prints:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl get deploy,sts -A -o json &lt;span class="o"&gt;|&lt;/span&gt; jq -r &lt;span class="s1"&gt;'.items[] | .spec.template.spec as $s
  | select($s.nodeSelector or $s.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution)
  | "\(.kind) \(.metadata.namespace)/\(.metadata.name) nodeSelector=\($s.nodeSelector // {} | tojson) required=\($s.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms // [] | map(.matchExpressions // [] | map("\(.key) \(.operator) \(.values // [] | join(","))") | join(" AND ")) | join(" OR "))"'&lt;/span&gt;
kubectl get nodes -l &lt;span class="s1"&gt;'role notin (observability)'&lt;/span&gt;   &lt;span class="c"&gt;# our rule; use yours&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For our old rule, &lt;code&gt;kubectl get nodes -l&lt;/code&gt; printed one node. Same problem if a stateless Deployment (no local volume, no GPU, no hostNetwork) matches one node: that Deployment cannot start anywhere while that node is down. Workloads pinned on purpose (a database on a local volume, a GPU job) show up here too; for them the answer is a restore, not a reschedule. After our fix, the 27 workloads that still match one node are databases, the registry, observability, a GPU job and a TURN server.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Spread rules that block the second replica, or push single replicas off their node.&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl get deploy -A -o json &lt;span class="o"&gt;|&lt;/span&gt; jq -r &lt;span class="s1"&gt;'
  .items[] | (.metadata.namespace+"/"+.metadata.name) as $n | .spec.replicas as $r |
  .spec.template.spec.topologySpreadConstraints[]? | select(.whenUnsatisfiable=="DoNotSchedule") |
  "\($n) replicas=\($r) key=\(.topologyKey) nodeTaintsPolicy=\(.nodeTaintsPolicy // "Ignore(default)") matchLabelKeys=\((.matchLabelKeys // [])|join(","))"'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Same problem if:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;key=kubernetes.io/hostname&lt;/code&gt;, &lt;code&gt;nodeTaintsPolicy=Ignore(default)&lt;/code&gt; and &lt;code&gt;replicas&lt;/code&gt; 2 or more: when one node is cordoned or unreachable it still counts as a place with 0 replicas, so with &lt;code&gt;maxSkew: 1&lt;/code&gt; only one replica can start on the remaining node.&lt;/li&gt;
&lt;li&gt;the same rule with an empty &lt;code&gt;matchLabelKeys&lt;/code&gt; on a Deployment that should stay on one preferred node: during every rolling update the old pod is still counted, and the new pod is forced onto the other node.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;After our fix all 9 app rules print &lt;code&gt;nodeTaintsPolicy=Honor matchLabelKeys=pod-template-hash&lt;/code&gt;. The k3s packaged CoreDNS Deployment still prints a hostname &lt;code&gt;DoNotSchedule&lt;/code&gt; rule with &lt;code&gt;Ignore(default)&lt;/code&gt;. With 1 replica it does not meet the first condition, and we left it as it is (see "Not changed, and why").&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Volumes that tie pods to one node.&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl get pv -o json &lt;span class="o"&gt;|&lt;/span&gt; jq -r &lt;span class="s1"&gt;'.items[] | select(.spec.nodeAffinity.required) |
  "\(.spec.claimRef.namespace)/\(.spec.claimRef.name) \(.spec.storageClassName) \([.spec.nodeAffinity.required.nodeSelectorTerms[].matchExpressions[].values[]]|join(","))"'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Same problem if a production database's PV is listed with one node. Its pod will report &lt;code&gt;didn't match PersistentVolume's node affinity&lt;/code&gt; on every other node. Moving it needs a restore from a copy kept elsewhere, or replicated storage. Our local-path PVs: 24 on &lt;code&gt;server-1&lt;/code&gt;, 14 on &lt;code&gt;agent-1&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. Is there room for the pods that would move?&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl describe nodes &lt;span class="o"&gt;|&lt;/span&gt; grep -E &lt;span class="s1"&gt;'^Name:|^Allocatable:|^Allocated resources:|^  (cpu|memory)[: ]'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Add the &lt;code&gt;Allocated resources&lt;/code&gt; requests of the node you might lose to those of the other node, and compare the sum with the other node's &lt;code&gt;Allocatable&lt;/code&gt;. This overstates the load that would move, because it includes DaemonSet pods and pods that cannot move; subtract them if the sum comes out close. Ours: the two &lt;code&gt;Allocated resources&lt;/code&gt; lines add up to 3,340m + 5,140m = 8,480m CPU, under the 12,000m allocatable, and 15,314 + 17,366 = 32,680 MiB memory, over the 31,452 MiB allocatable. After we subtracted DaemonSet pods and the eight pinned pods, 1,550m CPU and 8,896 MiB would move from &lt;code&gt;agent-1&lt;/code&gt;, and &lt;code&gt;server-1&lt;/code&gt; had 6,860m CPU and 14,086 MiB unrequested. Same problem if, after the subtraction, the requests that would move are larger than the other node's &lt;code&gt;Allocatable&lt;/code&gt; minus its own requests: some pods will report &lt;code&gt;Insufficient cpu&lt;/code&gt; or &lt;code&gt;Insufficient memory&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;5. Test it with a drain.&lt;/strong&gt; This one changes the cluster.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl drain &lt;span class="o"&gt;&amp;lt;&lt;/span&gt;node&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; --ignore-daemonsets --delete-emptydir-data --timeout=300s
kubectl get pods -A --field-selector status.phase=Pending \
  -o custom-columns=&lt;span class="s1"&gt;'NAMESPACE:.metadata.namespace,POD:.metadata.name,WHY:.status.conditions[0].message'&lt;/span&gt;
kubectl uncordon &lt;span class="o"&gt;&amp;lt;&lt;/span&gt;node&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;
kubectl rollout restart deployment/&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;name&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; -n &lt;span class="o"&gt;&amp;lt;&lt;/span&gt;namespace&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;   &lt;span class="c"&gt;# pods do not move back on their own&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Before draining, let running Jobs finish and look for pods without an owner:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl get pods -A -o json &lt;span class="o"&gt;|&lt;/span&gt; jq -r &lt;span class="s1"&gt;'.items[]|select(.metadata.ownerReferences==null)|.metadata.namespace+"/"+.metadata.name'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Drain deletes finished pods, and an ownerless one is gone for good. Exclude it with &lt;code&gt;--pod-selector&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Pass: every Deployment that is not pinned on purpose is Available on the other node, and each Pending pod has a reason you expected (&lt;code&gt;didn't match PersistentVolume's node affinity&lt;/code&gt;, &lt;code&gt;Insufficient nvidia.com/gpu&lt;/code&gt;, a hostname selector). Fail: a stateless app Pending with &lt;code&gt;didn't match Pod's node affinity/selector&lt;/code&gt; or &lt;code&gt;didn't match pod topology spread constraints&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;A drain evicts at once. A node that really dies is first marked unreachable, and Kubernetes adds a toleration "with tolerationSeconds=300" to every pod, so pods "remain bound to Nodes for 5 minutes after one of these problems is detected". Add at least five minutes to what the drain shows: the count starts when the node controller marks the node unreachable, not when the node fails.&lt;/p&gt;

&lt;p&gt;If you do not want to repeat checks 1 and 3 by hand after every change, &lt;a href="https://github.com/KyungHwanKim-devs/kubegap" rel="noopener noreferrer"&gt;kubegap&lt;/a&gt; runs them: it reports each workload that can run on only one node, pods tied to one node by a local volume, hostNetwork or a GPU, replicas of one Deployment on the same node, and workloads that depend on a node label. It runs check 4 only for losing the largest node, which for us is &lt;code&gt;server-1&lt;/code&gt;, so check 4 for losing &lt;code&gt;agent-1&lt;/code&gt; stays manual. Its report on our cluster had no finding for the spread defaults of check 2, so that one stays manual. Its run on 2026-11-11 is where this problem was found.&lt;/p&gt;

&lt;h2&gt;
  
  
  What we changed, and what we have not
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Changed
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;Commit 1, pushed 02:38:33.&lt;/strong&gt; In 16 Deployment manifests (15 deployed, one not deployed), &lt;code&gt;requiredDuringSchedulingIgnoredDuringExecution&lt;/code&gt; became &lt;code&gt;preferredDuringSchedulingIgnoredDuringExecution&lt;/code&gt; with weight 100 and the same &lt;code&gt;NotIn&lt;/code&gt; expression, and &lt;code&gt;nodeTaintsPolicy: Honor&lt;/code&gt; was added to the 10 spread rules. A server-side dry-run passed for all 16. Argo CD applied it by 02:39:23.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Commit 2, pushed 02:46:38.&lt;/strong&gt; &lt;code&gt;matchLabelKeys: [pod-template-hash]&lt;/code&gt; on the 10 spread rules. After its rollout settled at 02:50:25, every single-replica app was back on &lt;code&gt;agent-1&lt;/code&gt;, and &lt;code&gt;api-b&lt;/code&gt; and &lt;code&gt;api-d&lt;/code&gt; had one replica on each node.&lt;/p&gt;

&lt;p&gt;One Deployment after both commits:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;      &lt;span class="na"&gt;topologySpreadConstraints&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;maxSkew&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
          &lt;span class="na"&gt;topologyKey&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;kubernetes.io/hostname&lt;/span&gt;
          &lt;span class="na"&gt;whenUnsatisfiable&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;DoNotSchedule&lt;/span&gt;
          &lt;span class="na"&gt;nodeTaintsPolicy&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Honor&lt;/span&gt;
          &lt;span class="na"&gt;matchLabelKeys&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="s"&gt;pod-template-hash&lt;/span&gt;
          &lt;span class="na"&gt;labelSelector&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="na"&gt;matchLabels&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
              &lt;span class="na"&gt;app&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;api-d&lt;/span&gt;
      &lt;span class="na"&gt;affinity&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="na"&gt;nodeAffinity&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;preferredDuringSchedulingIgnoredDuringExecution&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;weight&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;100&lt;/span&gt;
              &lt;span class="na"&gt;preference&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                &lt;span class="na"&gt;matchExpressions&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                  &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;key&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;role&lt;/span&gt;
                    &lt;span class="na"&gt;operator&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;NotIn&lt;/span&gt;
                    &lt;span class="na"&gt;values&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="s"&gt;observability&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;We chose preferred with weight 100 rather than removing the rule, so that each app keeps its usual node every day and can still start on the other node when its usual node is gone.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The drain test.&lt;/strong&gt; A backup Job running on &lt;code&gt;agent-1&lt;/code&gt; was allowed to finish first (02:50:47). A probe of 20 public addresses started at 02:50:47, one round about every 5 s. Then:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight console"&gt;&lt;code&gt;$ kubectl drain agent-1 --ignore-daemonsets --delete-emptydir-data --pod-selector=app!=&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;the ownerless pod&lt;span class="s1"&gt;'s app label&amp;gt; --timeout=300s
node/agent-1 cordoned
Warning: ignoring DaemonSet-managed Pods: ...
evicting pod ...
...
node/agent-1 drained
# exit 0 at 2026-11-13T02:52:55Z
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The drain started at 02:51:48 and printed 83 eviction lines. The selector excluded an ownerless pod that had been Failed since May, because drain would have deleted it and nothing would have recreated it.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;At 02:52:55 two of the 15 app Deployments were not fully available, &lt;code&gt;admin-b&lt;/code&gt; (0 of 1) and &lt;code&gt;api-d&lt;/code&gt; (1 of 2); from 02:53:12 to 02:53:51 only &lt;code&gt;api-d&lt;/code&gt; was still short, and at 02:54:10 all 15 were fully available. That is 2 min 22 s after the drain started.&lt;/li&gt;
&lt;li&gt;The eight Pending pods were the ones shown in the first section, each with an expected reason.&lt;/li&gt;
&lt;li&gt;24 containers on &lt;code&gt;server-1&lt;/code&gt; started from images already there and 14 images were pulled, taking from 1.349 s to 2m9.6s. Eight of the nine images pulled during the first rollout were still on &lt;code&gt;server-1&lt;/code&gt;; for the ninth, one pod reported it already present at 02:52:00 and another reported a 1.349 s pull at 02:52:02, which we did not look into.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fswf7tmeehz8qxhf220zr.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fswf7tmeehz8qxhf220zr.png" alt="Grid of probe results for all 20 public addresses, one row each and one cell per 5-second round from 02:50 to 02:58 UTC: api-b and api-d stay green through the drain; single-replica apps show 1 to 15 red cells after 02:51:48; the API on a local volume on agent-1 and the search API stay red until after uncordon at 02:54:29." width="800" height="520"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;All 20 public addresses during the drain and the restore. The two apps with one replica per node never failed while agent-1 was drained; single-replica apps were down for about 5 to 85 seconds; the two tied to agent-1 stayed down until after uncordon.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;From the probe (an outage window is the first to the last failing round, plus one interval):&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The two 2-replica APIs never left their normal response code during the drain.&lt;/li&gt;
&lt;li&gt;Single-replica apps were off baseline for about 5 to 85 s: 1 round for the search web UI, 4 rounds from 02:51:54 to 02:52:19 for one site, 11 rounds from 02:52:04 to 02:53:02 for one admin app, 15 rounds to 02:53:24 for an internal API that has no node rule.&lt;/li&gt;
&lt;li&gt;A search API returned 503 from 02:52:04 to 02:54:24, because its embedding service needs the GPU on &lt;code&gt;agent-1&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;An API whose Postgres is on a local volume on &lt;code&gt;agent-1&lt;/code&gt; returned 502 and 503 from 02:51:48 to 02:54:51.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Restore.&lt;/strong&gt; &lt;code&gt;kubectl uncordon agent-1&lt;/code&gt; at 02:54:29; the pinned pods were Ready and nothing was Pending at 02:54:57. Pods do not move back by themselves, so we ran &lt;code&gt;kubectl rollout restart&lt;/code&gt; on the 15 app Deployments at 02:54:57 (settled 02:58:50, single-replica apps back on &lt;code&gt;agent-1&lt;/code&gt;) and, at 02:59:19, on 11 Deployments without node rules that the drain had moved to &lt;code&gt;server-1&lt;/code&gt;. At 03:04:37 there were 37 running pods on &lt;code&gt;agent-1&lt;/code&gt; and 63 on &lt;code&gt;server-1&lt;/code&gt;. During the restarts one probe round each returned no answer for two addresses (02:55:18 and 02:56:00), and the search API was off baseline until 03:04:33 while it restarted. All 20 addresses were at their normal code from 03:04:43.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;kubegap again.&lt;/strong&gt; A full rerun at 03:04:58: of the 60 findings the 2026-11-11 report listed under problem 5, 19 are gone (the 15 one-eligible-node findings of the app Deployments and 4 replicas-on-one-node findings) and 41 remain.&lt;/p&gt;

&lt;h3&gt;
  
  
  Not changed, and why
&lt;/h3&gt;

&lt;p&gt;These are our decisions for this cluster, with the reasons below. They are not advice for your cluster.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The control plane stays one k3s server on &lt;code&gt;server-1&lt;/code&gt;.&lt;/strong&gt; The k3s documentation: "HA embedded etcd cluster must be comprised of an odd number of server nodes for etcd to maintain quorum", and HA needs "Three or more server nodes". Two machines cannot form an etcd quorum that survives one failure, and we will not add a third machine: the cluster runs a few personal projects and small contract work. Losing &lt;code&gt;server-1&lt;/code&gt; is handled as a restore, from the etcd snapshots, server token and database dumps copied to &lt;code&gt;agent-1&lt;/code&gt; twice a day or daily; a restore from those copies has been tested. kubegap's rerun also shows that without &lt;code&gt;server-1&lt;/code&gt; the CPU requests would not fit: 6.0 cores allocatable against 7.61 requested.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Ingress controller replicas and CoreDNS were not spread.&lt;/strong&gt; Both run only on &lt;code&gt;server-1&lt;/code&gt; (ingress-nginx: 2 replicas, MetalLB L2 address, &lt;code&gt;externalTrafficPolicy: Local&lt;/code&gt;, PodDisruptionBudget minAvailable 1; CoreDNS: 1 replica, no PodDisruptionBudget). Both matter only when &lt;code&gt;server-1&lt;/code&gt; is lost, which is the restore case above. CoreDNS is a k3s packaged component; the k3s documentation says those manifests "should not be altered. The files are re-written to disk whenever K3s is started", so changing it would also need a &lt;code&gt;.skip&lt;/code&gt; file on the server so that k3s stops managing it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Pods tied to &lt;code&gt;agent-1&lt;/code&gt; by local-path volumes, hostNetwork or GPUs stay tied.&lt;/strong&gt; Moving them needs replicated storage, a network change or another GPU.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Where else this happened
&lt;/h2&gt;

&lt;p&gt;The Incident Library, a searchable collection of public post-incident reports, has no case with this exact cause (a required node affinity or a topology spread default leaving replacement pods Pending after a node loss). These three show a cluster that could not survive losing nodes: replicas packed onto one node, a control plane node that could not be replaced, and all workers NotReady at once.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Moonlight, 2019-01-18: on a three-node GKE cluster the scheduler sometimes put all three replicas of a web app on one node; CPU-heavy pods drove that node into kernel panics, and the crash moved from node to node until the site was down. Anti-affinity rules on all major Deployments restored it.&lt;/li&gt;
&lt;li&gt;Magic Pages, 2025-04-14: one of three control plane nodes of a hetzner-k3s cluster ran out of memory, and the cluster's own Hetzner API calls kept the rate limit at zero, so the node could not be replaced and every customer site went down.&lt;/li&gt;
&lt;li&gt;Saltside, 2017-04-26: parallel Helm installs saturated small gp2 root volumes and left all three workers of a kops cluster NotReady.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The three cases above are part of a set of six in which a NotReady node was a single point of failure: &lt;a href="https://benchlet.com/incidentlib/app?symptom=node-not-ready&amp;amp;factor=single-point-of-failure" rel="noopener noreferrer"&gt;Node NotReady + single point of failure, 6 cases&lt;/a&gt;. Two of the cases under &lt;a href="https://benchlet.com/incidentlib/app?component=kube-apiserver&amp;amp;factor=single-point-of-failure" rel="noopener noreferrer"&gt;Kubernetes API server + single point of failure, 7 cases&lt;/a&gt; are clusters that went down with their only control plane node, as ours would if &lt;code&gt;server-1&lt;/code&gt; were lost: a homelab whose only master node kernel-panicked after a power outage (2025-03-03), and NRE Labs, whose single-master kubeadm cluster kept losing its API server (2018-12-04).&lt;/p&gt;

&lt;h2&gt;
  
  
  Timeline
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Time (UTC)&lt;/th&gt;
&lt;th&gt;Event&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;2026-11-11 05:08:33&lt;/td&gt;
&lt;td&gt;kubegap full run reports problem 5: the two nodes cannot stand in for each other.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2026-11-13 02:25–02:28&lt;/td&gt;
&lt;td&gt;State recorded: 15 app Deployments with &lt;code&gt;role NotIn [observability]&lt;/code&gt;, which matches only &lt;code&gt;agent-1&lt;/code&gt;.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;02:38:33&lt;/td&gt;
&lt;td&gt;Commit 1 pushed: required → preferred weight 100, &lt;code&gt;nodeTaintsPolicy: Honor&lt;/code&gt;.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;02:39:23&lt;/td&gt;
&lt;td&gt;Argo CD has applied commit 1 to the 15 Deployments; rolling updates start.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;02:41:44–02:45:35&lt;/td&gt;
&lt;td&gt;Nine image pulls on &lt;code&gt;server-1&lt;/code&gt; (2m25s to 6m13s) for the pods the rollout moved there.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;02:46:38&lt;/td&gt;
&lt;td&gt;Commit 2 pushed: &lt;code&gt;matchLabelKeys: [pod-template-hash]&lt;/code&gt;.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;02:50:25&lt;/td&gt;
&lt;td&gt;Commit 2 rollout settled; single-replica apps back on &lt;code&gt;agent-1&lt;/code&gt;.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;02:50:47&lt;/td&gt;
&lt;td&gt;Backup Job on &lt;code&gt;agent-1&lt;/code&gt; complete; public probe starts.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;02:51:48&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;kubectl drain agent-1&lt;/code&gt; starts.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;02:52:55&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;node/agent-1 drained&lt;/code&gt;; 2 app Deployments not fully available.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;02:54:10&lt;/td&gt;
&lt;td&gt;All 15 app Deployments fully available on &lt;code&gt;server-1&lt;/code&gt;.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;02:54:12&lt;/td&gt;
&lt;td&gt;Eight pinned pods Pending with expected reasons; 89 pods on &lt;code&gt;server-1&lt;/code&gt;, 0 on &lt;code&gt;agent-1&lt;/code&gt;.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;02:54:29&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;kubectl uncordon agent-1&lt;/code&gt;.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;02:54:57&lt;/td&gt;
&lt;td&gt;Pinned pods Ready, nothing Pending; rollout restart of the 15 app Deployments.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;02:58:50&lt;/td&gt;
&lt;td&gt;App rollout settled.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;02:59:19&lt;/td&gt;
&lt;td&gt;Rollout restart of 11 Deployments without node rules.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;03:04:37&lt;/td&gt;
&lt;td&gt;37 pods on &lt;code&gt;agent-1&lt;/code&gt;, 63 on &lt;code&gt;server-1&lt;/code&gt;.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;03:04:43&lt;/td&gt;
&lt;td&gt;All 20 public addresses back at their normal code.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;03:04:58&lt;/td&gt;
&lt;td&gt;kubegap rerun: 19 of 60 problem-5 findings gone, 41 remain.&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  FAQ
&lt;/h2&gt;

&lt;h3&gt;
  
  
  pods not rescheduled when node goes down
&lt;/h3&gt;

&lt;p&gt;Pods are evicted only after the node is marked unreachable and the 300-second toleration has run out. If the replacement pods are then stuck Pending after the node failure, read the &lt;code&gt;FailedScheduling&lt;/code&gt; event, which starts &lt;code&gt;0/2 nodes are available:&lt;/code&gt; on a two-node cluster: the dead node shows &lt;code&gt;had untolerated taint {node.kubernetes.io/unreachable: }&lt;/code&gt; (&lt;code&gt;had untolerated taint(s)&lt;/code&gt; on 1.35 and later), and &lt;code&gt;didn't match Pod's node affinity/selector&lt;/code&gt; on the surviving node means a required node rule excludes it. Count the nodes it matches with &lt;code&gt;kubectl get nodes -l&lt;/code&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  required vs preferred node affinity
&lt;/h3&gt;

&lt;p&gt;&lt;code&gt;requiredDuringSchedulingIgnoredDuringExecution&lt;/code&gt; is a filter: no matching node, no scheduling. &lt;code&gt;preferredDuringSchedulingIgnoredDuringExecution&lt;/code&gt; is a score with weight 1 to 100, added to matching nodes that passed every filter: the pod usually goes to a matching node and goes elsewhere when none is available, but a filter such as a &lt;code&gt;DoNotSchedule&lt;/code&gt; spread rule can still remove the matching node. We use preferred with weight 100 so apps keep their usual node daily but can move when it fails.&lt;/p&gt;

&lt;h3&gt;
  
  
  didn't match pod topology spread constraints
&lt;/h3&gt;

&lt;p&gt;With &lt;code&gt;whenUnsatisfiable: DoNotSchedule&lt;/code&gt; and &lt;code&gt;maxSkew: 1&lt;/code&gt;, a node that is cordoned or unreachable still counts as a domain with zero pods while &lt;code&gt;nodeTaintsPolicy&lt;/code&gt; is unset (Ignore), so the second replica cannot join the first on the surviving node; the Kubernetes blog describes pods stuck Pending this way. Set &lt;code&gt;nodeTaintsPolicy: Honor&lt;/code&gt;. For rolling updates, add &lt;code&gt;matchLabelKeys: [pod-template-hash]&lt;/code&gt; so old pods are not counted.&lt;/p&gt;

&lt;h3&gt;
  
  
  k3s two node high availability
&lt;/h3&gt;

&lt;p&gt;Not for the control plane. k3s embedded etcd needs an odd number of servers, and HA needs three or more; two servers cannot keep quorum when one fails. With two machines, Deployments can move between nodes after fixing their rules, but the single server is recovered from etcd snapshots and dumps kept on the other node.&lt;/p&gt;

&lt;h3&gt;
  
  
  test node failure with kubectl drain
&lt;/h3&gt;

&lt;p&gt;Let running Jobs finish, and use &lt;code&gt;--pod-selector&lt;/code&gt; to exclude finished pods that have no owner, because drain deletes finished pods and nothing recreates an ownerless one. Then run &lt;code&gt;kubectl drain &amp;lt;node&amp;gt; --ignore-daemonsets --delete-emptydir-data&lt;/code&gt; and list Pending pods with their scheduling message. Pass if every stateless Deployment is Available elsewhere and each Pending pod has an expected reason. A drain evicts at once; after a real failure, eviction comes at least 300 seconds later, counted from when the node is marked unreachable. Uncordon, then &lt;code&gt;rollout restart&lt;/code&gt;, because pods do not move back.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://benchlet.com/guides/pods-pending-after-node-failure-node-affinity-topology-spread-k3s/" rel="noopener noreferrer"&gt;benchlet.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>kubernetes</category>
      <category>scheduling</category>
      <category>sre</category>
      <category>devops</category>
    </item>
    <item>
      <title>Kubernetes CronJob pg_dump backup: does the dump end up on the same node as the database?</title>
      <dc:creator>KYUNGHWAN KIM</dc:creator>
      <pubDate>Thu, 12 Nov 2026 19:54:17 +0000</pubDate>
      <link>https://dev.to/kkhdevs/kubernetes-cronjob-pgdump-backup-does-the-dump-end-up-on-the-same-node-as-the-database-5dda</link>
      <guid>https://dev.to/kkhdevs/kubernetes-cronjob-pgdump-backup-does-the-dump-end-up-on-the-same-node-as-the-database-5dda</guid>
      <description>&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F1bo9kmwhpq78lpbacls0.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F1bo9kmwhpq78lpbacls0.png" alt="Terminal line &amp;quot;kubectl get jobs -A | grep backup&amp;quot; with &amp;quot;7 × Complete 1/1&amp;quot;, above the sentence &amp;quot;5 of 7 database dumps and every k3s etcd snapshot were on the database's own disk.&amp;quot;" width="800" height="420"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Every backup Job completed. Five of the seven dumps were on the same disk as the databases they copy.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;If your &lt;strong&gt;Kubernetes CronJob pg_dump backup&lt;/strong&gt; writes to a PVC of the &lt;code&gt;local-path&lt;/code&gt; StorageClass (the k3s default) and &lt;code&gt;kubectl get jobs -A&lt;/code&gt; shows every backup Job as &lt;code&gt;Complete 1/1&lt;/code&gt;, that output does not tell you which node's disk the dump is on. On our cluster all seven backup Jobs completed on 2026-11-11, but five of the dumps were on the same single disk as the databases they copy, and every k3s etcd snapshot and the server token needed to restore them were on that disk too. A local-path volume is created on the node where the first pod that uses it is scheduled, and in five of our seven backup CronJobs nothing placed that pod on a node other than the database's. To see yours, list each PV with its node (check 1 below): if a backup claim and the database's data claim show the same NODE, the dump is on the database's node, and on the same disk if that node has only one. The cluster runs k3s v1.33.5+k3s1 on two computers: one server, which holds etcd and all seven backed-up databases on a single SSD, and one agent. All times are UTC.&lt;/p&gt;

&lt;h2&gt;
  
  
  What we saw
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;server-1&lt;/code&gt; is the only server and has one 500 GB SATA SSD, no RAID; &lt;code&gt;agent-1&lt;/code&gt; is the agent, on the same network. Storage is the k3s built-in local-path provisioner; applications are deployed by Argo CD. We captured this state on 2026-11-12 from 01:25 UTC; names are masked and columns are trimmed.&lt;/p&gt;

&lt;p&gt;Seven backup CronJobs (six &lt;code&gt;pg_dump&lt;/code&gt;, one &lt;code&gt;mysqldump&lt;/code&gt;) run daily between 02:50 and 03:50. The latest run of each (the output had three Jobs per CronJob and one old Job created by hand; only the newest Job of each CronJob is kept here, and the CONTAINERS, IMAGES and SELECTOR columns are cut):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ kubectl get jobs -A
NAMESPACE   NAME                               STATUS     COMPLETIONS   DURATION   AGE
prod-a      app-a-postgres-backup-29906115     Complete   1/1           9m7s       22h
prod-b      app-b-postgres-backup-29906130     Complete   1/1           4s         21h
prod-b      app-b-postgres-3-backup-29906130   Complete   1/1           4s         21h
prod-b      app-b-postgres-2-backup-29906130   Complete   1/1           4s         21h
prod-c      app-c-postgres-backup-29906150     Complete   1/1           4s         21h
prod-d      app-d-mysql-backup-29906090        Complete   1/1           11s        22h
prod-e      app-e-postgres-backup-29906100     Complete   1/1           4s         22h
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Each job writes a gzip dump (&lt;code&gt;&amp;lt;name&amp;gt;_&amp;lt;timestamp&amp;gt;.sql.gz&lt;/code&gt;; the mysqldump job &lt;code&gt;&amp;lt;name&amp;gt;-&amp;lt;date&amp;gt;-&amp;lt;time&amp;gt;.sql.gz&lt;/code&gt;) to a local-path PVC mounted at &lt;code&gt;/backup&lt;/code&gt; and deletes older files. No command in any of the seven sends anything outside the cluster: there is no &lt;code&gt;s3&lt;/code&gt;, &lt;code&gt;rclone&lt;/code&gt;, &lt;code&gt;restic&lt;/code&gt;, &lt;code&gt;scp&lt;/code&gt; or &lt;code&gt;rsync&lt;/code&gt; in them.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fxs0o5l2bujzar7047txr.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fxs0o5l2bujzar7047txr.png" alt="kubectl get pv output with columns NS, PVC, NODE and RECLAIM; five backup PVCs in red are on server-1, the same node as their database volumes, and two backup PVCs are on agent-1." width="800" height="409"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;&lt;code&gt;kubectl get pv&lt;/code&gt; at 2026-11-12 01:25 UTC. Red: a dump volume on the same node as its database, 5 of 7.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The same volumes, next to the node of each source database pod:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;CronJob&lt;/th&gt;
&lt;th&gt;dump PVC on&lt;/th&gt;
&lt;th&gt;source database on&lt;/th&gt;
&lt;th&gt;same node&lt;/th&gt;
&lt;th&gt;latest dump (2026-11-11)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;app-e-postgres-backup&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;server-1&lt;/td&gt;
&lt;td&gt;server-1&lt;/td&gt;
&lt;td&gt;yes&lt;/td&gt;
&lt;td&gt;62,739 bytes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;app-b-postgres-backup&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;server-1&lt;/td&gt;
&lt;td&gt;server-1&lt;/td&gt;
&lt;td&gt;yes&lt;/td&gt;
&lt;td&gt;411,735 bytes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;app-b-postgres-2-backup&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;server-1&lt;/td&gt;
&lt;td&gt;server-1&lt;/td&gt;
&lt;td&gt;yes&lt;/td&gt;
&lt;td&gt;11,176 bytes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;app-b-postgres-3-backup&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;agent-1&lt;/td&gt;
&lt;td&gt;server-1&lt;/td&gt;
&lt;td&gt;no&lt;/td&gt;
&lt;td&gt;8,287 bytes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;app-d-mysql-backup&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;server-1&lt;/td&gt;
&lt;td&gt;server-1&lt;/td&gt;
&lt;td&gt;yes&lt;/td&gt;
&lt;td&gt;15,777,210 bytes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;app-a-postgres-backup&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;agent-1&lt;/td&gt;
&lt;td&gt;server-1&lt;/td&gt;
&lt;td&gt;no&lt;/td&gt;
&lt;td&gt;1,024,335,407 bytes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;app-c-postgres-backup&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;server-1&lt;/td&gt;
&lt;td&gt;server-1&lt;/td&gt;
&lt;td&gt;yes&lt;/td&gt;
&lt;td&gt;2,434,541 bytes&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;All seven source databases are on &lt;code&gt;server-1&lt;/code&gt;, whose one disk holds a single filesystem at &lt;code&gt;/&lt;/code&gt; with the etcd database, the etcd snapshots and the local-path volumes. Every PV had reclaim policy &lt;code&gt;Delete&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fbq22tktaec1ubz72zugs.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fbq22tktaec1ubz72zugs.png" alt="kubectl get etcdsnapshotfiles output: five snapshots, all on node server-1, every LOCATION starting with file:///var/lib/rancher/k3s/server/db/snapshots/." width="799" height="128"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;All five k3s etcd snapshots, &lt;code&gt;file://&lt;/code&gt; on the server's own disk.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The five snapshots are &lt;code&gt;etcd-snapshot-server-1-&amp;lt;unix time&amp;gt;&lt;/code&gt;, 41,775,136 to 45,232,160 bytes each. The k3s unit has no &lt;code&gt;--etcd-s3*&lt;/code&gt; argument and there is no &lt;code&gt;/etc/rancher/k3s/config.yaml&lt;/code&gt;. The server token, &lt;code&gt;/var/lib/rancher/k3s/server/token&lt;/code&gt; (mode 600, root), is on the same disk. A cleanup CronJob keeps only the 5 newest snapshot files, daily at 03:00.&lt;/p&gt;

&lt;p&gt;One disk failure on &lt;code&gt;server-1&lt;/code&gt; would have removed the etcd database, all 5 snapshots and the token, every source database, and 5 of the 7 dumps (inference from the facts above). Only the &lt;code&gt;app-a&lt;/code&gt; and &lt;code&gt;app-b-postgres-3&lt;/code&gt; dumps on &lt;code&gt;agent-1&lt;/code&gt; would have been left.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why local-path created the dump volumes on the database's node
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;local-path creates the volume where the first pod runs.&lt;/strong&gt; Both StorageClasses use &lt;code&gt;WaitForFirstConsumer&lt;/code&gt;: the volume is created on the node where the first pod that uses the claim is scheduled. The second class is named after the server, but nothing in it pins a volume to that node: no &lt;code&gt;allowedTopologies&lt;/code&gt;, only a path.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="c1"&gt;# kubectl get storageclass local-path-server-1 -o yaml (trimmed)
&lt;/span&gt;&lt;span class="na"&gt;allowVolumeExpansion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;
&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;storage.k8s.io/v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;StorageClass&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;local-path-server-1&lt;/span&gt;
&lt;span class="na"&gt;parameters&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;path&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;/var/lib/rancher/k3s/storage&lt;/span&gt;
&lt;span class="na"&gt;provisioner&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;rancher.io/local-path&lt;/span&gt;
&lt;span class="na"&gt;reclaimPolicy&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Delete&lt;/span&gt;
&lt;span class="na"&gt;volumeBindingMode&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;WaitForFirstConsumer&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;After that the PV is fixed to the node. From the local-path-provisioner README: PersistentVolumes created by the provisioner "use the &lt;code&gt;kubernetes.io/hostname&lt;/code&gt; label for node affinity. This ensures the volume is only accessible on the node where it was provisioned."&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Only one of the seven jobs was placed on another node on purpose.&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;app-a-postgres-backup&lt;/code&gt; has &lt;code&gt;nodeSelector: kubernetes.io/hostname: agent-1&lt;/code&gt;. It was created on 2026-07-02 with that selector on purpose, and it was the only backup placed away from its database by design.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;app-b-postgres-3-backup&lt;/code&gt; had no nodeSelector and used &lt;code&gt;local-path-server-1&lt;/code&gt;. Its volume is on &lt;code&gt;agent-1&lt;/code&gt; only because its first pod was scheduled there (inference from the binding rule above).&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;app-c-postgres-backup&lt;/code&gt; had &lt;code&gt;nodeSelector: storage: server-1&lt;/code&gt;, which pinned its dump to the database's node.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;app-d-mysql-backup&lt;/code&gt; had &lt;code&gt;nodeSelector: kubernetes.io/hostname: server-1&lt;/code&gt; and wrote to a claim from the database StatefulSet's own &lt;code&gt;volumeClaimTemplate&lt;/code&gt;, so its dump volume was created with the database pod, on its node.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;app-b-postgres-backup&lt;/code&gt;, &lt;code&gt;app-b-postgres-2-backup&lt;/code&gt; and &lt;code&gt;app-e-postgres-backup&lt;/code&gt; had no nodeSelector, and their dump volumes were created on &lt;code&gt;server-1&lt;/code&gt;, where their databases run.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;A &lt;code&gt;Complete&lt;/code&gt; Job status does not record which disk the dump was written to.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;k3s keeps etcd snapshots on the server unless told otherwise.&lt;/strong&gt; From the k3s etcd-snapshot docs: "Scheduled snapshots are enabled by default, at 00:00 and 12:00 system time, with 5 snapshots retained per server node on local disk" and "K3s etcd snapshots are stored on the node file system, and may optionally be uploaded to an S3-compatible object store for disaster recovery scenarios." We had not set &lt;code&gt;--etcd-s3&lt;/code&gt;, and &lt;code&gt;server-1&lt;/code&gt; is the only server, so all five snapshots were on one disk.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The snapshot is unusable without the token.&lt;/strong&gt; From the k3s backup and restore docs: "you must also back up the server token file at /var/lib/rancher/k3s/server/token" and "If you do not use the same token value when restoring, the snapshot will be unusable". A copy of the snapshots alone would not have been enough.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The snapshot does not hold volume contents.&lt;/strong&gt; It is "a complete copy of the etcd datastore, along with cluster certificate authority (CA) certificates and private keys" (k3s docs), so it has the API objects but not the data in volumes (inference from what etcd stores): the dumps and the snapshots have to be moved separately.&lt;/p&gt;

&lt;h2&gt;
  
  
  Check where your Kubernetes CronJob pg_dump backup is stored
&lt;/h2&gt;

&lt;p&gt;These are read-only, except checks 5 and 6, which create throwaway objects. &lt;a href="https://github.com/KyungHwanKim-devs/kubegap" rel="noopener noreferrer"&gt;kubegap&lt;/a&gt;, an audit tool that reads the cluster of your current kube context, found this problem here on 2026-11-11 by reading the same three things as checks 1 to 3. It reports all three in one run, so you can run it instead of pasting those three commands. Checks 4 to 6 are run by hand.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;1. Which node holds each dump, and which node holds the database it copies?&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl get pv --sort-by=.spec.claimRef.namespace -o custom-columns=&lt;span class="s1"&gt;'NS:.spec.claimRef.namespace,PVC:.spec.claimRef.name,NODE:.spec.nodeAffinity.required.nodeSelectorTerms[0].matchExpressions[0].values[0],CLASS:.spec.storageClassName,RECLAIM:.spec.persistentVolumeReclaimPolicy'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Same problem if: the row of a backup PVC and the row of the database's data PVC show the same NODE, and that node has one disk (&lt;code&gt;lsblk -d -o NAME,TYPE,SIZE&lt;/code&gt; on the node lists one row of TYPE &lt;code&gt;disk&lt;/code&gt;; on &lt;code&gt;server-1&lt;/code&gt; it was &lt;code&gt;sda disk 465.8G&lt;/code&gt;). Read NODE, not CLASS: with local-path a StorageClass name does not choose the node; the first pod that uses the claim does (&lt;code&gt;WaitForFirstConsumer&lt;/code&gt;). RECLAIM &lt;code&gt;Delete&lt;/code&gt; means deleting the claim deletes the directory on the node.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Does any backup job send its result to another machine?&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl get cronjob -A -o json &lt;span class="o"&gt;|&lt;/span&gt; jq -r &lt;span class="s1"&gt;'.items[]
  | select(.metadata.name | test("backup|dump|snapshot"))
  | [.metadata.namespace, .metadata.name,
     (.spec.jobTemplate.spec.template.spec.nodeSelector // {} | tostring),
     ([.spec.jobTemplate.spec.template.spec.containers[] | ((.command // []) + (.args // [])) | join(" ")] | join(" ")
      | test("s3|rclone|restic|kopia|scp|rsync") | tostring)]
  | @tsv'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;On our cluster, before the change:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;prod-a  app-a-postgres-backup   {"kubernetes.io/hostname":"agent-1"}    false
prod-b  app-b-postgres-backup   {}  false
prod-b  app-b-postgres-3-backup {}  false
prod-b  app-b-postgres-2-backup {}  false
prod-c  app-c-postgres-backup   {"storage":"server-1"}  false
prod-d  app-d-mysql-backup  {"kubernetes.io/hostname":"server-1"}   false
prod-e  app-e-postgres-backup   {}  false
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Same problem if: the last column is &lt;code&gt;false&lt;/code&gt; and the nodeSelector column is empty or names the database's node. Ours had all seven &lt;code&gt;false&lt;/code&gt;, four with &lt;code&gt;{}&lt;/code&gt; and two naming &lt;code&gt;server-1&lt;/code&gt;. The match reads only the command and args, so &lt;code&gt;true&lt;/code&gt; means "look closer", and a script mounted from a ConfigMap shows &lt;code&gt;false&lt;/code&gt; even if it uploads: open the script before you conclude.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Where are the etcd snapshots?&lt;/strong&gt; (k3s with embedded etcd)&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl get etcdsnapshotfile
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Same problem if: every LOCATION starts with &lt;code&gt;file://&lt;/code&gt; and nothing copies that directory to another machine (check 2). Those snapshots are on the server's own disk, next to the etcd database they copy.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. Did you back up the server token anywhere but the server?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;On the server: &lt;code&gt;sudo ls -l /var/lib/rancher/k3s/server/token&lt;/code&gt;. Then look for the same file where your snapshot copies go.&lt;/p&gt;

&lt;p&gt;Same problem if: the token exists only on the server. Without the same token value, a restored snapshot is unusable (k3s docs, backup and restore).&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;5. Does the newest dump restore from the other machine?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;A Job in the dump's namespace that mounts the dump PVC read-only (it is scheduled on the node that holds the PVC), starts an empty server in an &lt;code&gt;emptyDir&lt;/code&gt;, restores with &lt;code&gt;ON_ERROR_STOP&lt;/code&gt;, and counts tables. The Job below is for PostgreSQL; use the image your backup uses (uid 70 is &lt;code&gt;postgres&lt;/code&gt; in the alpine image):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;batch/v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Job&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s"&gt;name:&lt;/span&gt; &lt;span class="s"&gt;restore-test&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;namespace:&lt;/span&gt; &lt;span class="s"&gt;&amp;lt;ns&amp;gt;&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;backoffLimit&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
  &lt;span class="na"&gt;ttlSecondsAfterFinished&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;3600&lt;/span&gt;
  &lt;span class="na"&gt;activeDeadlineSeconds&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;900&lt;/span&gt;
  &lt;span class="na"&gt;template&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;restartPolicy&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Never&lt;/span&gt;
      &lt;span class="na"&gt;securityContext&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s"&gt;runAsUser:&lt;/span&gt; &lt;span class="mi"&gt;70&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;runAsGroup:&lt;/span&gt; &lt;span class="mi"&gt;70&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;fsGroup:&lt;/span&gt; &lt;span class="mi"&gt;70&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
      &lt;span class="na"&gt;containers&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;restore&lt;/span&gt;
        &lt;span class="na"&gt;image&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;postgres:16-alpine&lt;/span&gt;
        &lt;span class="na"&gt;command&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"/bin/sh"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"-c"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
        &lt;span class="na"&gt;args&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="p"&gt;|&lt;/span&gt;
          &lt;span class="s"&gt;set -eu&lt;/span&gt;
          &lt;span class="s"&gt;F=$(ls -t /backup/*.sql.gz | head -1); echo "latest $F"&lt;/span&gt;
          &lt;span class="s"&gt;gzip -t "$F"&lt;/span&gt;
          &lt;span class="s"&gt;initdb -D /work/pg -U postgres &amp;gt;/dev/null&lt;/span&gt;
          &lt;span class="s"&gt;pg_ctl -D /work/pg -o "-c listen_addresses='' -c unix_socket_directories=/work" -w start &amp;gt;/dev/null&lt;/span&gt;
          &lt;span class="s"&gt;createdb -h /work -U postgres restored&lt;/span&gt;
          &lt;span class="s"&gt;zcat "$F" | psql -h /work -U postgres -d restored -v ON_ERROR_STOP=1 -q&lt;/span&gt;
          &lt;span class="s"&gt;psql -h /work -U postgres -d restored -Atc "select count(*) from pg_tables where schemaname not in ('pg_catalog','information_schema')"&lt;/span&gt;
        &lt;span class="na"&gt;volumeMounts&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s"&gt;name:&lt;/span&gt; &lt;span class="s"&gt;backup&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;mountPath:&lt;/span&gt; &lt;span class="s"&gt;/backup&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;readOnly:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
        &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s"&gt;name:&lt;/span&gt; &lt;span class="s"&gt;work&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;mountPath:&lt;/span&gt; &lt;span class="s"&gt;/work&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
      &lt;span class="na"&gt;volumes&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s"&gt;name:&lt;/span&gt; &lt;span class="s"&gt;backup&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;persistentVolumeClaim:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s"&gt;claimName:&lt;/span&gt; &lt;span class="s"&gt;&amp;lt;dump-pvc&amp;gt;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;readOnly:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s"&gt;name:&lt;/span&gt; &lt;span class="s"&gt;work&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;emptyDir:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then &lt;code&gt;kubectl get pod -n &amp;lt;ns&amp;gt; -l job-name=restore-test -o wide&lt;/code&gt; shows the node it ran on, and the table count should equal the source's (&lt;code&gt;select count(*) from pg_tables where schemaname not in ('pg_catalog','information_schema')&lt;/code&gt; in the source database).&lt;/p&gt;

&lt;p&gt;Same problem if: the Job ran on the database's node, fails, or the count differs.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;6. Does the etcd copy restore?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Copy the newest snapshot and the token from the other machine to a third one with Docker, check the sha256 against the copy, then restore with the same k3s version and no agent (no kubelet, so no workload from the snapshot can start):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;docker run --rm --privileged -e &lt;span class="nv"&gt;K3S_TOKEN&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"$(cat token)"&lt;/span&gt; -v k3s-test:/var/lib/rancher/k3s -v &lt;span class="s2"&gt;"$PWD"&lt;/span&gt;:/snap:ro \
  rancher/k3s:&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;your version, e.g. v1.33.5-k3s1&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; server --disable-agent \
  --cluster-reset --cluster-reset-restore-path=/snap/&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;snapshot file&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;
docker run -d --name k3s-test --privileged -e &lt;span class="nv"&gt;K3S_TOKEN&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"$(cat token)"&lt;/span&gt; -v k3s-test:/var/lib/rancher/k3s \
  rancher/k3s:&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;same version&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; server --disable-agent
docker &lt;span class="nb"&gt;exec&lt;/span&gt; k3s-test kubectl get ns,deploy,sts,cronjob -A --no-headers &lt;span class="o"&gt;|&lt;/span&gt; wc -l
docker rm -f k3s-test&lt;span class="o"&gt;;&lt;/span&gt; docker volume rm k3s-test
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The first command prints "Managed etcd cluster membership has been reset, restart without --cluster-reset flag now" and exits. Same problem if: it exits with an error, or the object counts are far from the live cluster's (objects created after the snapshot account for small differences). Delete the copied snapshot afterwards: it contains every Secret of the cluster.&lt;/p&gt;

&lt;h2&gt;
  
  
  What we changed and what we have not
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F0vhgqexw1kd7s4x9qlsj.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F0vhgqexw1kd7s4x9qlsj.png" alt="Before and after diagram of server-1 and agent-1. Before: server-1 holds 7 database servers, the etcd database, 5 of 7 dumps, 5 etcd snapshots and the server token; agent-1 holds 2 of 7 dumps. After: server-1 keeps the originals; agent-1 holds 7 of 7 dumps, the etcd snapshot copies taken twice a day and a copy of the server token." width="800" height="320"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;What was on each machine before and after. Red: lost together with the database if server-1's disk fails.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Changes 2 and 3 went through the manifests repository and Argo CD. Change 1 was made with &lt;code&gt;kubectl patch&lt;/code&gt; and recorded in the repository afterwards. The node settings on &lt;code&gt;server-1&lt;/code&gt; are recorded in the README next to the CronJob.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;1. &lt;code&gt;Retain&lt;/code&gt; on 30 of the 32 PVs (2026-11-12 01:35).&lt;/strong&gt; We set every PV outside the logging and monitoring namespaces to &lt;code&gt;Retain&lt;/code&gt; with &lt;code&gt;kubectl patch pv &amp;lt;name&amp;gt; -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'&lt;/code&gt;, and no pod restarted. Retain keeps the node directory when a claim is deleted and does nothing when the disk fails, so it is not the fix for this problem.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Every dump Job runs on &lt;code&gt;agent-1&lt;/code&gt;.&lt;/strong&gt; All seven CronJobs now have:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;jobTemplate&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;template&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;nodeSelector&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="na"&gt;kubernetes.io/hostname&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;agent-1&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The five that were on &lt;code&gt;server-1&lt;/code&gt; write to a new PVC of class &lt;code&gt;local-path&lt;/code&gt; (&lt;code&gt;app-b-postgres-backup-agent-1&lt;/code&gt;, &lt;code&gt;app-b-postgres-2-backup-agent-1&lt;/code&gt;, &lt;code&gt;app-c-postgres-backup-agent-1&lt;/code&gt;, &lt;code&gt;app-d-mysql-backup-agent-1&lt;/code&gt;, &lt;code&gt;app-e-postgres-backup-agent-1&lt;/code&gt;). Because of &lt;code&gt;WaitForFirstConsumer&lt;/code&gt;, each is created on &lt;code&gt;agent-1&lt;/code&gt;, where its first pod runs. Changing only the nodeSelector would not have moved the old volumes: an existing local-path PV keeps its node affinity, so a pod pinned elsewhere could not mount it (inference from the provisioner's node affinity rule). Retention is now 7 days for all seven (&lt;code&gt;find -mtime +6 -delete&lt;/code&gt;).&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. The etcd snapshots and the token are pulled to &lt;code&gt;agent-1&lt;/code&gt;.&lt;/strong&gt; A CronJob &lt;code&gt;kube-system/etcd-snapshot-backup&lt;/code&gt; on &lt;code&gt;agent-1&lt;/code&gt; runs at &lt;code&gt;30 0,12 * * *&lt;/code&gt;, 30 minutes after each k3s snapshot, and pulls &lt;code&gt;/var/lib/rancher/k3s/server/db/snapshots/&lt;/code&gt; and &lt;code&gt;/var/lib/rancher/k3s/server/token&lt;/code&gt; from &lt;code&gt;server-1&lt;/code&gt; with rsync over SSH into the PVC &lt;code&gt;etcd-snapshot-backup-agent-1&lt;/code&gt;, kept 7 days. Its name contains "backup", so the existing alert &lt;code&gt;BackupCronJobStale&lt;/code&gt; (no success for 26 hours) covers it.&lt;/p&gt;

&lt;p&gt;On &lt;code&gt;server-1&lt;/code&gt;, root may log in over SSH only with a forced-command key and only from &lt;code&gt;agent-1&lt;/code&gt; and the pod network:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ssh"&gt;&lt;code&gt;&lt;span class="c1"&gt;# /etc/ssh/sshd_config.d/10-etcd-snapshot-backup.conf
&lt;/span&gt;&lt;span class="nc"&gt;Match&lt;/span&gt; &lt;span class="nc"&gt;Address&lt;/span&gt; &lt;span class="mf"&gt;10.0&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mf"&gt;0.11&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mf"&gt;10.42&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mf"&gt;0.0&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mi"&gt;16&lt;/span&gt;
    &lt;span class="nc"&gt;PermitRootLogin&lt;/span&gt; &lt;span class="n"&gt;forced&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;commands&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;only&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ssh"&gt;&lt;code&gt;&lt;span class="c1"&gt;# /root/.ssh/authorized_keys (one line; key not shown)
&lt;/span&gt;&lt;span class="na"&gt;restrict,from&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"10.0.0.11,10.42.0.0/16"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="n"&gt;command&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"/usr/bin/rrsync -ro /var/lib/rancher/k3s/server/"&lt;/span&gt; &lt;span class="n"&gt;ssh&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;ed25519&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;public&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;etcd&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;snapshot&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;backup&lt;/span&gt;@&lt;span class="n"&gt;agent&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;rrsync -ro&lt;/code&gt; limits the key to read-only rsync under &lt;code&gt;/var/lib/rancher/k3s/server/&lt;/code&gt;. &lt;code&gt;sshd -t&lt;/code&gt; passed. &lt;code&gt;sshd -T -C user=root,host=x,addr=&amp;lt;address&amp;gt;&lt;/code&gt; was compared before and after for two addresses: &lt;code&gt;203.0.113.9&lt;/code&gt;, outside the Match, was unchanged, and &lt;code&gt;10.0.0.11&lt;/code&gt; changed only in &lt;code&gt;permitrootlogin&lt;/code&gt; (&lt;code&gt;no&lt;/code&gt; to &lt;code&gt;forced-commands-only&lt;/code&gt;). Addresses in &lt;code&gt;10.42.0.0/16&lt;/code&gt; are inside the Match too. The private key is kept in the secret manager the cluster already uses and reaches the pod as a Secret through External Secrets.&lt;/p&gt;

&lt;p&gt;The first copy failed. All three attempts of the Job, on &lt;code&gt;agent-1&lt;/code&gt;, ended the same way:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;ssh: connect to host 10.0.0.12 port 22: Operation timed out
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: unexplained error (code 255) at io.c(232) [Receiver=3.4.1]
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;agent-1&lt;/code&gt; node itself could not reach &lt;code&gt;server-1:22&lt;/code&gt; either. The ufw rows on &lt;code&gt;server-1&lt;/code&gt; before the change:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;22/tcp                     ALLOW IN    100.64.0.0/10
Anywhere                   ALLOW IN    10.42.0.0/16
Default: deny (incoming), allow (outgoing), allow (routed)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That allows &lt;code&gt;22/tcp&lt;/code&gt; from the Tailscale range and every port from the pod network, but nothing from &lt;code&gt;agent-1&lt;/code&gt;'s own address, &lt;code&gt;10.0.0.11&lt;/code&gt;. The pod's connection timed out although the pod network was allowed, so it reached &lt;code&gt;server-1&lt;/code&gt; from &lt;code&gt;10.0.0.11&lt;/code&gt; (inference). One rule fixed it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;ufw allow from 10.0.0.11 to any port 22 proto tcp comment &lt;span class="s2"&gt;"etcd-snapshot-backup from agent-1"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The next run copied 5 snapshots and the token to &lt;code&gt;agent-1&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Verification (2026-11-12 17:24 to 17:55).&lt;/strong&gt; Each backup CronJob was run once by hand after Argo CD applied the change: all seven ran on &lt;code&gt;agent-1&lt;/code&gt; and completed (&lt;code&gt;app-a-postgres-backup&lt;/code&gt;, the largest, in 9 minutes). Every backup PVC in use is now on &lt;code&gt;agent-1&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;The newest dump of each of the five moved databases was then restored from its PVC on &lt;code&gt;agent-1&lt;/code&gt; into an empty server (psql with &lt;code&gt;ON_ERROR_STOP=1&lt;/code&gt;, or mysql), and compared with counts taken inside the source pod at the same minute:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fmzvotp96ot887n9azdr4.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fmzvotp96ot887n9azdr4.png" alt="Restore test output: app-b-postgres, app-b-postgres-2, app-c-postgres, app-e-postgres and app-d-mysql all ran on agent-1, exited 0, and have equal table counts between source and restore; row counts are equal for the four PostgreSQL databases, and for app-d-mysql they were not counted at the source." width="800" height="200"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Five dumps restored from agent-1 at 17:42 UTC. Table and row counts equal the source; the MySQL source rows were not counted.&lt;/em&gt;&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;database&lt;/th&gt;
&lt;th&gt;restore&lt;/th&gt;
&lt;th&gt;tables source / restored&lt;/th&gt;
&lt;th&gt;rows source / restored&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;app-b-postgres&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;exit 0&lt;/td&gt;
&lt;td&gt;93 / 93&lt;/td&gt;
&lt;td&gt;576 / 576&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;app-b-postgres-2&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;exit 0&lt;/td&gt;
&lt;td&gt;103 / 103&lt;/td&gt;
&lt;td&gt;6 / 6&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;app-c-postgres&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;exit 0&lt;/td&gt;
&lt;td&gt;149 / 149&lt;/td&gt;
&lt;td&gt;122,805 / 122,805&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;app-e-postgres&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;exit 0&lt;/td&gt;
&lt;td&gt;13 / 13&lt;/td&gt;
&lt;td&gt;1,001 / 1,001&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;app-d-mysql&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;exit 0&lt;/td&gt;
&lt;td&gt;67 / 67&lt;/td&gt;
&lt;td&gt;not counted at the source / 620,704&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The etcd copy was restored the same way as check 6. The newest snapshot (12:00) and the token were copied from &lt;code&gt;agent-1&lt;/code&gt; with a matching sha256 and restored with the same k3s version (&lt;code&gt;--cluster-reset --cluster-reset-restore-path&lt;/code&gt;, then a normal start) in a container on a third machine, with the agent disabled so nothing from the snapshot could run:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F1cirdl834d0pe1e7t2qc.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F1cirdl834d0pe1e7t2qc.png" alt="etcd restore test output: the snapshot fetched from agent-1 with a matching sha256, live and restored object counts, the &amp;quot;Managed etcd cluster membership has been reset&amp;quot; message, and only the k3s process and its supervisor inside the container." width="800" height="512"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The 12:00 snapshot copy restored by k3s v1.33.5 with no agent.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Restored versus live: namespaces 28/28, deployments 68/68, statefulsets 22/22, cronjobs 29/30, secrets 145/147, configmaps 153/154, persistentvolumes 32/38. Every difference is an object created after the 12:00 snapshot: the new CronJob, two Secrets (the copy's SSH key and one from an unrelated change that day), the copy's known_hosts ConfigMap and the six new PVCs. The only processes in the container were &lt;code&gt;k3s&lt;/code&gt; and its supervisor.&lt;/p&gt;

&lt;p&gt;Three earlier attempts failed on the test script, not the copy; you may hit them with check 6:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;kubectl exec ... cat&lt;/code&gt; of the snapshot returned 43,974,656 bytes instead of 45,989,920; &lt;code&gt;kubectl cp&lt;/code&gt; with a sha256 check worked.&lt;/li&gt;
&lt;li&gt;With &lt;code&gt;--network none&lt;/code&gt; or an internal Docker network, k3s exited with &lt;code&gt;no default routes found&lt;/code&gt;; the default bridge network worked.&lt;/li&gt;
&lt;li&gt;With &lt;code&gt;--token-file&lt;/code&gt;, k3s exited with &lt;code&gt;/var/lib/rancher/k3s/server/token does not exist, please pass --token&lt;/code&gt;; &lt;code&gt;K3S_TOKEN&lt;/code&gt; worked.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;What we have not changed:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;No off-site copy.&lt;/strong&gt; Both copies are on a second computer in the same house: a disk failure on &lt;code&gt;server-1&lt;/code&gt; no longer takes the backups with it, but a fire, theft or power event that takes both machines would. This is a practice cluster, and I chose to stop at the second machine.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The old dump PVCs on &lt;code&gt;server-1&lt;/code&gt; are kept.&lt;/strong&gt; They hold the dumps taken before the change and are no longer written to.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Two dumps were not restore-tested in this change.&lt;/strong&gt; &lt;code&gt;app-a&lt;/code&gt; and &lt;code&gt;app-b-postgres-3&lt;/code&gt; were already on &lt;code&gt;agent-1&lt;/code&gt;; the restore test covered the five that moved.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Eight PVs are &lt;code&gt;Delete&lt;/code&gt;.&lt;/strong&gt; Read at 18:21 after the change, 30 PVs are &lt;code&gt;Retain&lt;/code&gt; and 8 are &lt;code&gt;Delete&lt;/code&gt;: the six new backup claims, which got &lt;code&gt;Delete&lt;/code&gt; from the StorageClass default, and one volume each in the logging and monitoring namespaces.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The snapshot copy holds every Secret and the cluster CA keys&lt;/strong&gt;, so access to &lt;code&gt;agent-1&lt;/code&gt;'s disk now matters as much as access to &lt;code&gt;server-1&lt;/code&gt;'s (inference).&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Where else this happened
&lt;/h2&gt;

&lt;p&gt;The &lt;a href="https://benchlet.com/incidentlib/app" rel="noopener noreferrer"&gt;Incident Library&lt;/a&gt;, a searchable collection of public incident reports, has three that show what happened elsewhere when a control plane or a data store depended on a single disk, node, zone or pod; the first is the disk failure that &lt;code&gt;server-1&lt;/code&gt; is exposed to:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;An individual engineer's blog, 2023-05-07 (IL-0185): a disk failure made the first control plane node of a Kubespray cluster read-only, and recovery needed the inventory reordered and a new control plane node joined.&lt;/li&gt;
&lt;li&gt;Devsisters, 2021-02-19 (IL-0172): an AWS Tokyo cooling failure took down 6 of 60 CockroachDB nodes on Kubernetes in 22 minutes; ranges lost quorum because no availability zone locality was configured, and a table backup taken from the surviving nodes did not work, which is the failure that checks 5 and 6 are there to catch before you need the backup.&lt;/li&gt;
&lt;li&gt;Palark, 2020-03-13 (IL-0196): one RedisFailover change restarted the only Redis pod that held data, and Sentinel promoted the empty new replica, erasing the data; as with the five dumps on &lt;code&gt;server-1&lt;/code&gt;, the data was in one place.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The library lists &lt;a href="https://benchlet.com/incidentlib/app?component=etcd&amp;amp;factor=single-point-of-failure" rel="noopener noreferrer"&gt;5 reports where etcd was a single point of failure&lt;/a&gt;, IL-0185 among them, and &lt;a href="https://benchlet.com/incidentlib/app?symptom=data-loss" rel="noopener noreferrer"&gt;2 where data was lost&lt;/a&gt;, the Devsisters and Palark cases above.&lt;/p&gt;

&lt;h2&gt;
  
  
  Timeline (UTC)
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;When&lt;/th&gt;
&lt;th&gt;What&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;2025-11-11&lt;/td&gt;
&lt;td&gt;First backup CronJob created; its dump volume is created on the database's node&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2026-07-02&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;app-a-postgres-backup&lt;/code&gt; placed on &lt;code&gt;agent-1&lt;/code&gt; on purpose, the only one&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2026-11-11 02:50–03:50&lt;/td&gt;
&lt;td&gt;All seven backup Jobs &lt;code&gt;Complete&lt;/code&gt;; five dumps on the database's disk&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2026-11-11 05:08&lt;/td&gt;
&lt;td&gt;kubegap audit lists backups and etcd snapshots that never leave the node&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2026-11-12 17:20–17:41&lt;/td&gt;
&lt;td&gt;Dump Jobs pinned to &lt;code&gt;agent-1&lt;/code&gt;, etcd copy CronJob added; first copy blocked by ufw, then done&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2026-11-12 17:42–17:55&lt;/td&gt;
&lt;td&gt;Five moved dumps and the etcd copy restored from &lt;code&gt;agent-1&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  FAQ
&lt;/h2&gt;

&lt;h3&gt;
  
  
  kubernetes cron job for postgres db backup
&lt;/h3&gt;

&lt;p&gt;A Kubernetes CronJob pg_dump backup, like a Kubernetes CronJob mysqldump backup, writes its dump to a PVC. With local-path, the claim is created on the first Job pod's node, possibly the database's. Pin the Job with &lt;code&gt;nodeSelector&lt;/code&gt; to another node with a new claim: an existing local-path PV stays on its node. Or upload it off the cluster.&lt;/p&gt;

&lt;h3&gt;
  
  
  kubernetes rancher local path specify node
&lt;/h3&gt;

&lt;p&gt;The StorageClass does not choose the node. With &lt;code&gt;volumeBindingMode: WaitForFirstConsumer&lt;/code&gt;, local-path creates the volume on the node where the first pod using the claim is scheduled, and then pins the PV there with &lt;code&gt;kubernetes.io/hostname&lt;/code&gt; node affinity. To choose the node, put &lt;code&gt;nodeSelector&lt;/code&gt; or node affinity on that first pod before the claim is bound.&lt;/p&gt;

&lt;h3&gt;
  
  
  where does k3s local-path store data
&lt;/h3&gt;

&lt;p&gt;In &lt;code&gt;/var/lib/rancher/k3s/storage&lt;/code&gt; on the node that holds the volume (&lt;code&gt;${HOME}/.rancher/k3s/storage&lt;/code&gt; in rootless mode). The node is recorded in the PV's node affinity. Read it with the &lt;code&gt;kubectl get pv&lt;/code&gt; command in check 1 above. With reclaim policy &lt;code&gt;Delete&lt;/code&gt;, deleting the claim deletes that directory on the node.&lt;/p&gt;

&lt;h3&gt;
  
  
  How do you verify that PostgreSQL backups on Kubernetes can actually be restored?
&lt;/h3&gt;

&lt;p&gt;Run a Job that mounts the dump's claim read-only, starts an empty PostgreSQL in an &lt;code&gt;emptyDir&lt;/code&gt;, restores with &lt;code&gt;psql -v ON_ERROR_STOP=1&lt;/code&gt;, and counts tables. Compare with the same count in the source. Check which node the Job ran on: for a backup meant to survive a disk failure, it must not be the database's node.&lt;/p&gt;

&lt;h3&gt;
  
  
  restore k3s etcd snapshot new machine
&lt;/h3&gt;

&lt;p&gt;Copy the snapshot and the server token file &lt;code&gt;/var/lib/rancher/k3s/server/token&lt;/code&gt; to the new machine. Run &lt;code&gt;k3s server --cluster-reset --cluster-reset-restore-path=&amp;lt;file&amp;gt;&lt;/code&gt; with the token in &lt;code&gt;K3S_TOKEN&lt;/code&gt; (with &lt;code&gt;--token-file&lt;/code&gt;, k3s exited with &lt;code&gt;/var/lib/rancher/k3s/server/token does not exist, please pass --token&lt;/code&gt;), then start k3s without &lt;code&gt;--cluster-reset&lt;/code&gt;. We used the same k3s version. Without the same token the snapshot is unusable.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://benchlet.com/guides/kubernetes-pg-dump-cronjob-backup-same-node-k3s-etcd-snapshot/" rel="noopener noreferrer"&gt;benchlet.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>kubernetes</category>
      <category>backup</category>
      <category>sre</category>
      <category>postmortem</category>
    </item>
    <item>
      <title>Sixteen pods evicted for disk, thirteen of them above the threshold</title>
      <dc:creator>KYUNGHWAN KIM</dc:creator>
      <pubDate>Thu, 12 Nov 2026 00:23:08 +0000</pubDate>
      <link>https://dev.to/kkhdevs/sixteen-pods-evicted-for-disk-thirteen-of-them-above-the-threshold-26ko</link>
      <guid>https://dev.to/kkhdevs/sixteen-pods-evicted-for-disk-thirteen-of-them-above-the-threshold-26ko</guid>
      <description>&lt;p&gt;On 2026-11-08 the kubelet on one of our two nodes evicted 16 pods for &lt;code&gt;ephemeral-storage&lt;/code&gt;, 14 production workloads and two cluster components, in three waves between 13:46 and 16:14 UTC. Only three of the sixteen eviction messages show free space under the kubelet's threshold. The other thirteen say 5.9 to 15.4 GiB were free, and nine of those say 12 GiB or more. None of the 136 containers the audit counted on the cluster had asked for any disk, so the kubelet had little to rank them by except priority and how much each one held on disk at that moment.&lt;/p&gt;

&lt;p&gt;This is a two-node k3s cluster that I run alone for a few side projects. Times are UTC. Object names below are role names, not the real ones.&lt;/p&gt;

&lt;h2&gt;
  
  
  What it looked like
&lt;/h2&gt;

&lt;p&gt;Three days later the failed pods were still there. The control plane garbage-collects terminated pods only past a threshold of 12,500, so on a small cluster they stay until something else deletes them:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ kubectl get pods -A --field-selector=status.phase=Failed
NAMESPACE   NAME                        READY   STATUS    RESTARTS   AGE
prod-a      turn-server-…-k2m9q         0/1     Evicted   0          236d
prod-a      signaling-…-r7t4c           0/1     Evicted   0          3d
prod-b      api-server-…-p3w8d          0/1     Evicted   0          28d
prod-b      api-server-…-z6h2n          0/1     Evicted   0          28d
prod-b      api-server-…-b9v5f          0/1     Evicted   0          9d
prod-b      api-server-…-j4c7s          0/1     Evicted   0          3d
…                                       (23 Failed; the 16 Evicted all on the same node)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Thirteen of the sixteen messages had the same shape; the other three stop after the available value:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;The node was low on resource: ephemeral-storage.
Threshold quantity: 5729877282, available: 15662516Ki.
Container app was using 17556Ki, request is 0, has larger consumption of ephemeral-storage.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The threshold is 5,729,877,282 bytes, 5.34 GiB. The node had 15,662,516 KiB available, 14.9 GiB. The container being evicted held 17 MiB. That is the message on a pod evicted at 14:03; the seven pods evicted between 16:13:49 and 16:14:16 carry "available" values between 12.1 and 15.4 GiB.&lt;/p&gt;

&lt;p&gt;The previous two write-ups in this series start from the same afternoon: the 4.5 GiB image that filled the disk and the signaling server that could not come back because its image had been garbage-collected, then the TURN server that had been crash-looping since September. This one is about the evictions themselves.&lt;/p&gt;

&lt;h2&gt;
  
  
  The three waves
&lt;/h2&gt;

&lt;p&gt;I had the one-minute free-space series from Prometheus and the eviction time of each pod from its last status condition. Put together:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fs4rwk7iul2b96qujkfi7.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fs4rwk7iul2b96qujkfi7.png" alt="Free space on the node's root filesystem on 2026-11-08 with the 16 evictions plotted at the available value from each message; three waves at 13:46, 14:03 and 16:10 to 16:14, with the 5 % threshold and the 16 GiB line drawn in; only three dots are under the threshold" width="800" height="440"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Wave&lt;/th&gt;
&lt;th&gt;Pods&lt;/th&gt;
&lt;th&gt;Free space in the messages&lt;/th&gt;
&lt;th&gt;One-minute series&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;13:46:24–13:46:51&lt;/td&gt;
&lt;td&gt;5&lt;/td&gt;
&lt;td&gt;5.3, 5.9, 5.9, 6.0, 6.1 GiB&lt;/td&gt;
&lt;td&gt;5.3 GiB at 13:46, 2.6 GiB from 13:48; &lt;code&gt;DiskPressure&lt;/code&gt; set at 13:47&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;14:03:48–14:03:56&lt;/td&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;14.9, 15.2 GiB&lt;/td&gt;
&lt;td&gt;14.4 GiB since 13:56; 15.3 at 14:04, 27.3 at 14:05&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;16:10:52–16:14:16&lt;/td&gt;
&lt;td&gt;9&lt;/td&gt;
&lt;td&gt;4.3, 4.5, then 12.1 → 15.4 GiB&lt;/td&gt;
&lt;td&gt;4.5 GiB at 16:10, 2.1 at 16:12–16:13, 14.5 at 16:14, 27.3 at 16:15&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Only the first pod of the first wave and the first two of the third were evicted with free space under the threshold.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why thirteen pods were evicted above the threshold
&lt;/h2&gt;

&lt;p&gt;The node's kubelet configuration:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight conf"&gt;&lt;code&gt;&lt;span class="n"&gt;evictionHard&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;           &lt;span class="n"&gt;imagefs&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;available&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="o"&gt;%&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;nodefs&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;available&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="o"&gt;%&lt;/span&gt;
&lt;span class="n"&gt;evictionMinimumReclaim&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;imagefs&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;available&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="o"&gt;%&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;nodefs&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;available&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="o"&gt;%&lt;/span&gt;
&lt;span class="n"&gt;evictionSoft&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;           &lt;span class="n"&gt;none&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;evictionMinimumReclaim&lt;/code&gt; is the setting I had never looked at. Once a hard threshold has been crossed, the kubelet does not stop evicting as soon as the signal is back above the threshold. It keeps evicting, one pod per loop, until the signal is above the threshold plus the minimum reclaim, both measured as a share of the filesystem. On a 106.7 GiB filesystem that is 5.34 GiB plus 10.67 GiB: eviction continues until 16.0 GiB are free.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fc9de69qkfexfbh4ob5g3.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fc9de69qkfexfbh4ob5g3.png" alt="Diagram of the three bands on a 32 GiB scale: eviction starts below 5.34 GiB, continues up to 16.0 GiB because of the 10 % minimum reclaim, stops above; the 16 evictions plotted by wave" width="799" height="373"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Both later waves ended within a minute of free space jumping past 16.0 GiB: the last evictions are at 14:03:56 and 16:14:16, the jumps show in the 14:05 (15.3 to 27.3 GiB) and 16:15 (14.5 to 27.3 GiB) samples. The evictions themselves freed almost nothing; the thirteen messages that state a usage add up to 182 MiB. What freed 12 GiB in a minute each time is not in any record I have. The first write-up put it down to the kubelet reclaiming unused images, which is what it does before it evicts, and the shape of the series is consistent with that. The image garbage collector's own high-water mark, 85 % used, is the same 16.0 GiB on this disk. After 14:05 the threshold was cleared, and from 14:20 the node sat at 12.9 GiB, above 5.34 and below 16.0, with no evictions until the second dip.&lt;/p&gt;

&lt;p&gt;The upstream default for &lt;code&gt;evictionMinimumReclaim&lt;/code&gt; is empty, which means "stop at the threshold". k3s sets 5 % thresholds and a 10 % minimum reclaim in its agent defaults, and this node runs them unchanged; I had not set either. On a node where the image store and the root filesystem are the same disk, 10 % is 10.67 GiB, and the 42 pods on this node hold 0.2 GiB between them today, and the sixteen evicted containers held 182 MiB in total, so evicting pods was never going to reach it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Nothing asked for disk
&lt;/h2&gt;

&lt;p&gt;The last line of the thirteen messages that have one ends with &lt;code&gt;request is 0&lt;/code&gt;. The audit counted what the scheduler and the kubelet actually knew about disk on this cluster:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;running containers                           136
with an ephemeral-storage request              0
with an ephemeral-storage limit                0
LimitRange objects in any namespace            0
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;With no requests, the scheduler places pods without looking at disk at all. When the kubelet has to pick what to evict under disk pressure it sorts pods three ways: pods whose usage exceeds their request first, then by priority, then by how far usage exceeds the request. With every request at zero, every pod holding anything on disk is in the first group, and the order inside it is priority and then whoever holds the most. One more rule sits in front of all of that: a pod the kubelet has no stats for ranks ahead of every pod it can measure. The one pod in wave 1 and the two in wave 3 whose messages carry no usage line are, I infer from the ranking code, exactly those, which is why at 16:10 and 16:12, with two runners still on the node, the signaling and TURN servers went first. The evicted containers held between 0.7 MiB and 37 MiB. The things actually consuming the disk, a 4.5 GiB model image being unpacked and, later, two CI runners building at once, are not things pod eviction can free: the image is node-level storage, and the runners are the pods holding the most, which puts them first in the sort only if they are still there when the kubelet looks.&lt;/p&gt;

&lt;p&gt;The CI runners run as a RunnerDeployment with a docker-in-docker sidecar, whose image layers go into the sidecar's writable layer and whose checkouts go into an &lt;code&gt;emptyDir&lt;/code&gt;, both on the node's root disk. They had CPU and memory requests and limits and nothing for storage, and no node selector, so a runner lands on whichever node has room, including the one that serves production. The audit I used to find this listed the runners as the probable cause of the 11-08 dip. For the first dip the Prometheus series does not support that: no runner pod was on the node between 13:30 and 13:55, and the free-space curve follows the model image deploy described in the first write-up; that attribution is an inference from timing, not a measurement. For the second dip it points the other way: two runner pods sat on the node from 16:04 to 16:13, the only time that day two were there at once, and free space went from 10.3 GiB at 16:07 to 2.1 GiB at 16:12. What they wrote is not recorded; the runners' own records are gone with the pods.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I could not see
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Per-container disk use on 11-08. &lt;code&gt;container_fs_usage_bytes&lt;/code&gt; has no series on this k3s install, and the kubelet's eviction counters were reset by a node reboot two days later. The eviction messages are the only per-pod record.&lt;/li&gt;
&lt;li&gt;Why the kubelet waited from 13:56, when free space was back at 14.4 GiB, until 14:03 to evict the next two pods. The kubelet log on the node was not readable from my session. One consequence is measured: the replacement pods created at 13:46 did not start until 14:09, so the node admitted nothing for 23 minutes.&lt;/li&gt;
&lt;li&gt;What fills the disk today. The kubelet's stats summary says the root filesystem has 72.9 GiB used, of which the image store accounts for 35.0 GiB and all 42 pods' ephemeral storage for 0.2 GiB. The node image list adds up to 15.3 GiB, so about 20 GiB of the image store belongs to nothing in the image list, which I expect to be containerd snapshots and content I cannot read, and about 38 GiB more is outside kubelet accounting entirely. Breaking that down needs root on the node.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The fix
&lt;/h2&gt;

&lt;p&gt;Three changes, all in git, applied by Argo CD.&lt;/p&gt;

&lt;p&gt;The runner and its docker-in-docker sidecar now request and limit ephemeral storage. The limits mean the kubelet evicts the runner on its own once the sidecar's writable layer passes 20 GiB, the runner's passes 10 GiB, or the pod as a whole, &lt;code&gt;emptyDir&lt;/code&gt; included, passes 30 GiB, before the node is under pressure. The request means that under pressure a runner that has grown past 5 GiB sorts ahead of pods that are a few MiB over zero:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;resources&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="s"&gt;cpu:&lt;/span&gt; &lt;span class="s"&gt;500m&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;memory:&lt;/span&gt; &lt;span class="s"&gt;1Gi&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;ephemeral-storage:&lt;/span&gt; &lt;span class="s"&gt;2Gi&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="na"&gt;limits&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;   &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="s"&gt;cpu:&lt;/span&gt; &lt;span class="s2"&gt;"2"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;  &lt;span class="s"&gt;memory:&lt;/span&gt; &lt;span class="s"&gt;4Gi&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;ephemeral-storage:&lt;/span&gt; &lt;span class="s"&gt;10Gi&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="na"&gt;dockerdContainerResources&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="s"&gt;ephemeral-storage:&lt;/span&gt; &lt;span class="s"&gt;5Gi&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="na"&gt;limits&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;   &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="s"&gt;ephemeral-storage:&lt;/span&gt; &lt;span class="s"&gt;20Gi&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The two GPU workloads request 2 Gi each. With a 2 Gi request and a few MiB of usage they no longer exceed their request, so the kubelet sorts them after every pod that does; while everything else requests zero, they are the last to go.&lt;/p&gt;

&lt;p&gt;And an alert that fires before the kubelet acts:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;alert&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;NodeRootDiskLow&lt;/span&gt;        &lt;span class="c1"&gt;# &amp;lt; 15 % for 10m, warning
&lt;/span&gt;&lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;alert&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;NodeRootDiskCritical&lt;/span&gt;   &lt;span class="c1"&gt;# &amp;lt;  8 % for  5m, critical → the route that mentions everyone
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;After the sync:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ kubectl get pods -A -o json | … ephemeral-storage requests
ci            runner-…-d8k3p   runner   2Gi / 10Gi
ci            runner-…-d8k3p   docker   5Gi / 20Gi
ci            runner-…-m2r6w   runner   2Gi / 10Gi
ci            runner-…-m2r6w   docker   5Gi / 20Gi
prod-b        embedding-…      app      2Gi / –
prod-c        ai-…             ai       2Gi / –
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fyusrpml42l9ow1v8l0lo.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fyusrpml42l9ow1v8l0lo.png" alt="Thirty days of free space on the node, 30-minute samples, with the 11-08 dip, the kubelet's 5 % line and the new 15 % and 8 % alert lines" width="800" height="400"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Timeline
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;When (UTC)&lt;/th&gt;
&lt;th&gt;What&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;2026-11-08 13:37&lt;/td&gt;
&lt;td&gt;Deployment of a new service updated to a 4.5 GiB image; its pod starts 13:40 (first write-up)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2026-11-08 13:46:24–13:46:51&lt;/td&gt;
&lt;td&gt;Wave 1: five pods evicted, 5.3–6.1 GiB free&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2026-11-08 13:47–14:09&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;DiskPressure=true&lt;/code&gt;; free space 2.6 GiB until 13:55, 14.4 from 13:56, 27.3 at 14:05 (the condition clears about five minutes after free space passed threshold plus minimum reclaim at 14:05)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2026-11-08 14:03:48–14:03:56&lt;/td&gt;
&lt;td&gt;Wave 2: two pods evicted, 14.9–15.2 GiB free&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2026-11-08 16:04–16:13&lt;/td&gt;
&lt;td&gt;Two CI runner pods on the node at once&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2026-11-08 16:10–16:19&lt;/td&gt;
&lt;td&gt;Second &lt;code&gt;DiskPressure&lt;/code&gt; window; 4.5 GiB at 16:10, 2.1 GiB at 16:12–16:13, 27.3 at 16:15&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2026-11-08 16:10:52–16:14:16&lt;/td&gt;
&lt;td&gt;Wave 3: nine pods evicted, 4.3 → 15.4 GiB free&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2026-11-10 06:01&lt;/td&gt;
&lt;td&gt;Node rebooted; eviction counters reset&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2026-11-11 05:08&lt;/td&gt;
&lt;td&gt;Cluster audit lists the 16 Failed pods and the zero ephemeral-storage requests&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2026-11-11 23:45&lt;/td&gt;
&lt;td&gt;ephemeral-storage requests and limits on the runners, requests on the two GPU workloads, two disk alerts, all rolled out&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  What I am changing
&lt;/h2&gt;

&lt;p&gt;No disk alert fired that day. The kube-prometheus-stack chart's &lt;code&gt;NodeFilesystemAlmostOutOfSpace&lt;/code&gt; rule wants free space under 5 % for 30 minutes; on 11-08 it was under 5 % for eight minutes at the longest stretch and thirteen in the day. The kubelet evicts at 5 % immediately; it had finished before the rule's 30 minutes were up.&lt;/p&gt;

&lt;p&gt;Done:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;ephemeral-storage&lt;/code&gt; requests and limits on the CI runner and its docker-in-docker sidecar; requests on the two GPU workloads.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;NodeRootDiskLow&lt;/code&gt; at 15 % for 10 minutes, &lt;code&gt;NodeRootDiskCritical&lt;/code&gt; at 8 % for 5 minutes, the latter on the route that mentions everyone.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Not done yet:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;evictionMinimumReclaim&lt;/code&gt;. It lives in the k3s agent configuration on the node, outside git. With the upstream default, no minimum reclaim, the day would have been three evictions: the first pod of wave 1 and the first two of wave 3. At 2–3 % it would have been seven: all five of wave 1, none of wave 2, two of wave 3. It needs a node-level change and a kubelet restart.&lt;/li&gt;
&lt;li&gt;A node for CI. The runners should not share a disk with production at all. That needs hardware I do not have this month.&lt;/li&gt;
&lt;li&gt;Root on the node to break down the 38 GiB that no pod or image accounts for.&lt;/li&gt;
&lt;li&gt;A default &lt;code&gt;ephemeral-storage&lt;/code&gt; request through a &lt;code&gt;LimitRange&lt;/code&gt; in every production namespace, small enough not to change scheduling, so a new Deployment cannot arrive with zero again; the two cluster components stay outside it.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If your eviction messages show "available" values far above the threshold, read the kubelet's &lt;code&gt;evictionMinimumReclaim&lt;/code&gt; first. &lt;code&gt;kubectl get --raw /api/v1/nodes/&amp;lt;node&amp;gt;/proxy/configz | jq .kubeletconfig.evictionMinimumReclaim&lt;/code&gt; shows it without logging in to the node.&lt;/p&gt;

&lt;p&gt;Public post-incident reports where a node ran out of disk and took unrelated workloads with it are collected under the &lt;a href="https://benchlet.com/incidentlib/app?symptom=node-disk-full" rel="noopener noreferrer"&gt;node disk full&lt;/a&gt; and &lt;a href="https://benchlet.com/incidentlib/app?symptom=pods-evicted" rel="noopener noreferrer"&gt;pods evicted&lt;/a&gt; symptoms in Incident Library. The audit that surfaced this is &lt;a href="https://github.com/KyungHwanKim-devs/kubegap" rel="noopener noreferrer"&gt;kubegap&lt;/a&gt;.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://benchlet.com/guides/diskpressure-evicted-production-pods-ci-runner-same-node/" rel="noopener noreferrer"&gt;benchlet.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>kubernetes</category>
      <category>devops</category>
      <category>sre</category>
      <category>docker</category>
    </item>
    <item>
      <title>A pod that ran for six months crash-looped on its first restart</title>
      <dc:creator>KYUNGHWAN KIM</dc:creator>
      <pubDate>Wed, 11 Nov 2026 12:30:56 +0000</pubDate>
      <link>https://dev.to/kkhdevs/a-pod-that-ran-for-six-months-crash-looped-on-its-first-restart-2cii</link>
      <guid>https://dev.to/kkhdevs/a-pod-that-ran-for-six-months-crash-looped-on-its-first-restart-2cii</guid>
      <description>&lt;p&gt;Our TURN server, the relay that lets two browsers talk when neither can be reached directly, had zero available replicas from about 2026-09-28 until 2026-11-11 11:58 UTC. The container died within a second of starting, every five minutes, about 280 times a day, for around 45 days. Its Deployment had not been touched since March. The alert for it fired on every day Prometheus still has.&lt;/p&gt;

&lt;p&gt;This is a two-node k3s cluster that I run alone for a few side projects. Times are UTC.&lt;/p&gt;

&lt;h2&gt;
  
  
  What it looked like
&lt;/h2&gt;

&lt;p&gt;A read-only audit of the cluster (&lt;a href="https://github.com/KyungHwanKim-devs/kubegap" rel="noopener noreferrer"&gt;kubegap&lt;/a&gt;) that I ran on 2026-11-11 at 05:08 for a different problem listed this one next to it: a Deployment in a production namespace with &lt;code&gt;0/1&lt;/code&gt; ready and a pod in &lt;code&gt;CrashLoopBackOff&lt;/code&gt; with 718 restarts. By 11:40, when I got to it, the number was 794.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ kubectl -n dam-prod get pod -l app=dam-coturn
NAME                          READY   STATUS                   RESTARTS            AGE
dam-coturn-549854dfcb-bchx5   0/1     CrashLoopBackOff         794 (2m11s ago)     2d19h
dam-coturn-549854dfcb-c8mfp   0/1     ContainerStatusUnknown   28 (2d19h ago)      2d21h
dam-coturn-549854dfcb-jgp4h   0/1     ContainerStatusUnknown   11621 (2d21h ago)   235d
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Three pods. The one created in March with 11,621 restarts, a replacement from 2026-11-08 with 28, and the current one with 794. The two older ones had been evicted on 11-08 when the node ran out of disk (that is the previous write-up); the eviction did not cause this, it only produced a new pod that failed the same way.&lt;/p&gt;

&lt;p&gt;The previous container's log had the answer in its first line, followed by the program's entire usage text:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ kubectl -n dam-prod logs dam-coturn-549854dfcb-bchx5 --previous | head -4
turnserver: unrecognized option '--no-dtls'
2026-11-11T11:38:04.215+0000 WARNING Cannot find config file: turnserver.conf. ...
2026-11-11T11:38:04.215+0000 WARNING Cannot find config file: turnserver.conf. ...
turnserver: unrecognized option '--no-dtls'
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Started 11:38:04, finished 11:38:04, exit code 255. The binary did not know a flag that was in the Deployment's &lt;code&gt;args&lt;/code&gt;, so it printed help and quit. A few hundred lines further down, the same usage text showed what had replaced it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt; --no-tls          Do not start TLS client listeners.
 --dtls            Start DTLS client listeners. DTLS is not started by default.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The alert had been firing since before the retention window. The Alertmanager list that morning had six groups, five of them routed to the Discord receiver and two of those for this namespace:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fu9psw6ydb8g2whqnr47y.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fu9psw6ydb8g2whqnr47y.png" alt="Alertmanager alert list before the fix: six groups, five routed to discord, including KubePodCrashLooping and KubeDeploymentReplicasMismatch for namespace dam-prod"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Which binary was this?
&lt;/h2&gt;

&lt;p&gt;The Deployment said &lt;code&gt;image: coturn/coturn:latest&lt;/code&gt; with &lt;code&gt;imagePullPolicy: Always&lt;/code&gt;. The pod's status said what that had resolved to:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Image ID:  docker.io/coturn/coturn@sha256:bbefd3e1fdfd...
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Docker Hub's tag API maps that digest to &lt;code&gt;4.18.0&lt;/code&gt;, pushed 2026-09-08 11:41, and &lt;code&gt;latest&lt;/code&gt; has pointed at it since 11:42, five seconds later. So the running binary was coturn 4.18.0, and the flag it rejected had to have been removed somewhere between whatever version the pod had pulled in March and 4.18.0.&lt;/p&gt;

&lt;p&gt;The coturn ChangeLog has one line about DTLS in 4.17.0: "Start the DTLS listeners only when --dtls is given (#2026)". That pull request, merged 2026-08-02, made DTLS opt-in but kept &lt;code&gt;--no-dtls&lt;/code&gt; as a deprecated alias that logs a warning. The source at the 4.17.0 tag still has the entry:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight c"&gt;&lt;code&gt;&lt;span class="cm"&gt;/* deprecated: */&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"no-dtls"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;optional_argument&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kc"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="no"&gt;NO_DTLS_OPT&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;At the 4.18.0 tag the line is gone. The commit that removed it is &lt;a href="https://github.com/coturn/coturn/pull/2079" rel="noopener noreferrer"&gt;coturn/coturn#2079&lt;/a&gt;, merged 2026-09-07 22:43, whose title and ChangeLog entry read "Update help with missing arguments". The body says what it does: "Remove the deprecated &lt;code&gt;--no-dtls&lt;/code&gt; and &lt;code&gt;--no-cli&lt;/code&gt; turnserver flags — both were already no-ops". Thirteen hours later Docker Hub's &lt;code&gt;latest&lt;/code&gt; moved to the image built from it. (The PR says it removed &lt;code&gt;--no-cli&lt;/code&gt; too, but its diff only drops that flag from the help text and the test scripts, not from the option table; the 4.18.0 binary still accepts it and logs an &lt;code&gt;ERROR&lt;/code&gt; line about it. More on that below.)&lt;/p&gt;

&lt;h2&gt;
  
  
  When did it start?
&lt;/h2&gt;

&lt;p&gt;Prometheus on this cluster keeps 30 days. The oldest sample for the March pod's restart counter is from 10-12 11:41, and it already reads 4,014. From there the counter climbs by 280 or 281 a day, 284 or 285 on two days, until the eviction on 11-08. Condensed from a one-day-step range query:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight prometheus"&gt;&lt;code&gt;&lt;span class="na"&gt;kube_pod_container_status_restarts_total{pod&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"dam-coturn-...-jgp4h"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;12&lt;/span&gt;   &lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;014&lt;/span&gt;
&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;13&lt;/span&gt;   &lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;294&lt;/span&gt;     &lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mi"&gt;280&lt;/span&gt;
&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;14&lt;/span&gt;   &lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;574&lt;/span&gt;     &lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mi"&gt;280&lt;/span&gt;
&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;
&lt;span class="mi"&gt;11&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;07&lt;/span&gt;  &lt;span class="mi"&gt;11&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;314&lt;/span&gt;
&lt;span class="mi"&gt;11&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;08&lt;/span&gt;  &lt;span class="mi"&gt;11&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;594&lt;/span&gt;
&lt;span class="mi"&gt;11&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;09&lt;/span&gt;  &lt;span class="mi"&gt;11&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;621&lt;/span&gt;     &lt;span class="n"&gt;evicted&lt;/span&gt; &lt;span class="mi"&gt;11&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;08&lt;/span&gt; &lt;span class="mi"&gt;13&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;46&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;counter&lt;/span&gt; &lt;span class="n"&gt;frozen&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;280 a day is one cycle every 308 s (86,400 ÷ 308 = 280.5): the five-minute back-off cap, plus about eight seconds for the kubelet to check the tag with the registry, create the container, watch it exit within a second, and notice. If the slope was the same before the retention window and the counter was near zero before that, 4,014 restarts take 14.3 days, which puts the first failure at about 2026-09-28, three weeks after &lt;code&gt;latest&lt;/code&gt; changed. A few restarts could predate it; the date is an estimate. The hourly series in the chart below starts an hour later and reads 4,018; its last point for the current pod is 786, the last scrape before kubectl read 794.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fxtj5tegxsf94mrhuolcw.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fxtj5tegxsf94mrhuolcw.png" alt="Restart counter of the three dam-coturn pods over the 30-day retention window, with the same slope continued back to zero at about 2026-09-28 and the 2026-09-08 marker where Docker Hub latest moved to 4.18.0"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;I cannot say why the container restarted on that day after running since 2026-03-20 with, as far as the counter arithmetic shows, no restarts before then. The node has rebooted since (current boot 11-10) and nothing older than 30 days is in Prometheus. One restart was enough: with &lt;code&gt;imagePullPolicy: Always&lt;/code&gt; the kubelet asks the registry what the tag points to on every container start, restarts included, and downloads only if the digest changed. The events for the current pod show it, &lt;code&gt;Pulling image "coturn/coturn:latest"&lt;/code&gt; 351 times in 29 hours, one per restart. From 09-08 on the answer to that pull was a binary that would not accept the Deployment's arguments.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ft62qphcgr17cda4xiu8n.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ft62qphcgr17cda4xiu8n.png" alt="Two timelines, upstream and cluster, side by side: the flag deprecated in August and removed in September upstream; the pod started in March, did not restart until late September, and then pulled the new image"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;There was nothing to roll back to, either. The ReplicaSet is the one from March, so no &lt;code&gt;imageID&lt;/code&gt; for the March container exists anywhere, and Prometheus does not keep container info that long. Which version had been running for six months is unknown.&lt;/p&gt;

&lt;h2&gt;
  
  
  The fix
&lt;/h2&gt;

&lt;p&gt;One commit, which Argo CD applied within the minute:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;strategy&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;type&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Recreate&lt;/span&gt;
  &lt;span class="na"&gt;template&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;containers&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;coturn&lt;/span&gt;
          &lt;span class="na"&gt;image&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;coturn/coturn:4.18.0@sha256:bbefd3e1fdfd...&lt;/span&gt;
          &lt;span class="na"&gt;imagePullPolicy&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;IfNotPresent&lt;/span&gt;
          &lt;span class="na"&gt;args&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="c1"&gt;# --no-dtls removed
&lt;/span&gt;            &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="s"&gt;--no-multicast-peers&lt;/span&gt;
            &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="s"&gt;--denied-peer-ip=10.0.0.0-10.255.255.255&lt;/span&gt;
            &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="s"&gt;--denied-peer-ip=172.16.0.0-172.31.255.255&lt;/span&gt;
            &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="s"&gt;--denied-peer-ip=192.168.0.0-192.168.255.255&lt;/span&gt;
            &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="s"&gt;--denied-peer-ip=127.0.0.0-127.255.255.255&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The pin is to the digest that was already on the node, so nothing new had to be pulled. The &lt;code&gt;denied-peer-ip&lt;/code&gt; ranges were overdue regardless: this relay was coming back onto the internet after six weeks off, and a TURN server that will relay to private addresses, including the relay's own LAN, is a way into the network behind it.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;Recreate&lt;/code&gt; is there because of the host port. This Deployment uses &lt;code&gt;hostNetwork&lt;/code&gt; and declares &lt;code&gt;hostPort: 3478&lt;/code&gt; for UDP and TCP, and the pod is pinned to one node by a &lt;code&gt;nodeSelector&lt;/code&gt;. I did not try a &lt;code&gt;RollingUpdate&lt;/code&gt;, but the arithmetic says it would not have finished: with one replica, &lt;code&gt;maxUnavailable: 25%&lt;/code&gt; rounds down to 0 and &lt;code&gt;maxSurge: 25%&lt;/code&gt; rounds up to 1, so the controller has to create the new pod before it may delete the old one; the scheduler will not place a second pod that needs port 3478 on a node where a pod already holds it, crashing or not, and there is no other node it may use. The new pod would sit in &lt;code&gt;Pending&lt;/code&gt; and the old one would keep crashing. &lt;code&gt;Recreate&lt;/code&gt; deletes first. Every future rollout will have a gap of about 15 s (measured on the rollouts today); with one replica there was never going to be a rollout without one.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ kubectl -n dam-prod get events --sort-by=.lastTimestamp | grep coturn   (condensed)
Scaled down replica set dam-coturn-549854dfcb from 1 to 0
Container image "coturn/coturn:4.18.0@sha256:bbefd3e1fdfd..." already present on machine
Started container coturn
Scaled up replica set dam-coturn-6f99fd45d5 from 0 to 1

$ kubectl -n dam-prod logs deploy/dam-coturn | grep -E "Version|Black|Relay address"
INFO Black listing: 10.0.0.0-10.255.255.255
INFO Black listing: 172.16.0.0-172.31.255.255
INFO Black listing: 192.168.0.0-192.168.255.255
INFO Black listing: 127.0.0.0-127.255.255.255
INFO Relay address to use: 192.168.0.101
INFO Coturn Version Coturn-4.18.0 'Gorst'
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A STUN Binding request to port 3478 got no reply at 11:42 and a &lt;code&gt;Binding Success&lt;/code&gt; at 11:58:21, both from the LAN and through the router's public address. That checks that the server answers, not that it relays; I did not run an Allocate with the credentials.&lt;/p&gt;

&lt;p&gt;Two lines in the new log needed a second commit. &lt;code&gt;ERROR no-cli option is deprecated, see --cli&lt;/code&gt; means that flag will go the way &lt;code&gt;--no-dtls&lt;/code&gt; went. And &lt;code&gt;WARNING NO EXPLICIT LISTENER ADDRESS(ES) ARE CONFIGURED&lt;/code&gt; was followed by the server opening 3478 on every interface the host has, including the cluster's and the VPN's. The second commit dropped &lt;code&gt;--no-cli&lt;/code&gt; and added &lt;code&gt;--listening-ip&lt;/code&gt; for the one address the relay should bind; the next pod's log lists listeners on that address only. A third added the link-local, 100.64/10 and 0/8 ranges to the denied list, since the VPN gives this node an address in 100.64/10. One line, &lt;code&gt;ERROR CONFIG: Unknown argument:&lt;/code&gt; with nothing after the colon, is still there and I have not found what produces it.&lt;/p&gt;

&lt;p&gt;Fifteen minutes later neither &lt;code&gt;KubePodCrashLooping&lt;/code&gt; nor &lt;code&gt;KubeDeploymentReplicasMismatch&lt;/code&gt; was active for the namespace, in Prometheus or in Alertmanager:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fykfbjgczn84plwiraepr.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fykfbjgczn84plwiraepr.png" alt="Alertmanager alert list after the fix: the two dam-prod groups are gone; KubeCPUOvercommit, KubeMemoryOvercommit, Watchdog and two KubeJobFailed remain"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Timeline
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;When (UTC)&lt;/th&gt;
&lt;th&gt;What&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;2026-03-19&lt;/td&gt;
&lt;td&gt;Deployment written: &lt;code&gt;coturn/coturn:latest&lt;/code&gt;, &lt;code&gt;imagePullPolicy: Always&lt;/code&gt;, &lt;code&gt;--no-dtls&lt;/code&gt; in args&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2026-03-20 15:58&lt;/td&gt;
&lt;td&gt;Pod starts&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2026-08-02&lt;/td&gt;
&lt;td&gt;coturn PR #2026 merged (4.17.0): DTLS opt-in, &lt;code&gt;--no-dtls&lt;/code&gt; deprecated but accepted&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2026-09-07 22:43&lt;/td&gt;
&lt;td&gt;coturn PR #2079 merged (4.18.0): &lt;code&gt;--no-dtls&lt;/code&gt; removed. ChangeLog line: "Update help with missing arguments"&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2026-09-08 11:42&lt;/td&gt;
&lt;td&gt;Docker Hub &lt;code&gt;latest&lt;/code&gt; → 4.18.0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;≈ 2026-09-28&lt;/td&gt;
&lt;td&gt;First restart (estimated from the counter slope). Kubelet pulls &lt;code&gt;latest&lt;/code&gt;, binary rejects &lt;code&gt;--no-dtls&lt;/code&gt;, exit 255. &lt;code&gt;CrashLoopBackOff&lt;/code&gt; at ~280 a day from here on&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2026-10-12&lt;/td&gt;
&lt;td&gt;Oldest sample Prometheus still has: counter 4,014, &lt;code&gt;KubePodCrashLooping&lt;/code&gt; already firing&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2026-11-08 13:46&lt;/td&gt;
&lt;td&gt;Node &lt;code&gt;DiskPressure&lt;/code&gt;; pod evicted. Two replacements fail the same way&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2026-11-11 05:08&lt;/td&gt;
&lt;td&gt;Cluster audit lists the Deployment with 0 available and the log line&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2026-11-11 11:57&lt;/td&gt;
&lt;td&gt;Manifest commit: digest pin, flag removed, &lt;code&gt;Recreate&lt;/code&gt;, denied peer ranges&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2026-11-11 11:58&lt;/td&gt;
&lt;td&gt;New pod running; STUN answers. About 45 days&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2026-11-11 12:08&lt;/td&gt;
&lt;td&gt;Second commit (12:00) rolled out: &lt;code&gt;--no-cli&lt;/code&gt; gone, &lt;code&gt;--listening-ip&lt;/code&gt; set. Third (12:19): more denied ranges&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  What I am changing
&lt;/h2&gt;

&lt;p&gt;The alert fired for the whole window. &lt;code&gt;KubePodCrashLooping&lt;/code&gt; for this Deployment's pods is in every sample Prometheus has. In Prometheus the firing set on 10-12 and on 11-11 was the same list: five &lt;code&gt;CPUThrottlingHigh&lt;/code&gt;, &lt;code&gt;KubeCPUOvercommit&lt;/code&gt;, &lt;code&gt;KubeDeploymentReplicasMismatch&lt;/code&gt;, two &lt;code&gt;KubeJobFailed&lt;/code&gt;, &lt;code&gt;KubePodCrashLooping&lt;/code&gt;, &lt;code&gt;KubeQuotaFullyUsed&lt;/code&gt;, plus &lt;code&gt;KubeMemoryOvercommit&lt;/code&gt; from 11-09. Alertmanager inhibits the throttling and quota ones, so what reached Discord was the overcommit alerts, the job failures and this crash loop, re-sent on the route's repeat interval. The crash-loop alert even arrived as a fresh notification three times, once per pod name, and was still not acted on. The previous incident's alert was new and went unnoticed among these standing ones. This one was one of the standing ones.&lt;/p&gt;

&lt;p&gt;Done:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Image pinned by digest, &lt;code&gt;imagePullPolicy: IfNotPresent&lt;/code&gt;, &lt;code&gt;strategy: Recreate&lt;/code&gt; for the host-network workload.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;--no-dtls&lt;/code&gt; and &lt;code&gt;--no-cli&lt;/code&gt; gone; &lt;code&gt;--denied-peer-ip&lt;/code&gt; for RFC 1918, loopback, link-local, 100.64/10 and 0/8, &lt;code&gt;--no-multicast-peers&lt;/code&gt;, &lt;code&gt;--listening-ip&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Not done yet:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The other 14 images on this cluster referenced by a mutable tag. Each gets the same treatment: digest, &lt;code&gt;IfNotPresent&lt;/code&gt;, and a deliberate upgrade step instead of an accidental one.&lt;/li&gt;
&lt;li&gt;The TURN credentials. The relay still authenticates with a static username and password that are compiled into the web client. The standard answer is &lt;code&gt;--use-auth-secret&lt;/code&gt; with short-lived credentials minted by the signaling server, which is an application change, not a manifest change.&lt;/li&gt;
&lt;li&gt;The standing alerts, again. Fix or silence each one so that a firing alert means something; page on &lt;code&gt;available == 0&lt;/code&gt; for anything with a public hostname.&lt;/li&gt;
&lt;li&gt;A startup-log check: grep the first lines of a new container's log for &lt;code&gt;ERROR&lt;/code&gt; and &lt;code&gt;deprecated&lt;/code&gt; and fail the rollout on a match.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For any Deployment on a mutable tag that has been up for months, record what is actually running before anything restarts it: &lt;code&gt;kubectl get pod -o jsonpath='{.status.containerStatuses[*].imageID}'&lt;/code&gt;. The tag in the manifest does not tell you.&lt;/p&gt;

&lt;p&gt;Public post-incident reports that start the same way, a container restarting until something upstream of it is changed, are collected under the &lt;a href="https://benchlet.com/incidentlib/app?symptom=crash-loop" rel="noopener noreferrer"&gt;crash loop&lt;/a&gt; symptom in Incident Library; the ones where an upgrade removed something the configuration relied on are under &lt;a href="https://benchlet.com/incidentlib/app?root_cause=upgrade-breaking-change" rel="noopener noreferrer"&gt;breaking change in an upgrade&lt;/a&gt;.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://benchlet.com/guides/crashloopbackoff-coturn-unrecognized-option-no-dtls/" rel="noopener noreferrer"&gt;benchlet.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>kubernetes</category>
      <category>devops</category>
      <category>docker</category>
      <category>sre</category>
    </item>
    <item>
      <title>ImagePullBackOff on an image nobody had touched since March</title>
      <dc:creator>KYUNGHWAN KIM</dc:creator>
      <pubDate>Wed, 11 Nov 2026 10:14:33 +0000</pubDate>
      <link>https://dev.to/kkhdevs/imagepullbackoff-on-an-image-nobody-had-touched-since-march-5gg3</link>
      <guid>https://dev.to/kkhdevs/imagepullbackoff-on-an-image-nobody-had-touched-since-march-5gg3</guid>
      <description>&lt;p&gt;Our WebRTC signaling server was unavailable for 2 days, 17 hours and 55 minutes, from 2026-11-08 13:46 to 2026-11-11 07:41 UTC. The last commit to its code was on 2026-03-23 and the Deployment had pointed at the same image tag since then. The outage started with a disk filling up on the node. It lasted because of what a registry cron job had done to that image on 2026-03-24.&lt;/p&gt;

&lt;p&gt;This is a two-node k3s cluster that I run alone for a few side projects. Times are UTC.&lt;/p&gt;

&lt;h2&gt;
  
  
  What it looked like
&lt;/h2&gt;

&lt;p&gt;By the morning of 2026-11-11 the Deployment's ReplicaSet had three pods. The original, created in March, had ended on 2026-11-08 at 13:46:42 with exit code 0 and phase &lt;code&gt;Succeeded&lt;/code&gt;. A replacement created one second later had never run and ended with &lt;code&gt;ContainerStatusUnknown&lt;/code&gt;, exit code 137. A second replacement, created at 16:10:52, was &lt;code&gt;Pending&lt;/code&gt; with this event repeating every five minutes:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Failed to pull image "registry.example.com/dam-signaling:prod-&amp;lt;sha&amp;gt;":
  failed to pull and unpack image ...:
  .../manifests/sha256:211614369c9d... : not found
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The audit tool I mention below counted 1,565 of these events in 24 hours on that node. On the registry side, the server log had 1,068 requests for that manifest digest since 2026-11-09 23:58, every one answered &lt;code&gt;404 manifest unknown&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;The usual causes of &lt;code&gt;manifest unknown&lt;/code&gt; did not fit. The tag existed in the registry's filesystem. The same node had pulled other images from the same registry that afternoon. The Deployment had not changed since March.&lt;/p&gt;

&lt;h2&gt;
  
  
  Finding it
&lt;/h2&gt;

&lt;p&gt;I found the outage on 2026-11-11 at 05:08, from a read-only audit of the cluster (&lt;a href="https://github.com/KyungHwanKim-devs/kubegap" rel="noopener noreferrer"&gt;kubegap&lt;/a&gt;) that I had run for another reason. It listed the Deployment with zero available replicas and the pull failures next to it. The investigation then went through the disk, the registry, and the node cache, in that order.&lt;/p&gt;

&lt;h3&gt;
  
  
  The disk
&lt;/h3&gt;

&lt;p&gt;The node's root filesystem is 106.7 GiB. On 2026-11-08 free space went from 19.4 GiB at 12:45 to 2.6 GiB at 13:48. The node reported &lt;code&gt;DiskPressure&lt;/code&gt; from 13:47 to 14:10 and again from 16:10 to 16:20.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight prometheus"&gt;&lt;code&gt;&lt;span class="na"&gt;node_filesystem_avail_bytes{mountpoint&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"/"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;      &lt;span class="nc"&gt;GiB&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;one&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;minute&lt;/span&gt; &lt;span class="n"&gt;samples&lt;/span&gt;
&lt;span class="mi"&gt;12&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;45&lt;/span&gt;  &lt;span class="mf"&gt;19.4&lt;/span&gt;
&lt;span class="mi"&gt;12&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;58&lt;/span&gt;   &lt;span class="mf"&gt;8.1&lt;/span&gt;
&lt;span class="mi"&gt;13&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;07&lt;/span&gt;  &lt;span class="mf"&gt;11.8&lt;/span&gt;
&lt;span class="mi"&gt;13&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;38&lt;/span&gt;  &lt;span class="mf"&gt;11.4&lt;/span&gt;
&lt;span class="mi"&gt;13&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;48&lt;/span&gt;   &lt;span class="mf"&gt;2.6&lt;/span&gt;     &lt;span class="nc"&gt;DiskPressure&lt;/span&gt; &lt;span class="mi"&gt;13&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;47&lt;/span&gt;–&lt;span class="mi"&gt;14&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;
&lt;span class="mi"&gt;13&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;56&lt;/span&gt;  &lt;span class="mf"&gt;14.4&lt;/span&gt;
&lt;span class="mi"&gt;14&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;05&lt;/span&gt;  &lt;span class="mf"&gt;27.3&lt;/span&gt;
&lt;span class="mi"&gt;14&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;18&lt;/span&gt;  &lt;span class="mf"&gt;13.3&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fvlnymly04yc22p1315b2.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fvlnymly04yc22p1315b2.png" alt="Free space on the node root filesystem on 2026-11-08, one-minute samples, with the two DiskPressure windows shaded and the 5 % evictionHard line."&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;I caused this. That afternoon I pushed 25 deploy commits for a new service, and one of them shipped a 4.5 GiB image with the service's ML models inside. The Deployment picked it up at 13:37 and the pod started at 13:40. The kubelet's &lt;code&gt;evictionHard&lt;/code&gt; is &lt;code&gt;imagefs.available&amp;lt;5%&lt;/code&gt; and &lt;code&gt;nodefs.available&amp;lt;5%&lt;/code&gt;; on k3s with a single root filesystem those are the same disk. When a node crosses that threshold the kubelet first reclaims node-level resources, which for an image filesystem means deleting every image no container is using, and then evicts pods if it is still under. The jump from 2.6 GiB to 27.3 GiB free between 13:48 and 14:05 is consistent with that. By 16:15 the node's eviction counters read 23 for &lt;code&gt;imagefs&lt;/code&gt; and 27 for &lt;code&gt;nodefs&lt;/code&gt;, and 16 pods across the cluster carried the &lt;code&gt;Evicted&lt;/code&gt; reason.&lt;/p&gt;

&lt;p&gt;The signaling pod is not among those 16. It ended at 13:46:42, one minute before the &lt;code&gt;DiskPressure&lt;/code&gt; condition was recorded, with exit code 0. That is not what an eviction looks like, and I could not read the kubelet journal on that node that day, so I do not know what stopped it. The sequence I can show is: the container stopped inside the disk-pressure window; the first replacement died during the second window; the second replacement needed an image that was no longer in the node's store.&lt;/p&gt;

&lt;p&gt;The models-in-image problem got the obvious fix the same evening: models on a volume, 1.6 GiB image.&lt;/p&gt;

&lt;h3&gt;
  
  
  The registry
&lt;/h3&gt;

&lt;p&gt;The pull was failing on a tag that had been pulled without trouble in March. The registry is &lt;code&gt;docker/distribution&lt;/code&gt; 2.8.3 with filesystem storage, so I looked at the files:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;repositories/dam-signaling/_manifests/tags/prod-&amp;lt;sha&amp;gt;/current/link
  -&amp;gt; sha256:447fa22a...      OCI image index, 2 children

children of the index
  sha256:211614369c9d...     linux/amd64 image manifest   (what containerd asked for)
  sha256:203e8548...         annotation vnd.docker.reference.type: attestation-manifest

repositories/dam-signaling/_manifests/revisions/sha256/
  447fa22a...                present
  211614369c9d...            missing
  203e8548...                missing
  (directory mtime 2026-03-24 04:00:01)

blobs referenced by the image manifest: 10
  present     4   (base layers shared with other images)
  missing     6   (config + 5 layers)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The tag pointed at an image index. The index was there. Both manifests the index pointed at had been unlinked from the repository, and 6 of the 10 blobs behind the platform manifest were gone. The revisions directory had last been written at 04:00:01 on 2026-03-24, the night after the push.&lt;/p&gt;

&lt;p&gt;04:00 is when the registry's garbage-collection CronJob runs. From mid-March until 2026-06-02 that job ran &lt;code&gt;registry garbage-collect --delete-untagged=true&lt;/code&gt;. I had turned the flag on to reclaim disk. I turned it off after another image broke, and did not go back to check what else the flag had removed.&lt;/p&gt;

&lt;p&gt;What happens, in three parts:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Since buildx 0.10 (January 2023; &lt;code&gt;docker/build-push-action&lt;/code&gt; v4 picked it up), a push attaches a provenance attestation by default. Even for one platform the result is an OCI image index with two children, the platform manifest and the attestation manifest. The tag points at the index. The children have no tag of their own.&lt;/li&gt;
&lt;li&gt;distribution 2.x's garbage collector with &lt;code&gt;--delete-untagged&lt;/code&gt; enumerates manifests, keeps those a tag resolves to, and deletes the rest. It does not descend into an index. It therefore kept the index, deleted both children, and then deleted every blob that only the children referenced.&lt;/li&gt;
&lt;li&gt;Afterwards the tag list is intact and the tag resolves to the index. A client that follows the index to the platform manifest gets &lt;code&gt;manifest unknown&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ffvj0dgdvj9zde7bb326w.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ffvj0dgdvj9zde7bb326w.png" alt="Tag to OCI image index to its two children. The index stays; both children and six of ten blobs were removed by garbage collection with --delete-untagged."&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;This was reported as &lt;a href="https://github.com/distribution/distribution/issues/3004" rel="noopener noreferrer"&gt;distribution/distribution#3004&lt;/a&gt; in 2019 and again as &lt;a href="https://github.com/distribution/distribution/issues/3178" rel="noopener noreferrer"&gt;#3178&lt;/a&gt; in 2020. The fix, &lt;a href="https://github.com/distribution/distribution/pull/4285" rel="noopener noreferrer"&gt;#4285&lt;/a&gt;, was merged in April 2024 and is only in the 3.x releases. 2.8.3, the last 2.x release, predates it.&lt;/p&gt;

&lt;h3&gt;
  
  
  The node cache
&lt;/h3&gt;

&lt;p&gt;containerd on the node had the image, and with the default &lt;code&gt;imagePullPolicy: IfNotPresent&lt;/code&gt; for a non-&lt;code&gt;latest&lt;/code&gt; tag nothing asks the registry for an image that is already present. The pod restarted five times between March and November, most recently on 2026-10-28, each time from the local store. The first client to ask the registry for the platform manifest after March was the replacement pod on 2026-11-08, after the cached copy had been removed.&lt;/p&gt;

&lt;p&gt;Knowing what to look for, I checked every image tag referenced by a running workload against the registry filesystem:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;images referenced by workloads                            40
complete                                                  34
broken (index kept, children unlinked, blobs missing)      6
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;All six had been pushed in March 2026 while the flag was on: four on the 14th between 19:02 and 19:14, two on the 23rd. Every image pushed after May was either a single manifest or an intact index. Five of the six broken ones were still running from the node's cache, with no pullable copy anywhere.&lt;/p&gt;

&lt;h2&gt;
  
  
  The fix
&lt;/h2&gt;

&lt;p&gt;Rebuild from the same commit with two lines added to the workflow:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;uses&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;docker/build-push-action@v6&lt;/span&gt;
  &lt;span class="na"&gt;with&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;provenance&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;
    &lt;span class="na"&gt;platforms&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;linux/amd64&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;With no attestation and one platform, buildx pushes a single &lt;code&gt;application/vnd.oci.image.manifest.v1+json&lt;/code&gt;, and there is no index child for this collector to orphan. The new tag resolved to that manifest with every referenced blob present. The pod was ready at 07:41:37 on 2026-11-11.&lt;/p&gt;

&lt;p&gt;Then the other five. Four of those repositories had CI that pushed to GitHub's container registry and edited manifest paths that no longer existed, so whatever had put their images into our registry in March, it was not the workflow in the repo. Three of the four built successfully and failed at the deploy step before I noticed. I pointed the four workflows at the real registry and the real manifest paths, rebuilt, and by 09:20 all six workloads were running from complete single-manifest images.&lt;/p&gt;

&lt;h2&gt;
  
  
  Timeline
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;When (UTC)&lt;/th&gt;
&lt;th&gt;What&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;2026-03-14 18:19&lt;/td&gt;
&lt;td&gt;Registry GC CronJob created; &lt;code&gt;--delete-untagged=true&lt;/code&gt; is in a commit dated the next day. Four of the six images that would break are pushed between 19:02 and 19:14&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2026-03-23 13:59&lt;/td&gt;
&lt;td&gt;Signaling image pushed as an OCI index with an attestation child; pod starts&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2026-03-24 04:00&lt;/td&gt;
&lt;td&gt;GC unlinks both children and deletes 6 of 10 blobs. Pod keeps running from the node's cache&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2026-06-02&lt;/td&gt;
&lt;td&gt;Flag removed from the live CronJob after a different image breaks. Existing damage not surveyed&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2026-11-08 13:37&lt;/td&gt;
&lt;td&gt;4.5 GiB image for another service deployed; pod starts 13:40&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2026-11-08 13:46:42&lt;/td&gt;
&lt;td&gt;Signaling pod ends, exit 0. First replacement created&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2026-11-08 13:47&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;DiskPressure&lt;/code&gt;. Image reclaim and evictions begin&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2026-11-08 14:30&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;KubeDeploymentReplicasMismatch&lt;/code&gt; for the Deployment starts firing&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2026-11-08 16:10&lt;/td&gt;
&lt;td&gt;Second &lt;code&gt;DiskPressure&lt;/code&gt; window. First replacement fails (137); second replacement created, &lt;code&gt;ImagePullBackOff&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2026-11-11 05:08&lt;/td&gt;
&lt;td&gt;Cluster audit lists the unavailable Deployment and 1,565 pull failures&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2026-11-11 07:41&lt;/td&gt;
&lt;td&gt;Rebuilt image running. Outage 2 d 17 h 55 min&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2026-11-11 09:20&lt;/td&gt;
&lt;td&gt;Other five images rebuilt and rolled out&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  What I am changing
&lt;/h2&gt;

&lt;p&gt;The alert rule existed and fired. &lt;code&gt;KubeDeploymentReplicasMismatch&lt;/code&gt; went pending at 14:00 and firing at 14:30 on 2026-11-08 and stayed firing until the fix. Alertmanager's route sends everything above &lt;code&gt;info&lt;/code&gt; to a Slack channel and repeats every four hours. I have not gone back through the channel history, and I did not act on it. One week earlier, on 2026-11-01, that channel already had nine alerts firing at the same time, including a TURN server that the audit dates as crash-looping since September. The rule was there. The channel was not being read.&lt;/p&gt;

&lt;p&gt;Done:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;provenance: false&lt;/code&gt; and &lt;code&gt;platforms: linux/amd64&lt;/code&gt; in the six workflows that push to this registry.&lt;/li&gt;
&lt;li&gt;Models out of the image and onto a volume.&lt;/li&gt;
&lt;li&gt;Four CI pipelines reconnected to the manifests they are meant to update.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Not done yet:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The remaining workflows in the organisation, or upgrade the registry to distribution 3.x, which handles index children in GC.&lt;/li&gt;
&lt;li&gt;A daily job that walks every image tag referenced by a workload: resolve the tag, fetch the index, fetch each child, &lt;code&gt;HEAD&lt;/code&gt; each blob. On a broken image the tag list and the index both look normal, so the check has to go down to the blobs.&lt;/li&gt;
&lt;li&gt;Fix or silence the nine standing alerts. A page, not a Slack line, for &lt;code&gt;available == 0&lt;/code&gt; lasting more than ten minutes on anything with a public hostname.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;ephemeral-storage&lt;/code&gt; requests on the one workload allowed to be large, and an alert on node disk under 15 %.&lt;/li&gt;
&lt;li&gt;The registry's manifests under the same GitOps reconciliation as everything else. They are in git, but nothing applies them: the live CronJob was patched by hand on 2 June, two days after the commit that removed the flag.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If you run, or have ever run, &lt;code&gt;docker/distribution&lt;/code&gt; 2.x with &lt;code&gt;--delete-untagged&lt;/code&gt;, check every multi-manifest image pushed in that window by asking for the platform manifest, not the tag. A running pod only proves that the node has the image.&lt;/p&gt;

&lt;p&gt;Public post-incident reports where pods could not come back because the registry could not serve their image are collected under the &lt;a href="https://benchlet.com/incidentlib/app?symptom=image-pull-failures" rel="noopener noreferrer"&gt;image pull failures&lt;/a&gt; symptom in Incident Library, searchable by the error text you saw.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://benchlet.com/guides/imagepullbackoff-manifest-unknown-registry-garbage-collection/" rel="noopener noreferrer"&gt;benchlet.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>kubernetes</category>
      <category>devops</category>
      <category>docker</category>
      <category>sre</category>
    </item>
    <item>
      <title>The Loupe — training your eye to catch AI's bugs before you ship</title>
      <dc:creator>KYUNGHWAN KIM</dc:creator>
      <pubDate>Fri, 26 Jun 2026 09:47:22 +0000</pubDate>
      <link>https://dev.to/kkhdevs/the-loupe-training-your-eye-to-catch-ais-bugs-before-you-ship-1apn</link>
      <guid>https://dev.to/kkhdevs/the-loupe-training-your-eye-to-catch-ais-bugs-before-you-ship-1apn</guid>
      <description>&lt;p&gt;AI writes the code in seconds. The question nobody answers: is it actually right?&lt;/p&gt;

&lt;p&gt;I'm building The Loupe — short, real-world exercises where you verify AI-written code before you ship it. The dangerous bugs aren't the crashes; they're the lines that look right and quietly aren't.&lt;/p&gt;

&lt;p&gt;A well-known Korean dev creator (Zerocho) just featured it in a short video, which made my week 🙏&lt;/p&gt;

&lt;p&gt;▶ The 60-second idea: &lt;a href="https://youtube.com/shorts/0xcYQquVZp0?si=nbydJs49F6Vyk5Vp" rel="noopener noreferrer"&gt;https://youtube.com/shorts/0xcYQquVZp0?si=nbydJs49F6Vyk5Vp&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Would love feedback from this community — what's the sneakiest "looks-right-but-wrong" bug you've shipped?&lt;/p&gt;

</description>
      <category>webdev</category>
      <category>ai</category>
      <category>career</category>
      <category>programming</category>
    </item>
    <item>
      <title>The half of software that didn't get cheaper</title>
      <dc:creator>KYUNGHWAN KIM</dc:creator>
      <pubDate>Tue, 23 Jun 2026 23:37:51 +0000</pubDate>
      <link>https://dev.to/kkhdevs/the-half-of-software-that-didnt-get-cheaper-3mpi</link>
      <guid>https://dev.to/kkhdevs/the-half-of-software-that-didnt-get-cheaper-3mpi</guid>
      <description>&lt;p&gt;Something flipped this year, and most of the conversation is still pointed at the wrong half of it.&lt;/p&gt;

&lt;p&gt;The cost of &lt;em&gt;producing&lt;/em&gt; code collapsed. A feature that used to eat a day now takes an hour. We all feel it. What almost nobody is pricing in is that the cost of the other half — understanding what the code does and being sure it's right — didn't move at all. If anything, it went up.&lt;/p&gt;

&lt;p&gt;That's the whole thesis: writing got cheap, reading and trusting did not. And that gap, not the speed of generation, is the thing that now decides whether you can actually ship.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Why reading got harder, not easier&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;When a teammate opens a pull request, every line carries intent. You can ask "why did you do this?" and get a real answer, because a person made a decision there. You review against that intent.&lt;/p&gt;

&lt;p&gt;Generated code has no intent behind any single line. It's locally plausible everywhere — each line looks like something a competent developer might write — but there's no reasoning underneath to lean on. So you can't review it the way you review a human diff. You can't ask the author why, because there was no author and no why. You're left checking behavior directly, which is slower and far less forgiving.&lt;/p&gt;

&lt;p&gt;This is also why the line-by-line diff, the unit we've reviewed software in for twenty years, quietly stopped working. The diff assumes each changed line was a human choice worth scrutinizing. When the lines are generated, the diff tells you &lt;em&gt;what&lt;/em&gt; changed but nothing about &lt;em&gt;where the risk is&lt;/em&gt;. The dangerous failures aren't in the lines that changed — they're in how those changes interact with everything that didn't.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The bottleneck moved from typing to judging&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;For most of the history of this craft, the scarce skill was production: can you build the thing. That's no longer scarce. The&lt;br&gt;
scarce skill is now judgment: can you look at a working-looking pile of code and tell whether it's actually correct.&lt;/p&gt;

&lt;p&gt;And the surveys keep drawing the same shape. Reviewers rate AI-assisted code highly at review time — clean, readable, idiomatic. Then it ships, and incident counts go up, senior engineers spend more time firefighting, and a meaningful chunk of that code needs significant rework. "Looks reviewed" and "is correct" came apart. Readability — the thing the code optimizes for — is exactly what lowers a reviewer's guard. Rough code makes you slow down; beautiful code makes you sign off on behavior you never checked.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The imbalance nobody's fixing&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Here's what bothers me most. We spent a year building incredible tools for the half that got cheap: autocomplete, agents, parallel PRs, generate-more-of-everything. We built almost nothing for the half that stayed expensive.&lt;/p&gt;

&lt;p&gt;Everyone now has a way to produce more code. Almost no one has a way to get &lt;em&gt;better at judging&lt;/em&gt; it. There's no gym for the skill that suddenly matters most. You're expected to develop it by accident, in production, on code you didn't write — which is the worst possible place to learn it.&lt;/p&gt;

&lt;p&gt;That asymmetry is the gap I keep coming back to, and it's the reason I'm building theloupe.dev: a place to practice the half&lt;br&gt;
that didn't get cheaper — reading code you didn't write and deciding, with reasons, whether it's actually right.&lt;/p&gt;

&lt;p&gt;I'll get into how you even practice that — how you make something feel like a real production failure instead of a toy — in the next post. For now I just want to put the thesis down plainly: the cost of writing fell off a cliff, the cost of being sure didn't, and we've been building almost entirely for the wrong one.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>webdev</category>
      <category>programming</category>
      <category>claude</category>
    </item>
  </channel>
</rss>
