<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Rabin Adhikari</title>
    <description>The latest articles on DEV Community by Rabin Adhikari (@softverseadmin).</description>
    <link>https://dev.to/softverseadmin</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4217550%2Fc892b60b-191f-41b6-9516-5fd2465af4a2.png</url>
      <title>DEV Community: Rabin Adhikari</title>
      <link>https://dev.to/softverseadmin</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/softverseadmin"/>
    <language>en</language>
    <item>
      <title>Running AI Agents on Kubernetes Without Handing Them the Cluster: Agent Sandbox + gVisor, Step by Step</title>
      <dc:creator>Rabin Adhikari</dc:creator>
      <pubDate>Thu, 12 Nov 2026 10:07:26 +0000</pubDate>
      <link>https://dev.to/softverseadmin/running-ai-agents-on-kubernetes-without-handing-them-the-cluster-agent-sandbox-gvisor-step-by-32an</link>
      <guid>https://dev.to/softverseadmin/running-ai-agents-on-kubernetes-without-handing-them-the-cluster-agent-sandbox-gvisor-step-by-32an</guid>
      <description>&lt;h1&gt;
  
  
  Running AI Agents on Kubernetes Without Handing Them the Cluster: Agent Sandbox + gVisor, Step by Step
&lt;/h1&gt;

&lt;blockquote&gt;
&lt;h2&gt;
  
  
  TL;DR
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;An AI agent is not a normal workload.&lt;/strong&gt; It executes code that was generated seconds ago and never reviewed, it takes instructions from content it reads (web pages, tickets, documents — prompt injection is a design condition, not an edge case), and it holds real credentials: LLM API keys, tool tokens, sometimes cloud access. Your isolation has to assume the code inside is hostile.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;This is already a production question.&lt;/strong&gt; The CNCF and SlashData State of Cloud Native Development Q3 2026 report (released 10 November 2026) found that &lt;strong&gt;49% of developers integrating AI into their services already have agentic systems in production&lt;/strong&gt; — and KubeCon + CloudNativeCon North America 2026 ran a dedicated AI Inference + Agentic track for the first time.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The stack in this guide:&lt;/strong&gt; a dedicated namespace with quotas → an agent identity with &lt;em&gt;zero&lt;/em&gt; Kubernetes permissions → gVisor kernel isolation via RuntimeClass → the Kubernetes &lt;strong&gt;Agent Sandbox&lt;/strong&gt; CRD (a SIG Apps project) for lifecycle, templates and warm pools → default-deny networking with named egress holes → short-lived, scoped credentials → audit on everything.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;One honesty note up front:&lt;/strong&gt; Agent Sandbox is young — its APIs are at &lt;code&gt;v1beta1&lt;/code&gt; and the current release is v1.0.10. Treat the CRD as the lifecycle convenience it is. The load-bearing security layers are the ones underneath it: namespace, RBAC, RuntimeClass, NetworkPolicy. Those work with or without the CRD.&lt;/li&gt;
&lt;/ul&gt;
&lt;/blockquote&gt;

&lt;p&gt;Every platform team has had some version of the same meeting this year. A product team wants to ship an agent: it reads tickets, writes code, calls internal tools, maybe opens pull requests. It needs to &lt;em&gt;run&lt;/em&gt; somewhere, it needs credentials to be useful, and it will execute — by design — whatever the model decides to execute. The question lands on the platform team: where does this thing live?&lt;/p&gt;

&lt;p&gt;The default answer, "just deploy it like a service," quietly assumes the code in the pod is trusted because a human wrote and reviewed it. For an agent, that assumption is false on day one. This guide builds the other answer: a sandbox stack on Kubernetes, layer by layer, with working manifests. The pattern is the same one you would use for any untrusted workload — CI runners executing strangers' code are the closest classical analogue — tuned for what makes agents different.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why an agent is a different threat model
&lt;/h2&gt;

&lt;p&gt;Four properties separate an agent runtime from an ordinary microservice:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;It runs unreviewed code.&lt;/strong&gt; The whole point of a coding or tool-using agent is that it generates commands, scripts and tool calls at runtime. There is no code review gate between the model and the shell.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Its instructions arrive over untrusted channels.&lt;/strong&gt; An agent that reads a web page, a support ticket or a repository README is reading text an attacker may control. A crafted document can try to redirect the agent — read this file, call that tool, send this data somewhere. You cannot patch this at the model layer with confidence, so you bound what a redirected agent can reach.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;It is credentialed by design.&lt;/strong&gt; An agent with no keys can do nothing. An agent with an LLM API key, a Git token and a database password is useful — and is a standing, machine-speed confused deputy holding all of them.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;It is long-lived and stateful.&lt;/strong&gt; Agent sessions run for minutes to hours, accumulate files and context, and often persist a workspace. That is a different shape from both stateless services and batch jobs, and it is exactly the shape Kubernetes has historically been awkward at — one reason the Sandbox abstraction exists.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Put together, the design question changes. For a normal workload you ask, "what if this pod is compromised?" For an agent you ask, "this pod will eventually do something it should not — what can it reach when it does?" Every step below answers that question at a different layer. If you are newer to cluster hardening in general, our &lt;a href="https://www.devopsenginer.com/blog/kubernetes-security-basics-guide-2026" rel="noopener noreferrer"&gt;Kubernetes security basics guide&lt;/a&gt; covers the foundations this builds on.&lt;/p&gt;

&lt;h2&gt;
  
  
  The isolation stack at a glance
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Layer&lt;/th&gt;
&lt;th&gt;Control in this guide&lt;/th&gt;
&lt;th&gt;What it contains when the agent misbehaves&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Kernel&lt;/td&gt;
&lt;td&gt;gVisor (&lt;code&gt;runsc&lt;/code&gt;) via RuntimeClass&lt;/td&gt;
&lt;td&gt;A container escape reaches a user-space kernel, not the host kernel&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Identity&lt;/td&gt;
&lt;td&gt;Dedicated ServiceAccount, token automount off, zero RBAC&lt;/td&gt;
&lt;td&gt;A stolen pod token buys nothing at the API server&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Namespace&lt;/td&gt;
&lt;td&gt;Dedicated namespace, Pod Security Standards &lt;code&gt;restricted&lt;/code&gt;, quotas&lt;/td&gt;
&lt;td&gt;Blast radius and resource abuse stay inside one fence&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Lifecycle&lt;/td&gt;
&lt;td&gt;Agent Sandbox CRD: templates, warm pools, claim TTLs&lt;/td&gt;
&lt;td&gt;No forgotten, immortal agent pods; sessions expire by default&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Network&lt;/td&gt;
&lt;td&gt;Default-deny NetworkPolicy + named egress allows&lt;/td&gt;
&lt;td&gt;Exfiltration and lateral movement have nowhere to go&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Secrets&lt;/td&gt;
&lt;td&gt;Scoped, short-lived, per-session credentials&lt;/td&gt;
&lt;td&gt;A leaked credential is narrow and expires quickly&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Visibility&lt;/td&gt;
&lt;td&gt;API audit logs, pod logs shipped out, alerts&lt;/td&gt;
&lt;td&gt;Misuse is visible after the fact — and during it&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;No single layer is the security story. The story is that an attacker (or a confused model) has to defeat all of them in sequence, and each one is independently testable.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 1 — A namespace that assumes the worst
&lt;/h2&gt;

&lt;p&gt;Agents get their own namespace. Not a shared &lt;code&gt;tools&lt;/code&gt; namespace, not the namespace of the product they serve — their own, so that quotas, policies and network rules apply to agents and nothing else.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Namespace&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;agents&lt;/span&gt;
  &lt;span class="na"&gt;labels&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;pod-security.kubernetes.io/enforce&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;restricted&lt;/span&gt;
    &lt;span class="na"&gt;pod-security.kubernetes.io/audit&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;restricted&lt;/span&gt;
    &lt;span class="na"&gt;pod-security.kubernetes.io/warn&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;restricted&lt;/span&gt;
&lt;span class="nn"&gt;---
&lt;/span&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ResourceQuota&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;agents-quota&lt;/span&gt;
  &lt;span class="na"&gt;namespace&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;agents&lt;/span&gt;
&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;hard&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;pods&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"20"&lt;/span&gt;
    &lt;span class="na"&gt;requests.cpu&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"4"&lt;/span&gt;
    &lt;span class="na"&gt;requests.memory&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;8Gi&lt;/span&gt;
    &lt;span class="na"&gt;limits.cpu&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"8"&lt;/span&gt;
    &lt;span class="na"&gt;limits.memory&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;16Gi&lt;/span&gt;
    &lt;span class="na"&gt;requests.ephemeral-storage&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;10Gi&lt;/span&gt;
    &lt;span class="na"&gt;limits.ephemeral-storage&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;20Gi&lt;/span&gt;
    &lt;span class="na"&gt;count/persistentvolumeclaims&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"20"&lt;/span&gt;
&lt;span class="nn"&gt;---
&lt;/span&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;LimitRange&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;agents-defaults&lt;/span&gt;
  &lt;span class="na"&gt;namespace&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;agents&lt;/span&gt;
&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;limits&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;type&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Container&lt;/span&gt;
      &lt;span class="na"&gt;default&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="na"&gt;cpu&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"1"&lt;/span&gt;
        &lt;span class="na"&gt;memory&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;1Gi&lt;/span&gt;
        &lt;span class="na"&gt;ephemeral-storage&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;2Gi&lt;/span&gt;
      &lt;span class="na"&gt;defaultRequest&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="na"&gt;cpu&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;250m&lt;/span&gt;
        &lt;span class="na"&gt;memory&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;512Mi&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Three decisions in this manifest are worth more than the rest:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Pod Security Standards at &lt;code&gt;restricted&lt;/code&gt;&lt;/strong&gt; means the API server itself rejects pods that ask for privilege escalation, host namespaces, or root-by-default. You get a baseline even from manifests you did not write — including ones a controller generates.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The quota caps the fleet, not just the pod.&lt;/strong&gt; An agent stuck in a reasoning loop is a cost and availability incident before it is a security one. Twenty pods and 8 CPU of limits is a starting point for a pilot, sized so a runaway can hurt the &lt;code&gt;agents&lt;/code&gt; namespace and nothing else. (Our &lt;a href="https://www.devopsenginer.com/blog/kubernetes-resource-requests-limits-guide-2026" rel="noopener noreferrer"&gt;resource requests and limits guide&lt;/a&gt; covers the sizing discipline in depth.)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Ephemeral storage is quota'd.&lt;/strong&gt; Agents write files — checkouts, build artifacts, downloads. Unbounded ephemeral storage is how one session fills a node disk and evicts its neighbours.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;One step further, on clusters that can afford it: taint a dedicated node pool for agent workloads and give the sandbox template the matching toleration. Kernel isolation (Step 3) reduces what a hostile pod can do to a node; physical separation removes the shared fate entirely. For a first deployment, the namespace fence plus gVisor is a defensible start; note the node pool as the scale-up move.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 2 — Identity: the agent gets zero Kubernetes permissions
&lt;/h2&gt;

&lt;p&gt;This is the step teams most often get backwards, so be deliberate. There are &lt;strong&gt;two&lt;/strong&gt; identities in this design, and only one of them ever touches the API server:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The &lt;strong&gt;orchestrator&lt;/strong&gt; — your service that receives a request, creates a &lt;code&gt;SandboxClaim&lt;/code&gt;, and hands back a session. It needs a small, explicit Role.&lt;/li&gt;
&lt;li&gt;The &lt;strong&gt;agent runtime&lt;/strong&gt; — the pod that runs generated code. It needs &lt;strong&gt;no Kubernetes permissions at all&lt;/strong&gt;, and its ServiceAccount token should not even be mounted.
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ServiceAccount&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;agent-runtime&lt;/span&gt;
  &lt;span class="na"&gt;namespace&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;agents&lt;/span&gt;
&lt;span class="na"&gt;automountServiceAccountToken&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;
&lt;span class="nn"&gt;---
&lt;/span&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ServiceAccount&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;agent-orchestrator&lt;/span&gt;
  &lt;span class="na"&gt;namespace&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;agents&lt;/span&gt;
&lt;span class="nn"&gt;---
&lt;/span&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;rbac.authorization.k8s.io/v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Role&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;agent-orchestrator&lt;/span&gt;
  &lt;span class="na"&gt;namespace&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;agents&lt;/span&gt;
&lt;span class="na"&gt;rules&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;apiGroups&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"extensions.agents.x-k8s.io"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="na"&gt;resources&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"sandboxclaims"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="na"&gt;verbs&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"create"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"get"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"list"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"watch"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"delete"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
  &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;apiGroups&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"agents.x-k8s.io"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="na"&gt;resources&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"sandboxes"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="na"&gt;verbs&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"get"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"list"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"watch"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="nn"&gt;---
&lt;/span&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;rbac.authorization.k8s.io/v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;RoleBinding&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;agent-orchestrator&lt;/span&gt;
  &lt;span class="na"&gt;namespace&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;agents&lt;/span&gt;
&lt;span class="na"&gt;roleRef&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;apiGroup&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;rbac.authorization.k8s.io&lt;/span&gt;
  &lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Role&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;agent-orchestrator&lt;/span&gt;
&lt;span class="na"&gt;subjects&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ServiceAccount&lt;/span&gt;
    &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;agent-orchestrator&lt;/span&gt;
    &lt;span class="na"&gt;namespace&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;agents&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Read the asymmetry: the orchestrator can create claims and read sandboxes &lt;em&gt;in this namespace only&lt;/em&gt; — no secrets, no pods/exec, no other namespaces. The agent runtime's ServiceAccount exists so the pod has a named identity, but with &lt;code&gt;automountServiceAccountToken: false&lt;/code&gt; and no RoleBindings, code inside the sandbox that goes looking for the usual &lt;code&gt;/var/run/secrets/kubernetes.io/serviceaccount/token&lt;/code&gt; finds nothing to steal. If your agent framework insists it needs to "call Kubernetes tools," that is the moment to stop and redesign: give the agent an application-level API with the three operations it actually needs, mediated by your orchestrator — never raw cluster credentials. Our &lt;a href="https://www.devopsenginer.com/blog/kubernetes-rbac-guide-2026" rel="noopener noreferrer"&gt;Kubernetes RBAC guide&lt;/a&gt; covers the audit pattern for keeping Roles this small in practice.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 3 — Kernel isolation: gVisor via RuntimeClass
&lt;/h2&gt;

&lt;p&gt;Namespaces and RBAC govern the API plane. They do nothing about the kernel plane: a container under the standard &lt;code&gt;runc&lt;/code&gt; runtime shares the host's kernel, so a kernel exploit or an overly broad syscall path is a node compromise. An agent executing generated code is precisely the workload that should not share a kernel with your other tenants.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;gVisor&lt;/strong&gt; inserts a user-space kernel (the Sentry) between the container and the host. The container's syscalls are intercepted and serviced by gVisor, which makes a much smaller, carefully chosen set of calls to the host on the workload's behalf. A bug in the workload's code now has to escape a second, purpose-built boundary before it sees the real kernel.&lt;/p&gt;

&lt;p&gt;On self-managed nodes with containerd, register the &lt;code&gt;runsc&lt;/code&gt; runtime (the shim is &lt;code&gt;containerd-shim-runsc-v1&lt;/code&gt;; the runtime type is &lt;code&gt;io.containerd.runsc.v1&lt;/code&gt;):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight toml"&gt;&lt;code&gt;&lt;span class="c1"&gt;# /etc/containerd/config.toml (excerpt) — then restart containerd
&lt;/span&gt;&lt;span class="k"&gt;[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runsc]&lt;/span&gt;
  &lt;span class="na"&gt;runtime_type&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"io.containerd.runsc.v1"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then expose it to the scheduler as a RuntimeClass — the handler name must match the runtime name you registered in containerd:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;node.k8s.io/v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;RuntimeClass&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;gvisor&lt;/span&gt;
&lt;span class="na"&gt;handler&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;runsc&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;On GKE, you do not edit containerd configs: GKE Sandbox is the managed form of the same idea — a node pool where pods declaring &lt;code&gt;runtimeClassName: gvisor&lt;/code&gt; run under gVisor. Either way, from the pod spec's point of view the contract is identical, which is why the template in Step 4 carries a single line: &lt;code&gt;runtimeClassName: gvisor&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Verify with a throwaway pod before you build anything on top — apply a test pod with the runtime class, confirm it reaches &lt;code&gt;Running&lt;/code&gt;, and check that &lt;code&gt;kubectl get pod &amp;lt;name&amp;gt; -o yaml&lt;/code&gt; shows the &lt;code&gt;runtimeClassName&lt;/code&gt; you set. A pod stuck in &lt;code&gt;ContainerCreating&lt;/code&gt; with runtime errors is a node-side installation problem, and it is much cheaper to meet it now than after the templates land.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;gVisor or Kata?&lt;/strong&gt; They are the two realistic choices, and the decision is mostly about your nodes:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;gVisor&lt;/th&gt;
&lt;th&gt;Kata Containers&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Isolation model&lt;/td&gt;
&lt;td&gt;User-space kernel intercepts syscalls&lt;/td&gt;
&lt;td&gt;Each pod runs in a lightweight VM with its own guest kernel&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Node requirement&lt;/td&gt;
&lt;td&gt;Runs anywhere containerd runs&lt;/td&gt;
&lt;td&gt;Needs KVM — bare metal or nested virtualisation on cloud nodes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Overhead profile&lt;/td&gt;
&lt;td&gt;Syscall- and I/O-heavy workloads pay a tax&lt;/td&gt;
&lt;td&gt;VM memory footprint and cold-start cost per pod&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Compatibility&lt;/td&gt;
&lt;td&gt;Some syscalls are unimplemented; exotic workloads can notice&lt;/td&gt;
&lt;td&gt;Near-native Linux inside the guest&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Pick it when&lt;/td&gt;
&lt;td&gt;General agent/code-execution sandboxing on ordinary nodes&lt;/td&gt;
&lt;td&gt;You already run KVM-capable nodes and want VM-grade separation&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Agent runtimes — shells, interpreters, file tools, HTTP calls — are a good gVisor fit. If your agents will run heavy builds or unusual low-level tooling, benchmark both against your real workload before standardising; the RuntimeClass abstraction means the choice stays a one-line change per template.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 4 — Agent Sandbox: lifecycle as a CRD
&lt;/h2&gt;

&lt;p&gt;With the fence (Step 1), the identities (Step 2) and the kernel boundary (Step 3) in place, you still need to manage the sandboxes themselves: create one per session, keep it warm enough to start fast, give it a stable identity and workspace, and — most importantly — make it &lt;em&gt;die&lt;/em&gt; on schedule. That lifecycle is what the Kubernetes &lt;strong&gt;Agent Sandbox&lt;/strong&gt; project provides.&lt;/p&gt;

&lt;p&gt;Agent Sandbox is a SIG Apps project that adds a &lt;code&gt;Sandbox&lt;/code&gt; custom resource: a single, stateful pod with a stable hostname, optional persistent storage, and lifecycle management (creation, scheduled deletion, pause and resume) — the "lightweight, single-container VM" shape that agent sessions actually have, instead of forcing them into Deployments or one-member StatefulSets.&lt;/p&gt;

&lt;p&gt;Be precise about what it is &lt;strong&gt;not&lt;/strong&gt;: the project describes itself as a sandbox &lt;em&gt;orchestrator&lt;/em&gt;. It deliberately delegates low-level isolation to secure runtimes like gVisor or Kata, by creating pods configured with the RuntimeClass you choose. The CRD gives you lifecycle and consistency; the security still comes from Steps 1–3 and 5–6. Designs that treat "we installed the Sandbox CRD" as "we sandboxed the agent" have it exactly backwards.&lt;/p&gt;

&lt;p&gt;Install it pinned — never from &lt;code&gt;latest&lt;/code&gt; — and verify the CRDs and controller afterwards:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="kd"&gt;export&lt;/span&gt; &lt;span class="nv"&gt;VERSION&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"v1.0.10"&lt;/span&gt;
kubectl apply -f https://github.com/kubernetes-sigs/agent-sandbox/releases/download/&lt;span class="nv"&gt;${VERSION}&lt;/span&gt;/sandbox-with-extensions.yaml
kubectl get crd sandboxes.agents.x-k8s.io
kubectl get deploy agent-sandbox-controller -n agent-sandbox-system
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;(v1.0.10 is the current release as of this writing, November 2026; the project asks for Kubernetes 1.25 or later, with 1.29+ recommended so CRD CEL validation rules are fully enforced.) The extensions manifest adds the three resources that make the project practical: &lt;code&gt;SandboxTemplate&lt;/code&gt;, &lt;code&gt;SandboxWarmPool&lt;/code&gt; and &lt;code&gt;SandboxClaim&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;A minimal &lt;code&gt;Sandbox&lt;/code&gt; shows the shape — one pod template, nothing more:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;agents.x-k8s.io/v1beta1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Sandbox&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;agent-session-demo&lt;/span&gt;
  &lt;span class="na"&gt;namespace&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;agents&lt;/span&gt;
&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;podTemplate&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;runtimeClassName&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;gvisor&lt;/span&gt;
      &lt;span class="na"&gt;restartPolicy&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Never&lt;/span&gt;
      &lt;span class="na"&gt;containers&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;agent&lt;/span&gt;
          &lt;span class="c1"&gt;# Stand-in: use your own agent runtime image and entrypoint
&lt;/span&gt;          &lt;span class="na"&gt;image&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;python:3.12-slim&lt;/span&gt;
          &lt;span class="na"&gt;command&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"sleep"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"infinity"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;In production you would rarely create bare &lt;code&gt;Sandbox&lt;/code&gt; objects by hand. You define one hardened &lt;strong&gt;SandboxTemplate&lt;/strong&gt; — owned by the platform team, reviewed like code — and let everything else instantiate from it. Here is the centrepiece of this guide; every field earns its place:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;extensions.agents.x-k8s.io/v1beta1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;SandboxTemplate&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;agent-sandbox-template&lt;/span&gt;
  &lt;span class="na"&gt;namespace&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;agents&lt;/span&gt;
&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="c1"&gt;# The controller manages a NetworkPolicy for sandboxes from this template.
&lt;/span&gt;  &lt;span class="na"&gt;networkPolicyManagement&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Managed&lt;/span&gt;
  &lt;span class="na"&gt;networkPolicy&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;egress&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="c1"&gt;# DNS only — name resolution is not general network access
&lt;/span&gt;      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;ports&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;protocol&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;UDP&lt;/span&gt;
            &lt;span class="na"&gt;port&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;53&lt;/span&gt;
          &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;protocol&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;TCP&lt;/span&gt;
            &lt;span class="na"&gt;port&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;53&lt;/span&gt;
      &lt;span class="c1"&gt;# TLS out — tightened further in Step 5
&lt;/span&gt;      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;ports&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;protocol&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;TCP&lt;/span&gt;
            &lt;span class="na"&gt;port&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;443&lt;/span&gt;
  &lt;span class="na"&gt;volumeClaimTemplates&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;workspace&lt;/span&gt;
      &lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="na"&gt;accessModes&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"ReadWriteOnce"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
        &lt;span class="na"&gt;resources&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="na"&gt;storage&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;5Gi&lt;/span&gt;
  &lt;span class="na"&gt;podTemplate&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;serviceAccountName&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;agent-runtime&lt;/span&gt;
      &lt;span class="na"&gt;automountServiceAccountToken&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;
      &lt;span class="na"&gt;runtimeClassName&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;gvisor&lt;/span&gt;
      &lt;span class="na"&gt;restartPolicy&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Never&lt;/span&gt;
      &lt;span class="na"&gt;securityContext&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="na"&gt;runAsNonRoot&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
        &lt;span class="na"&gt;runAsUser&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;10001&lt;/span&gt;
        &lt;span class="na"&gt;fsGroup&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;10001&lt;/span&gt;
        &lt;span class="na"&gt;seccompProfile&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;type&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;RuntimeDefault&lt;/span&gt;
      &lt;span class="na"&gt;containers&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;agent&lt;/span&gt;
          &lt;span class="c1"&gt;# Stand-in: your agent runtime image belongs here
&lt;/span&gt;          &lt;span class="na"&gt;image&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;python:3.12-slim&lt;/span&gt;
          &lt;span class="na"&gt;command&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"sleep"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"infinity"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
          &lt;span class="na"&gt;securityContext&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="na"&gt;allowPrivilegeEscalation&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;
            &lt;span class="na"&gt;readOnlyRootFilesystem&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
            &lt;span class="na"&gt;capabilities&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
              &lt;span class="na"&gt;drop&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"ALL"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
          &lt;span class="na"&gt;resources&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="na"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
              &lt;span class="na"&gt;cpu&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;250m&lt;/span&gt;
              &lt;span class="na"&gt;memory&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;512Mi&lt;/span&gt;
            &lt;span class="na"&gt;limits&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
              &lt;span class="na"&gt;cpu&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"1"&lt;/span&gt;
              &lt;span class="na"&gt;memory&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;1Gi&lt;/span&gt;
              &lt;span class="na"&gt;ephemeral-storage&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;2Gi&lt;/span&gt;
          &lt;span class="na"&gt;env&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;LLM_API_KEY&lt;/span&gt;
              &lt;span class="na"&gt;valueFrom&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                &lt;span class="na"&gt;secretKeyRef&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;llm-credentials&lt;/span&gt;
                  &lt;span class="na"&gt;key&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;api-key&lt;/span&gt;
                  &lt;span class="c1"&gt;# Fail closed: no key, no sandbox
&lt;/span&gt;                  &lt;span class="na"&gt;optional&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;
          &lt;span class="na"&gt;volumeMounts&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;workspace&lt;/span&gt;
              &lt;span class="na"&gt;mountPath&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;/workspace&lt;/span&gt;
            &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;tmp&lt;/span&gt;
              &lt;span class="na"&gt;mountPath&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;/tmp&lt;/span&gt;
      &lt;span class="na"&gt;volumes&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;tmp&lt;/span&gt;
          &lt;span class="na"&gt;emptyDir&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="na"&gt;sizeLimit&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;512Mi&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Details that matter:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The template is the security policy.&lt;/strong&gt; Runtime class, non-root, dropped capabilities, read-only root filesystem, token automount off, per-container limits — all fixed here, once, by the platform team. Application teams get sandboxes; they do not get to renegotiate the isolation line by line. That separation is the entire point of the template/claim split.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Managed network policy&lt;/strong&gt; means the controller maintains a NetworkPolicy for sandboxes created from this template, with the DNS-and-443 egress shape above as the starting point. Step 5 hardens networking at the namespace level as well — belt and braces, deliberately.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The workspace volume is per-sandbox.&lt;/strong&gt; Each sandbox gets its own PVC from the template, so one session's files are never another session's files, and storage dies with the sandbox.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;optional: false&lt;/code&gt; on the secret.&lt;/strong&gt; If the LLM credentials Secret is missing, the pod refuses to start. Fail closed, find out in staging — not a runtime that silently runs unauthenticated, and not a fallback to some shared default key.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Claims and warm pools are how sessions actually start. A &lt;code&gt;SandboxWarmPool&lt;/code&gt; keeps a small number of template-conformant sandboxes pre-created; a &lt;code&gt;SandboxClaim&lt;/code&gt; checks one out:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;extensions.agents.x-k8s.io/v1beta1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;SandboxWarmPool&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;agent-warmpool&lt;/span&gt;
  &lt;span class="na"&gt;namespace&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;agents&lt;/span&gt;
&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;replicas&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;
  &lt;span class="na"&gt;sandboxTemplateRef&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;agent-sandbox-template&lt;/span&gt;
&lt;span class="nn"&gt;---
&lt;/span&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;extensions.agents.x-k8s.io/v1beta1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;SandboxClaim&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;session-0001&lt;/span&gt;
  &lt;span class="na"&gt;namespace&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;agents&lt;/span&gt;
&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;warmPoolRef&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;agent-warmpool&lt;/span&gt;
  &lt;span class="na"&gt;lifecycle&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="c1"&gt;# Reap the claim (and its sandbox) 10 minutes after the session finishes
&lt;/span&gt;    &lt;span class="na"&gt;ttlSecondsAfterFinished&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;600&lt;/span&gt;
    &lt;span class="na"&gt;shutdownPolicy&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Delete&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two operational rules for pools. First, &lt;strong&gt;a pool is a trust boundary&lt;/strong&gt;: never serve two teams — and certainly never two tenants — from the same warm pool, because pool members are interchangeable by design. One pool per template per trust level. Second, &lt;strong&gt;a claimed sandbox is spent&lt;/strong&gt;: it serves exactly one session, and when the claim ends the sandbox is deleted, not wiped and returned. The pool tops itself back up to &lt;code&gt;spec.replicas&lt;/code&gt; with fresh sandboxes — that refill behaviour is what makes warm starts safe to offer. And the claim's lifecycle fields give every session a hard expiry even if your orchestrator crashes mid-cleanup: expiry is enforced by the claim controller, not by your code remembering to tidy up.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 5 — Network: default-deny, then named holes
&lt;/h2&gt;

&lt;p&gt;Network policy is where "sandbox" stops being a metaphor. The template's managed policy is a start; the namespace gets its own explicit posture so the rules survive template edits and apply to anything else that ever lands in &lt;code&gt;agents&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;networking.k8s.io/v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;NetworkPolicy&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;default-deny-all&lt;/span&gt;
  &lt;span class="na"&gt;namespace&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;agents&lt;/span&gt;
&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;podSelector&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="na"&gt;policyTypes&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="s"&gt;Ingress&lt;/span&gt;
    &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="s"&gt;Egress&lt;/span&gt;
&lt;span class="nn"&gt;---
&lt;/span&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;networking.k8s.io/v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;NetworkPolicy&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;agent-egress&lt;/span&gt;
  &lt;span class="na"&gt;namespace&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;agents&lt;/span&gt;
&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;podSelector&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="na"&gt;policyTypes&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="s"&gt;Egress&lt;/span&gt;
  &lt;span class="na"&gt;egress&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="c1"&gt;# DNS to the cluster resolver — nothing else in kube-system
&lt;/span&gt;    &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;to&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;namespaceSelector&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="na"&gt;matchLabels&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
              &lt;span class="na"&gt;kubernetes.io/metadata.name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;kube-system&lt;/span&gt;
      &lt;span class="na"&gt;ports&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;protocol&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;UDP&lt;/span&gt;
          &lt;span class="na"&gt;port&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;53&lt;/span&gt;
        &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;protocol&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;TCP&lt;/span&gt;
          &lt;span class="na"&gt;port&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;53&lt;/span&gt;
    &lt;span class="c1"&gt;# TLS to the outside world — deliberately nothing on port 80,
&lt;/span&gt;    &lt;span class="c1"&gt;# which keeps the cloud metadata endpoint (169.254.169.254)
&lt;/span&gt;    &lt;span class="c1"&gt;# unreachable even before the rules below are added
&lt;/span&gt;    &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;ports&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;protocol&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;TCP&lt;/span&gt;
          &lt;span class="na"&gt;port&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;443&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Three points of craft here. First, ingress stays closed: there is no rule allowing anything &lt;em&gt;into&lt;/em&gt; the sandboxes, because agents should be reached through your orchestrator (or the project's sandbox router), never directly. Second, note what is absent — port 80 egress. The cloud instance metadata service lives on a link-local address over plain HTTP, and it is the single highest-value target inside any pod. Default-deny plus a 443-only allow means the metadata endpoint is unreachable by construction. Third, be honest about the limit of the 443 rule: NetworkPolicy works on IPs and ports, not names, so "443 anywhere" still reaches anything that listens on 443, including services you did not intend. It is the pragmatic minimum, not the finished state.&lt;/p&gt;

&lt;p&gt;The finished state is a named destination list. If your CNI supports FQDN policy (Cilium is the common case), express the LLM egress as the exact host it should be:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;cilium.io/v2&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;CiliumNetworkPolicy&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;agent-egress-llm&lt;/span&gt;
  &lt;span class="na"&gt;namespace&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;agents&lt;/span&gt;
&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;endpointSelector&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="na"&gt;egress&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;toFQDNs&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;matchName&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;api.example-llm-provider.com&lt;/span&gt;
      &lt;span class="na"&gt;toPorts&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;ports&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;port&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"443"&lt;/span&gt;
              &lt;span class="na"&gt;protocol&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;TCP&lt;/span&gt;
    &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;toEndpoints&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;matchLabels&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="na"&gt;k8s:io.kubernetes.pod.namespace&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;kube-system&lt;/span&gt;
            &lt;span class="na"&gt;k8s-app&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;kube-dns&lt;/span&gt;
      &lt;span class="na"&gt;toPorts&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;ports&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;port&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"53"&lt;/span&gt;
              &lt;span class="na"&gt;protocol&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;UDP&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Swap in your provider's real API hostname. The alternative shape — used when the CNI cannot do FQDNs — is an egress proxy: deny all direct egress, allow exactly one proxy service, and let the proxy enforce the allowlist (and hold the real provider credentials, which Step 6 picks up). Either pattern beats "443 to the world" by a wide margin. Our &lt;a href="https://www.devopsenginer.com/blog/kubernetes-network-policies-guide-2026" rel="noopener noreferrer"&gt;Kubernetes network policies guide&lt;/a&gt; walks through policy composition and the usual debugging traps if you are building these for the first time.&lt;/p&gt;

&lt;p&gt;One more warning that experience earns the hard way: NetworkPolicy only constrains traffic if your CNI enforces it. Confirm enforcement on a test pod (try to reach a disallowed destination and watch it fail) before you claim the control exists. An unenforced policy is a comment, not a boundary.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 6 — Secrets and credentials: short-lived, scoped, or not at all
&lt;/h2&gt;

&lt;p&gt;Every layer so far assumed the agent might go wrong. Credentials decide how much that costs. The rules, in descending order of preference:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;The sandbox holds nothing it does not need for this session.&lt;/strong&gt; No kubeconfigs, no cloud CLI credential files, no shared "team" tokens baked into the image. If a credential is in the sandbox and the session is manipulated, the credential is gone — design so that "gone" is a small event.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Scope what it must hold.&lt;/strong&gt; The LLM key in the template is a dedicated key for the agent platform — not the organisation's primary provider account — with a spend limit set at the provider. A leaked agent key should be a billing alert, not an incident. Keep provider keys and rotation in a namespace-scoped Secret, managed like any other workload secret (our &lt;a href="https://www.devopsenginer.com/blog/kubernetes-configmaps-secrets-guide-2026" rel="noopener noreferrer"&gt;ConfigMaps and Secrets guide&lt;/a&gt; covers the handling patterns).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Prefer per-session, execution-scoped credentials.&lt;/strong&gt; The strongest pattern: the sandbox holds no standing secret at all. Your orchestrator (or an egress proxy) holds the real credential, and the agent receives a token minted for one session — or even one tool execution — that is revoked when the run ends. The Agent Sandbox project's own examples include exactly this pattern: execution-scoped credentials delivered to a reused sandbox, with egress pinned to a credential proxy and the token revoked when the process exits. Even if you do not adopt that example wholesale, steal its shape.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Cloud access goes through an identity, never a file.&lt;/strong&gt; If the agent legitimately needs cloud APIs, attach a per-namespace workload identity whose role is three verbs on one resource — the same least-privilege discipline as on-prem RBAC, applied to the cloud side (see our &lt;a href="https://www.devopsenginer.com/blog/aws-iam-least-privilege-guide-2026" rel="noopener noreferrer"&gt;AWS IAM least privilege guide&lt;/a&gt;). Long-lived access keys inside a sandbox are a finding, full stop.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The rare exception: something in the sandbox genuinely must authenticate &lt;em&gt;to the Kubernetes API&lt;/em&gt; — for example, a tool gateway you run in-pod. Do not fall back to the mounted default token. Issue a projected, audience-bound token with a one-hour life instead:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="c1"&gt;# Pod-spec fragment — add to the template's containers/volumeMounts and volumes
&lt;/span&gt;&lt;span class="na"&gt;volumeMounts&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;tool-token&lt;/span&gt;
    &lt;span class="na"&gt;mountPath&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;/var/run/secrets/tokens&lt;/span&gt;
    &lt;span class="na"&gt;readOnly&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
&lt;span class="c1"&gt;# ...and under spec.volumes:
&lt;/span&gt;&lt;span class="na"&gt;volumes&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;tool-token&lt;/span&gt;
    &lt;span class="na"&gt;projected&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;sources&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;serviceAccountToken&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="na"&gt;audience&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;tool-gateway&lt;/span&gt;
            &lt;span class="na"&gt;expirationSeconds&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;3600&lt;/span&gt;
            &lt;span class="na"&gt;path&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;token&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The audience means the token is only acceptable to your tool gateway; the expiry means a stolen copy is a one-hour problem; the projection means it is minted per pod, not stored in a Secret that outlives the session.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 7 — Audit and observability: assume you will need the tape
&lt;/h2&gt;

&lt;p&gt;When an agent does something expensive or strange — and the interesting incidents are usually "expensive" before they are "malicious" — the first questions are forensic: which session, created by whom, ran what, talked to whom, for how long? Wire the answers up before the first real session:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;API audit logs.&lt;/strong&gt; Sandbox and claim creation/deletion are the control-plane events that matter. Your audit trail should answer "who or what created this &lt;code&gt;SandboxClaim&lt;/code&gt;, from which identity, at what time" without archaeology. An orchestrator identity (Step 2) makes this attribution clean: every sandbox traces to one claim, every claim to one orchestrator request.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Pod logs, shipped out of the sandbox.&lt;/strong&gt; Collect sandbox stdout/stderr into your central logging with the sandbox and claim names as labels. Logs that live only inside the sandbox are destroyed with it — by your own TTL, at the worst possible moment. Agent frameworks should also emit structured tool-call logs at the application layer (which tool, what arguments, what came back); ship those through the orchestrator, not from inside the sandbox, so a manipulated agent cannot quietly stop reporting itself.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Lifecycle and resource signals.&lt;/strong&gt; Alert on: sandboxes living far longer than the session norm, claim TTLs being hit repeatedly (sessions dying mid-work), ResourceQuota saturation in the &lt;code&gt;agents&lt;/code&gt; namespace (the signature of a runaway loop), and — where your CNI reports them — NetworkPolicy drop spikes from sandbox pods, which are often the first visible sign of an agent trying to reach something it should not.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Events and controller logs.&lt;/strong&gt; The agent-sandbox controller's events on claims and pools (adoption failures, refill problems) are your early warning that the warm pool is empty and sessions are cold-starting — an availability problem your users will report as "the agent product is slow."&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Our &lt;a href="https://www.devopsenginer.com/blog/kubernetes-observability-guide-2026" rel="noopener noreferrer"&gt;Kubernetes observability guide&lt;/a&gt; covers the logging/metrics plumbing; the agent-specific work is the labelling discipline above, so sandbox telemetry joins up with session telemetry from your application.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I would not run yet
&lt;/h2&gt;

&lt;p&gt;A guide that only sells the stack is not trustworthy, so the reservations, plainly:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Do not let the CRD's youth become your risk.&lt;/strong&gt; Agent Sandbox's APIs are &lt;code&gt;v1beta1&lt;/code&gt; and the project is releasing fast (v1.0.10 at this writing). Pin the version, read the release notes before upgrading — recent releases have changed router NetworkPolicy behaviour in ways that need action — and keep your GitOps manifests as the source of truth so a rollback is a revert. The pattern in this guide does not depend on the CRD; if the project stalled tomorrow, plain Pods plus Steps 1–3 and 5–7 would still stand.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Do not give an agent write access to anything production-shaped in v1 of your rollout.&lt;/strong&gt; Read-only tools first. The sandbox limits blast radius; it does not make a bad write un-happen. Irreversible actions (sends, deletes, merges, payments) belong behind human approval at the application layer — no kernel feature gives you that.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Do not share warm pools across trust boundaries&lt;/strong&gt;, and do not return used sandboxes to a pool. Ever. (Step 4 — it bears repeating because it is the mistake the abstraction makes easiest.)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Do not skip the egress work because "443 is fine."&lt;/strong&gt; The metadata endpoint, internal services on 443, and simple data exfiltration to an attacker-controlled host all fit through a careless 443 rule. The proxy or FQDN allowlist is not polish; it is the control.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Do not run agent workloads on your general-purpose nodes "temporarily"&lt;/strong&gt; without at least the RuntimeClass layer. Temporary has a way of becoming the architecture.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The final hardening checklist
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Namespace and resources&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;[ ] Dedicated &lt;code&gt;agents&lt;/code&gt; namespace; Pod Security Standards enforced at &lt;code&gt;restricted&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;[ ] ResourceQuota caps pods, CPU, memory and ephemeral storage for the whole namespace&lt;/li&gt;
&lt;li&gt;[ ] LimitRange defaults set so no container runs unbounded&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://dev.toScale-up"&gt; &lt;/a&gt; Agent workloads tainted onto a dedicated node pool&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Identity&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;[ ] Agent runtime ServiceAccount with &lt;code&gt;automountServiceAccountToken: false&lt;/code&gt; and no RoleBindings&lt;/li&gt;
&lt;li&gt;[ ] Orchestrator identity scoped to claims/sandboxes in the agents namespace only&lt;/li&gt;
&lt;li&gt;[ ] No kubeconfigs, cloud key files or shared tokens inside the sandbox image&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Kernel and pod hardening&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;[ ] gVisor (or Kata) registered on nodes; RuntimeClass created and verified with a test pod&lt;/li&gt;
&lt;li&gt;[ ] Template sets &lt;code&gt;runtimeClassName&lt;/code&gt;, &lt;code&gt;runAsNonRoot&lt;/code&gt;, seccomp &lt;code&gt;RuntimeDefault&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;[ ] Containers: &lt;code&gt;allowPrivilegeEscalation: false&lt;/code&gt;, &lt;code&gt;readOnlyRootFilesystem: true&lt;/code&gt;, all capabilities dropped&lt;/li&gt;
&lt;li&gt;[ ] Per-container requests and limits, including ephemeral storage&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Lifecycle&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;[ ] Agent Sandbox installed pinned (v1.0.10 at this writing), CRDs and controller verified&lt;/li&gt;
&lt;li&gt;[ ] One hardened SandboxTemplate per workload type, owned and reviewed by the platform team&lt;/li&gt;
&lt;li&gt;[ ] Warm pools are per-trust-boundary; claimed sandboxes are never reused&lt;/li&gt;
&lt;li&gt;[ ] Claims carry lifecycle expiry (&lt;code&gt;ttlSecondsAfterFinished&lt;/code&gt; + &lt;code&gt;shutdownPolicy: Delete&lt;/code&gt;)&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Network&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;[ ] Default-deny NetworkPolicy (ingress and egress) on the namespace&lt;/li&gt;
&lt;li&gt;[ ] Egress allows DNS and named destinations only; no port-80 egress (metadata stays unreachable)&lt;/li&gt;
&lt;li&gt;[ ] FQDN allowlist or egress proxy in place — not bare "443 to the world"&lt;/li&gt;
&lt;li&gt;[ ] Policy enforcement verified with a live test, not assumed from the YAML&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Secrets and audit&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;[ ] LLM credentials are dedicated, spend-capped keys in a namespace-scoped Secret; sandbox fails closed without them&lt;/li&gt;
&lt;li&gt;[ ] Per-session/execution-scoped tokens where the design allows; projected audience-bound tokens for any in-pod API need&lt;/li&gt;
&lt;li&gt;[ ] API audit covers sandbox/claim create and delete; pod and tool-call logs ship outside the sandbox&lt;/li&gt;
&lt;li&gt;[ ] Alerts on sandbox lifetime anomalies, quota saturation and policy drops&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  FAQ
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Is a regular container not sandboxed enough?&lt;/strong&gt;&lt;br&gt;
A regular container under &lt;code&gt;runc&lt;/code&gt; shares the host kernel — namespaces and cgroups isolate resources and visibility, but every syscall lands on the same kernel your other tenants use. For trusted, reviewed code that trade-off is normal. For code generated at runtime from untrusted instructions, the extra kernel boundary (gVisor or Kata) is the layer that turns a container escape into a much harder problem.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Does Agent Sandbox replace gVisor?&lt;/strong&gt;&lt;br&gt;
No — they solve different problems and compose. Agent Sandbox is an orchestrator: it manages the lifecycle of singleton, stateful sandbox pods (templates, warm pools, claims, expiry). gVisor and Kata are runtimes: they provide the actual isolation boundary, and the sandbox's pod template selects one via &lt;code&gt;runtimeClassName&lt;/code&gt;. Using the CRD without a secure runtime gives you tidy lifecycle management around an ordinary container.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Can I build this without the Agent Sandbox CRD?&lt;/strong&gt;&lt;br&gt;
Yes. Plain Pods (or one-member StatefulSets) with the same namespace, RBAC, RuntimeClass, NetworkPolicy and secret patterns give you the same security posture. What you give up is the ergonomic lifecycle: templates as a single reviewed source, warm pools for fast session starts, claims with enforced expiry. Plenty of teams start with plain Pods and adopt the CRD when session volume makes the lifecycle worth a controller.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;gVisor or Kata Containers — which should I choose?&lt;/strong&gt;&lt;br&gt;
Start with gVisor unless you know otherwise: it runs on ordinary nodes and fits interpreter/shell/HTTP-shaped agent workloads well. Choose Kata when your nodes support KVM (or nested virtualisation) and you want VM-grade separation, or when your workloads trip over gVisor's unimplemented syscalls. Because the choice lives in one &lt;code&gt;runtimeClassName&lt;/code&gt; line per template, you can run both and assign per workload type.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Is Agent Sandbox production-ready?&lt;/strong&gt;&lt;br&gt;
Honestly: it is a young, fast-moving SIG project — APIs at &lt;code&gt;v1beta1&lt;/code&gt;, releases landing roughly monthly, and occasional behaviour changes that need operator action (recent router NetworkPolicy changes are a good example). The safe way to adopt it is the way this guide is structured: let the CRD own lifecycle convenience, let the boring, stable primitives underneath own the security, pin your version, and test upgrades like you would any controller. Teams that need a fully settled API today can run the same architecture on plain Pods and adopt the CRD later without redesigning anything.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Sources: kubernetes-sigs/agent-sandbox — project README, releases (v1.0.10) and official examples (Sandbox, SandboxTemplate, SandboxClaim, SandboxWarmPool manifests), checked 12 Nov 2026; gVisor documentation, containerd configuration (runsc runtime type &lt;code&gt;io.containerd.runsc.v1&lt;/code&gt;); CNCF and SlashData, "State of Cloud Native Development Q3 2026" — CNCF announcement, 10 Nov 2026 (49% of developers integrating AI reporting agentic systems in production); Kubernetes documentation on RuntimeClass and Pod Security Standards. Project status, versions and statistics checked against these sources on 12 Nov 2026.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>kubernetes</category>
      <category>ai</category>
      <category>devops</category>
      <category>security</category>
    </item>
    <item>
      <title>Ingress NGINX Is Retired and Unpatched: A Working Engineer's Migration Checklist to Gateway API (2026)</title>
      <dc:creator>Rabin Adhikari</dc:creator>
      <pubDate>Thu, 12 Nov 2026 08:48:52 +0000</pubDate>
      <link>https://dev.to/softverseadmin/ingress-nginx-is-retired-and-unpatched-a-working-engineers-migration-checklist-to-gateway-api-32pm</link>
      <guid>https://dev.to/softverseadmin/ingress-nginx-is-retired-and-unpatched-a-working-engineers-migration-checklist-to-gateway-api-32pm</guid>
      <description>&lt;h1&gt;
  
  
  Ingress NGINX Is Retired and Unpatched: A Working Engineer's Migration Checklist to Gateway API (2026)
&lt;/h1&gt;

&lt;blockquote&gt;
&lt;h2&gt;
  
  
  TL;DR
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The community Ingress NGINX controller (&lt;code&gt;kubernetes/ingress-nginx&lt;/code&gt;) is retired.&lt;/strong&gt; Kubernetes announced it on 11 November 2025, shipped the final releases on 19 March 2026, and archived the repository on 24 March 2026. There will be no further releases, bugfixes, or security patches — ever.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;This is no longer theoretical:&lt;/strong&gt; on 5 November 2026 the Kubernetes Security Response Committee published CVE-2026-116472 against ingress-nginx and stated plainly, "Kubernetes will not be issuing a fix for this CVE." Every future CVE gets the same answer.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The Ingress API itself is not deprecated&lt;/strong&gt; — it is GA and frozen. Only this one controller is dead. F5's separate NGINX Ingress Controller is also unaffected. What you are replacing is the controller, and Gateway API is the replacement Kubernetes recommends.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The migration risk is not the YAML conversion — it is the annotations.&lt;/strong&gt; ingress-nginx behaviours you rely on (regex paths, rewrites, canary weights, snippets) either change meaning or vanish under Gateway API. This checklist inventories them first, then converts.&lt;/li&gt;
&lt;/ul&gt;
&lt;/blockquote&gt;

&lt;p&gt;If you run production traffic through ingress-nginx, you are now running it through unmaintained software that sits on a public IP and parses untrusted input for a living. The controller still works — that is exactly the trap. Nothing pages you today; the risk compounds silently with every new CVE disclosure, and the project's vulnerability pattern (annotation-driven configuration injection, the same class behind IngressNightmare in March 2025) means the next one is a matter of time, not imagination.&lt;/p&gt;

&lt;p&gt;This is the migration runbook in checklist form: stop the bleeding today, inventory what you actually depend on, learn which annotations silently break, pick a Gateway API implementation, convert, verify, cut over, and keep a rollback that actually works.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why migrate now: the timeline, verified
&lt;/h2&gt;

&lt;p&gt;Every date below is from the Kubernetes project's own announcements and advisories:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Date&lt;/th&gt;
&lt;th&gt;What happened&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;11 Nov 2025&lt;/td&gt;
&lt;td&gt;SIG Network and the Security Response Committee &lt;a href="https://kubernetes.io/blog/2025/11/11/ingress-nginx-retirement/" rel="noopener noreferrer"&gt;announce the retirement&lt;/a&gt;: best-effort maintenance until March 2026, then "no further releases, no bugfixes, and no updates to resolve any security vulnerabilities." The planned replacement controller, InGate, is retired alongside it.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;19 Mar 2026&lt;/td&gt;
&lt;td&gt;Final ingress-nginx releases ship: controller v1.15.1, v1.14.5, v1.13.9. There will never be a v1.15.2 from the Kubernetes project.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;24 Mar 2026&lt;/td&gt;
&lt;td&gt;The &lt;code&gt;kubernetes/ingress-nginx&lt;/code&gt; GitHub repository is archived and made read-only.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;5 Nov 2026&lt;/td&gt;
&lt;td&gt;CVE-2026-116472 is published: the &lt;code&gt;mirror-target&lt;/code&gt; annotation can be used to reach an unauthenticated configuration API inside the controller and drop all traffic through it. The &lt;a href="https://discuss.kubernetes.io/t/security-advisory-cve-2026-116472-ingress-nginx-mirror-target-self-dos/35554" rel="noopener noreferrer"&gt;official advisory&lt;/a&gt; states Kubernetes will not issue a fix — the first post-retirement proof of what "retired" means.&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Three clarifications that prevent wasted work:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;You do not have to leave the Ingress API.&lt;/strong&gt; It remains GA and supported; it is simply frozen. You could move to another maintained Ingress controller instead. Gateway API is the recommended target because it is where the ecosystem's development now happens — and because every major controller now implements it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;F5's NGINX Ingress Controller is a different project.&lt;/strong&gt; Similar name, different repository, different maintainers, still maintained. If you run F5's controller, this article is not about you.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Existing deployments keep running.&lt;/strong&gt; Nobody is remotely disabling your controller. The decision is not "migrate or it stops" — it is "migrate on your schedule, or migrate during an incident on a CVE's schedule."&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Step 0 — Today's stopgap for CVE-2026-116472 (30 minutes)
&lt;/h2&gt;

&lt;p&gt;Do this before the migration project, whatever its timeline. The advisory's own mitigations:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;[ ] &lt;strong&gt;Set the annotation risk level.&lt;/strong&gt; In the ingress-nginx controller ConfigMap, set &lt;code&gt;annotations-risk-level: "Medium"&lt;/code&gt; (or &lt;code&gt;"Low"&lt;/code&gt;). The &lt;code&gt;mirror-target&lt;/code&gt; annotation is classified above Medium, so this blocks the CVE's vector. Warning: if any team legitimately uses High/Critical-risk annotations, this will break them — check the inventory in Step 1 first, or apply this and watch for rejected Ingresses.&lt;/li&gt;
&lt;li&gt;[ ] &lt;strong&gt;Or reject the annotation at admission.&lt;/strong&gt; Use your policy engine (Kyverno, OPA/Gatekeeper, or a ValidatingAdmissionPolicy) to reject any Ingress carrying &lt;code&gt;nginx.ingress.kubernetes.io/mirror-target&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;[ ] &lt;strong&gt;Check for existing use and abuse.&lt;/strong&gt; Search your clusters for &lt;code&gt;mirror-target&lt;/code&gt; values pointing at localhost addresses — the advisory flags that as an exploitation indicator: &lt;code&gt;kubectl get ingress -A -o yaml | grep -B5 "mirror-target"&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;[ ] &lt;strong&gt;Remember who can pull this trigger:&lt;/strong&gt; anyone with permission to create an Ingress. Review which namespaces and service accounts hold Ingress write access — that is now your real privilege boundary for this controller (our &lt;a href="https://www.devopsenginer.com/blog/kubernetes-rbac-guide-2026" rel="noopener noreferrer"&gt;Kubernetes RBAC guide&lt;/a&gt; covers the audit pattern).&lt;/li&gt;
&lt;li&gt;[ ] &lt;strong&gt;Know your fork status.&lt;/strong&gt; The only patched build of this CVE is SUSE's Rancher Prime fork (1.15.2-prime1). Managed add-ons from cloud vendors may carry their own support terms — check your vendor's statement rather than assuming upstream status applies to you.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Step 1 — Inventory: find every install and every annotation
&lt;/h2&gt;

&lt;p&gt;You cannot convert what you have not counted. Run this on every cluster, including the staging cluster everyone forgets.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Find the controller itself&lt;/strong&gt; (the command from the official retirement post):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl get pods --all-namespaces --selector app.kubernetes.io/name=ingress-nginx
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Also check for installs under custom release names, and for more than one ingress-nginx per cluster — shared platform clusters often have two.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Count the estate:&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl get ingress -A &lt;span class="o"&gt;|&lt;/span&gt; wc -l
kubectl get ingressclass
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Extract every nginx annotation in use, with counts.&lt;/strong&gt; This output is the single most valuable artifact of the whole migration — it is your conversion workload, sorted:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl get ingress -A -o json &lt;span class="o"&gt;|&lt;/span&gt; jq -r &lt;span class="s1"&gt;'
  .items[]
  | (.metadata.annotations // {}) | keys[]
  | select(startswith("nginx.ingress.kubernetes.io/"))
'&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; sort &lt;span class="o"&gt;|&lt;/span&gt; uniq -c &lt;span class="o"&gt;|&lt;/span&gt; sort -rn
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Per-Ingress annotation detail&lt;/strong&gt;, for the conversion tickets:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl get ingress -A -o json &lt;span class="o"&gt;|&lt;/span&gt; jq -r &lt;span class="s1"&gt;'
  .items[]
  | .metadata.namespace + "/" + .metadata.name + "  "
  + (((.metadata.annotations // {}) | keys
      | map(select(startswith("nginx.ingress.kubernetes.io/")))) | join(", "))
'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Do not forget the controller-level configuration.&lt;/strong&gt; Global behaviour lives outside the Ingress objects:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;[ ] Dump the controller ConfigMap (&lt;code&gt;kubectl -n ingress-nginx get cm ingress-nginx-controller -o yaml&lt;/code&gt;): body-size defaults, timeout defaults, &lt;code&gt;proxy-real-ip-cidr&lt;/code&gt;, HSTS, error pages, Lua plugins.&lt;/li&gt;
&lt;li&gt;[ ] List every &lt;code&gt;*-snippet&lt;/code&gt; annotation and every ConfigMap snippet. &lt;strong&gt;Snippets have no Gateway API equivalent at all&lt;/strong&gt; — each one is a small redesign, not a translation.&lt;/li&gt;
&lt;li&gt;[ ] Note TLS termination points, default backends, and any Ingress with &lt;code&gt;pathType: ImplementationSpecific&lt;/code&gt; — that pathType is a smell that regex behaviour is in play.&lt;/li&gt;
&lt;li&gt;[ ] Record which teams own which namespaces. The annotation report by namespace is your migration plan's work-breakdown.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Step 2 — The annotations that silently break or change behaviour
&lt;/h2&gt;

&lt;p&gt;This is the section that decides whether your migration is boring or an incident. Gateway API is a smaller, stricter, portable API. ingress-nginx is a decade of accumulated NGINX behaviours. Where they differ, nothing errors — traffic just behaves differently.&lt;/p&gt;

&lt;h3&gt;
  
  
  Behaviour differences that exist even with zero annotations
&lt;/h3&gt;

&lt;p&gt;From the Kubernetes project's own pre-migration write-up (&lt;a href="https://kubernetes.io/blog/2026/02/27/ingress-nginx-before-you-migrate/" rel="noopener noreferrer"&gt;Five Surprising Ingress-NGINX Behaviors&lt;/a&gt;) — verify each against your estate:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Regex paths are prefix matches, and case-insensitive.&lt;/strong&gt; An ingress-nginx regex path matches anything &lt;em&gt;starting with&lt;/em&gt; a match, ignoring case. Gateway API &lt;code&gt;RegularExpression&lt;/code&gt; matches are exact semantics per the implementation — paths you believed were narrow may have been wide, and vice versa after conversion.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;use-regex&lt;/code&gt; leaks across Ingresses.&lt;/strong&gt; Enabling it on one Ingress makes &lt;em&gt;every&lt;/em&gt; path for that host, across all Ingresses, regex. Gateway API has no such cross-object side effect — behaviour that "just worked" because of the leak stops working.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Missing trailing slash gets a free 301.&lt;/strong&gt; Requests to a path without a trailing slash are redirected to the slashed version. Gateway API does not do this implicitly; clients and health checks that depended on it see 404s.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;URLs are normalised before matching.&lt;/strong&gt; Dot segments and duplicate slashes are collapsed first. If your backends or your security rules assumed the raw path, test encoded and unnormalised paths explicitly after migration.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The default backend catches more than you think.&lt;/strong&gt; Unmatched hosts and paths land on the ingress-nginx default backend with its specific response behaviour. Your Gateway needs an explicit catch-all route or the error shape changes.&lt;/li&gt;
&lt;/ol&gt;

&lt;h3&gt;
  
  
  Annotation mapping: what converts, what changes, what dies
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;ingress-nginx annotation&lt;/th&gt;
&lt;th&gt;Gateway API equivalent&lt;/th&gt;
&lt;th&gt;Verdict&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;rewrite-target&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;URLRewrite&lt;/code&gt; filter (&lt;code&gt;ReplacePrefixMatch&lt;/code&gt; / &lt;code&gt;ReplaceFullPath&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;⚠️ Converts for prefix rewrites. Regex capture groups (&lt;code&gt;/$2&lt;/code&gt;) have &lt;strong&gt;no equivalent&lt;/strong&gt; — restructure the paths instead.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;use-regex: "true"&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;path.type: RegularExpression&lt;/code&gt; per match&lt;/td&gt;
&lt;td&gt;⚠️ Converts per-route, but loses the cross-Ingress side effect (see above) and prefix/case behaviour differs.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;canary: "true"&lt;/code&gt; + &lt;code&gt;canary-weight&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Weighted &lt;code&gt;backendRefs&lt;/code&gt; on one HTTPRoute rule&lt;/td&gt;
&lt;td&gt;✅ Converts cleanly — see Step 4.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;canary-by-header&lt;/code&gt; / &lt;code&gt;canary-by-cookie&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Header/cookie match on a separate rule&lt;/td&gt;
&lt;td&gt;✅ Converts, as an explicit match instead of a shadow Ingress.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;ssl-redirect&lt;/code&gt; / &lt;code&gt;force-ssl-redirect&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;RequestRedirect&lt;/code&gt; filter (scheme &lt;code&gt;https&lt;/code&gt;) on the HTTP listener route&lt;/td&gt;
&lt;td&gt;✅ Converts.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;permanent-redirect&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;RequestRedirect&lt;/code&gt; filter with &lt;code&gt;statusCode: 301&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;✅ Converts.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;enable-cors&lt;/code&gt; + &lt;code&gt;cors-*&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;CORS&lt;/code&gt; filter on HTTPRoute (Standard channel since Gateway API v1.5)&lt;/td&gt;
&lt;td&gt;✅ Converts.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;proxy-body-size&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;td&gt;❌ No standard field. Implementation-specific (e.g. a policy object). Silently dropping it means your data plane's default body limit applies — find out what it is.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;proxy-connect-timeout&lt;/code&gt; / &lt;code&gt;proxy-read-timeout&lt;/code&gt; / &lt;code&gt;proxy-send-timeout&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;timeouts&lt;/code&gt; on the HTTPRoute rule&lt;/td&gt;
&lt;td&gt;⚠️ The API has &lt;code&gt;timeouts.request&lt;/code&gt; / &lt;code&gt;timeouts.backendRequest&lt;/code&gt;, but support varies by implementation. Verify against your controller's conformance, don't assume.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;auth-url&lt;/code&gt; / &lt;code&gt;auth-signin&lt;/code&gt; / &lt;code&gt;auth-method&lt;/code&gt; (external auth)&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;td&gt;❌ No standard equivalent yet. Each implementation has its own extension (Envoy Gateway's SecurityPolicy, NGINX Gateway Fabric's policies, Traefik middleware). Budget real redesign time.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;affinity&lt;/code&gt; / &lt;code&gt;session-cookie-*&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Session persistence&lt;/td&gt;
&lt;td&gt;⚠️ Not portable: it lives in the Experimental channel / implementation-specific configuration, and Gateway API v1.7 reshaped it (moved from the route to the backend). Pin behaviour per implementation.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;limit-rps&lt;/code&gt; / &lt;code&gt;limit-connections&lt;/code&gt; (rate limiting)&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;td&gt;❌ No standard rate-limit API. Implementation policy objects only.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;configuration-snippet&lt;/code&gt; / &lt;code&gt;server-snippet&lt;/code&gt; / &lt;code&gt;location-snippet&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;td&gt;❌ &lt;strong&gt;Dies, by design.&lt;/strong&gt; Raw NGINX config injection was the project's core security problem. Every snippet is a feature to re-implement properly or retire.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;mirror-target&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;td&gt;❌ No equivalent — and it is the CVE-2026-116472 vector. If you use traffic mirroring, it becomes an implementation feature (e.g. &lt;code&gt;RequestMirror&lt;/code&gt; filter exists for mirroring to a backend — check semantics before assuming parity).&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;backend-protocol: HTTPS&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;BackendTLSPolicy&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;✅ Converts — TLS to the backend is a first-class policy now.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;x-forwarded-prefix&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;URLRewrite&lt;/code&gt; + header handling per implementation&lt;/td&gt;
&lt;td&gt;⚠️ Partially. Test what your backends actually read.&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Rule of the migration:&lt;/strong&gt; any row marked ❌ in your inventory is a design task with an owner and a date, not a line in a conversion script. If Step 1 shows snippets or external auth in production, your timeline is set by those rows, not by the easy 90%.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 3 — Choose a Gateway API implementation
&lt;/h2&gt;

&lt;p&gt;Gateway API is a specification; you still run a controller. Current release: &lt;strong&gt;Gateway API v1.7.0&lt;/strong&gt; (October 2026), in which HTTPRoute retries graduated to the Standard channel. Install the CRDs from the Standard channel unless you knowingly need Experimental features:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl apply --server-side -f \
  https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.7.0/standard-install.yaml
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The realistic shortlist, by situation rather than by hype:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;If you…&lt;/th&gt;
&lt;th&gt;Start with&lt;/th&gt;
&lt;th&gt;Why&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Want the closest "NGINX, but maintained" landing&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;NGINX Gateway Fabric&lt;/strong&gt; (F5)&lt;/td&gt;
&lt;td&gt;Same data plane family, F5-maintained, implements Gateway API natively. Familiar failure modes.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Want a neutral, Envoy-based default&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;Envoy Gateway&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;CNCF project built specifically as a Gateway API implementation; clean policy model (BackendTrafficPolicy, SecurityPolicy) for the ❌ rows above.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Already run Istio&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;Istio&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Its Gateway API support is its recommended ingress path; no second data plane to operate.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Already run Cilium as CNI&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;Cilium&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Gateway API support built into the networking layer you already run.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Are on GKE&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;GKE Gateway Controller&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Google-managed, integrates with Google Cloud load balancing and policies.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Are on EKS&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;AWS Load Balancer Controller&lt;/strong&gt; (Gateway API) or a self-managed option above&lt;/td&gt;
&lt;td&gt;LBC's Gateway API support provisions ALBs from Gateway resources; verify the specific features you need — its supported subset is documented and narrower than the full API. (Context: our &lt;a href="https://www.devopsenginer.com/blog/amazon-eks-guide-2026" rel="noopener noreferrer"&gt;Amazon EKS guide&lt;/a&gt;.)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Run Traefik today&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;Traefik&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Implements Gateway API alongside its Ingress support, easing a staged move.&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Decision rules that matter more than the brand:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;[ ] &lt;strong&gt;Check the conformance profile for the exact features you use&lt;/strong&gt; (CORS, rewrites, timeouts, GRPCRoute), not the logo on the implementations page.&lt;/li&gt;
&lt;li&gt;[ ] &lt;strong&gt;Match the version pairing:&lt;/strong&gt; install the Gateway API CRD version your controller's release was tested against, not blindly the newest.&lt;/li&gt;
&lt;li&gt;[ ] &lt;strong&gt;Decide who owns the Gateway.&lt;/strong&gt; Gateway API splits ownership: platform team owns GatewayClass + Gateway, app teams own HTTPRoutes. If your org chart cannot support that split, a single-team Gateway per namespace is the pragmatic start.&lt;/li&gt;
&lt;li&gt;[ ] &lt;strong&gt;Confirm your ❌ rows.&lt;/strong&gt; External auth, rate limiting, body size: get the implementation's policy object for each one &lt;em&gt;in writing in the ticket&lt;/em&gt; before committing.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Step 4 — Convert: ingress2gateway first, hand-fix second
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Generate a first draft automatically.&lt;/strong&gt; The Kubernetes project's &lt;a href="https://github.com/kubernetes-sigs/ingress2gateway" rel="noopener noreferrer"&gt;ingress2gateway&lt;/a&gt; tool (1.0 released 20 March 2026; current release v1.2.0) reads your live Ingresses — or exported YAML — and emits Gateway + HTTPRoute resources, with warnings for constructs it cannot translate:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;go install github.com/kubernetes-sigs/ingress2gateway@v1.2.0
ingress2gateway print --providers=ingress-nginx &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; gateway-draft.yaml
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Treat the output as scaffolding. Every warning it prints maps to a row in the Step 2 table; clear them one by one. (If you manage Ingresses through Helm or GitOps, convert the &lt;em&gt;rendered&lt;/em&gt; manifests, then work the result back into your charts — our &lt;a href="https://www.devopsenginer.com/blog/helm-charts-tutorial-2026" rel="noopener noreferrer"&gt;Helm tutorial&lt;/a&gt; and &lt;a href="https://www.devopsenginer.com/blog/argo-cd-tutorial-gitops-2026" rel="noopener noreferrer"&gt;Argo CD GitOps guide&lt;/a&gt; cover the repo side.)&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;A worked conversion.&lt;/strong&gt; This Ingress carries the three most common behaviours — regex rewrite, forced HTTPS, and a second canary Ingress (shown merged here as weighted backends):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;networking.k8s.io/v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Ingress&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;app&lt;/span&gt;
  &lt;span class="na"&gt;namespace&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;production&lt;/span&gt;
  &lt;span class="na"&gt;annotations&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;nginx.ingress.kubernetes.io/use-regex&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"true"&lt;/span&gt;
    &lt;span class="na"&gt;nginx.ingress.kubernetes.io/rewrite-target&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;/$2&lt;/span&gt;
    &lt;span class="na"&gt;nginx.ingress.kubernetes.io/ssl-redirect&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"true"&lt;/span&gt;
&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;ingressClassName&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;nginx&lt;/span&gt;
  &lt;span class="na"&gt;tls&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;hosts&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"app.example.com"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
      &lt;span class="na"&gt;secretName&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;app-tls&lt;/span&gt;
  &lt;span class="na"&gt;rules&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;host&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;app.example.com&lt;/span&gt;
      &lt;span class="na"&gt;http&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="na"&gt;paths&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;path&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;/api(/|$)(.*)&lt;/span&gt;
            &lt;span class="na"&gt;pathType&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ImplementationSpecific&lt;/span&gt;
            &lt;span class="na"&gt;backend&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
              &lt;span class="na"&gt;service&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;api&lt;/span&gt;
                &lt;span class="na"&gt;port&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="s"&gt;number:&lt;/span&gt; &lt;span class="mi"&gt;8080&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
          &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;path&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;/&lt;/span&gt;
            &lt;span class="na"&gt;pathType&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Prefix&lt;/span&gt;
            &lt;span class="na"&gt;backend&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
              &lt;span class="na"&gt;service&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;web&lt;/span&gt;
                &lt;span class="na"&gt;port&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="s"&gt;number:&lt;/span&gt; &lt;span class="mi"&gt;80&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The Gateway API shape — note that the regex-capture rewrite (&lt;code&gt;/$2&lt;/code&gt;) does &lt;strong&gt;not&lt;/strong&gt; survive; the honest conversion changes the match to a prefix and the rewrite to &lt;code&gt;ReplacePrefixMatch&lt;/code&gt;. If your API truly depends on capture-group rewrites, that is a ❌ row: fix the paths at the app or accept an implementation-specific filter:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;gateway.networking.k8s.io/v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Gateway&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;app-gateway&lt;/span&gt;
  &lt;span class="na"&gt;namespace&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;production&lt;/span&gt;
&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;gatewayClassName&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;envoy&lt;/span&gt;        &lt;span class="c1"&gt;# your implementation's class
&lt;/span&gt;  &lt;span class="na"&gt;listeners&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;http&lt;/span&gt;
      &lt;span class="na"&gt;protocol&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;HTTP&lt;/span&gt;
      &lt;span class="na"&gt;port&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;80&lt;/span&gt;
    &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;https&lt;/span&gt;
      &lt;span class="na"&gt;protocol&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;HTTPS&lt;/span&gt;
      &lt;span class="na"&gt;port&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;443&lt;/span&gt;
      &lt;span class="na"&gt;tls&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="na"&gt;mode&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Terminate&lt;/span&gt;
        &lt;span class="na"&gt;certificateRefs&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;app-tls&lt;/span&gt;
&lt;span class="nn"&gt;---
&lt;/span&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;gateway.networking.k8s.io/v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;HTTPRoute&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;app-https-redirect&lt;/span&gt;
  &lt;span class="na"&gt;namespace&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;production&lt;/span&gt;
&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;parentRefs&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;app-gateway&lt;/span&gt;
      &lt;span class="na"&gt;sectionName&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;http&lt;/span&gt;
  &lt;span class="na"&gt;hostnames&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"app.example.com"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
  &lt;span class="na"&gt;rules&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;filters&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;type&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;RequestRedirect&lt;/span&gt;
          &lt;span class="na"&gt;requestRedirect&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="na"&gt;scheme&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;https&lt;/span&gt;
            &lt;span class="na"&gt;statusCode&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;301&lt;/span&gt;
&lt;span class="nn"&gt;---
&lt;/span&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;gateway.networking.k8s.io/v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;HTTPRoute&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;app&lt;/span&gt;
  &lt;span class="na"&gt;namespace&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;production&lt;/span&gt;
&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;parentRefs&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;app-gateway&lt;/span&gt;
      &lt;span class="na"&gt;sectionName&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;https&lt;/span&gt;
  &lt;span class="na"&gt;hostnames&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"app.example.com"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
  &lt;span class="na"&gt;rules&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;matches&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;path&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="na"&gt;type&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;PathPrefix&lt;/span&gt;
            &lt;span class="na"&gt;value&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;/api&lt;/span&gt;
      &lt;span class="na"&gt;filters&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;type&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;URLRewrite&lt;/span&gt;
          &lt;span class="na"&gt;urlRewrite&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="na"&gt;path&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
              &lt;span class="na"&gt;type&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ReplacePrefixMatch&lt;/span&gt;
              &lt;span class="na"&gt;replacePrefixMatch&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;/&lt;/span&gt;
      &lt;span class="na"&gt;backendRefs&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;api&lt;/span&gt;
          &lt;span class="na"&gt;port&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;8080&lt;/span&gt;
    &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;matches&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;path&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="na"&gt;type&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;PathPrefix&lt;/span&gt;
            &lt;span class="na"&gt;value&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;/&lt;/span&gt;
      &lt;span class="na"&gt;backendRefs&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;web&lt;/span&gt;
          &lt;span class="na"&gt;port&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;80&lt;/span&gt;
          &lt;span class="na"&gt;weight&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;90&lt;/span&gt;          &lt;span class="c1"&gt;# was: canary Ingress with canary-weight
&lt;/span&gt;        &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;web-v2&lt;/span&gt;
          &lt;span class="na"&gt;port&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;80&lt;/span&gt;
          &lt;span class="na"&gt;weight&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;What changed, deliberately: one Ingress became three objects with separate owners; the canary stopped being a shadow Ingress and became arithmetic on one rule; TLS moved from the route to the listener; the regex became a prefix because the capture-group semantics were the thing being retired.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Validate before applying anywhere:&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl apply --dry-run=server -f gateway-draft.yaml
kubectl apply -f gateway-draft.yaml
kubectl get gateway,httproute -n production
kubectl describe httproute app -n production   &lt;span class="c"&gt;# Status.Parents: Accepted + Programmed&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;An HTTPRoute whose status is not &lt;code&gt;Accepted: True&lt;/code&gt; and &lt;code&gt;Programmed: True&lt;/code&gt; is not routing anything — this status check replaces ingress-nginx's "apply and pray," and it belongs in your CI as well as your runbook. (The networking model underneath — Services, endpoints, DNS — is unchanged; our &lt;a href="https://www.devopsenginer.com/blog/kubernetes-ingress-networking-basics-2026" rel="noopener noreferrer"&gt;Kubernetes networking &amp;amp; ingress basics&lt;/a&gt; post covers it end to end.)&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 5 — TLS, explicitly
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;[ ] &lt;strong&gt;Certificates live on the listener now&lt;/strong&gt; (&lt;code&gt;certificateRefs&lt;/code&gt;), not on each route. One wildcard on the Gateway can retire a shelf of per-Ingress secrets — or concentrate your blast radius. Choose consciously.&lt;/li&gt;
&lt;li&gt;[ ] &lt;strong&gt;cert-manager works with Gateway API&lt;/strong&gt; (its &lt;code&gt;gateway-shim&lt;/code&gt; provisions certificates from Gateway listeners). Enable the Gateway API support flag on cert-manager and test one non-production Gateway first; current cert-manager releases support the stable ListenerSet as well — check your version's release notes before relying on it.&lt;/li&gt;
&lt;li&gt;[ ] &lt;strong&gt;Cross-namespace references need a ReferenceGrant.&lt;/strong&gt; A route in namespace A pointing at a Service or Secret in namespace B is denied by default. This is a feature — and the number-one cause of "my route is Accepted but not Programmed" in multi-team clusters.&lt;/li&gt;
&lt;li&gt;[ ] &lt;strong&gt;Backend TLS is a policy, not an annotation:&lt;/strong&gt; &lt;code&gt;BackendTLSPolicy&lt;/code&gt; replaces &lt;code&gt;backend-protocol: HTTPS&lt;/code&gt; and gives you CA control via &lt;code&gt;validation.caCertificateRefs&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;[ ] &lt;strong&gt;Test the chain, not just the handshake:&lt;/strong&gt; SNI behaviour per hostname, HTTP→HTTPS redirect status code (301 vs 308 matters to API clients), and HSTS headers if your old ConfigMap set them.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Step 6 — Test and verify before any traffic moves
&lt;/h2&gt;

&lt;p&gt;Run the new Gateway in parallel, on its own address, and prove parity route by route:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;[ ] &lt;strong&gt;Status first:&lt;/strong&gt; every Gateway &lt;code&gt;Programmed&lt;/code&gt;, every HTTPRoute &lt;code&gt;Accepted&lt;/code&gt; + &lt;code&gt;Programmed&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;[ ] &lt;strong&gt;Golden-path parity:&lt;/strong&gt; for each host/path, compare old vs new on: status code, redirect target, response headers (especially CORS and cache headers), and body for the top endpoints. A diff script against both load-balancer addresses pays for itself immediately.&lt;/li&gt;
&lt;li&gt;[ ] &lt;strong&gt;The Step 2 traps, tested by name:&lt;/strong&gt; case-sensitivity of paths, trailing-slash behaviour, unnormalised paths (&lt;code&gt;//api&lt;/code&gt;, &lt;code&gt;/api/../api&lt;/code&gt;), oversized request bodies (your dropped &lt;code&gt;proxy-body-size&lt;/code&gt;), slow clients against timeouts, WebSocket/gRPC if you run them.&lt;/li&gt;
&lt;li&gt;[ ] &lt;strong&gt;404/503 shapes:&lt;/strong&gt; unmatched host, unmatched path, backend down. Monitoring and client retries often key off these bodies and codes.&lt;/li&gt;
&lt;li&gt;[ ] &lt;strong&gt;Load test the new data plane&lt;/strong&gt; at production-shaped traffic before cutover if the controller is new to you — you are changing the component on the hot path (the verification discipline from our &lt;a href="https://www.devopsenginer.com/blog/devsecops-checklist-securing-pipeline-2025" rel="noopener noreferrer"&gt;DevSecOps checklist&lt;/a&gt; applies: every claim in this step gets evidence, not vibes).&lt;/li&gt;
&lt;li&gt;[ ] &lt;strong&gt;Dashboards and alerts re-pointed&lt;/strong&gt; at the new controller's metrics before cutover, not after the first incident.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Step 7 — Cut over with a rollback that is a DNS change, not a project
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;[ ] &lt;strong&gt;Keep ingress-nginx running and untouched&lt;/strong&gt; through the whole cutover. Rollback = traffic back to the old address. Deleting the old controller "to force commitment" converts every surprise into an outage.&lt;/li&gt;
&lt;li&gt;[ ] &lt;strong&gt;Shift traffic gradually:&lt;/strong&gt; weighted DNS records (or your CDN / global load balancer) — 5% → 25% → 50% → 100%, with a bake period at each step and error-rate/latency comparisons against the old path. (Progressive-delivery discipline transfers directly — see our &lt;a href="https://www.devopsenginer.com/blog/argo-rollouts-progressive-delivery-2026" rel="noopener noreferrer"&gt;Argo Rollouts guide&lt;/a&gt;.)&lt;/li&gt;
&lt;li&gt;[ ] &lt;strong&gt;Pre-write the rollback trigger:&lt;/strong&gt; e.g. "5xx rate on the Gateway exceeds the ingress-nginx baseline by X for Y minutes → DNS weight back to 0, page platform on-call." Decide the numbers now, while nobody is tired.&lt;/li&gt;
&lt;li&gt;[ ] &lt;strong&gt;Hold the old path for an agreed bake period&lt;/strong&gt; (two weeks is a common, defensible choice) before decommissioning — long enough for monthly batch jobs and forgotten webhooks to have fired once.&lt;/li&gt;
&lt;li&gt;[ ] &lt;strong&gt;Decommission in order:&lt;/strong&gt; drain → remove DNS → uninstall controller → delete Ingress objects last, from Git, so the estate's history stays readable.&lt;/li&gt;
&lt;li&gt;[ ] &lt;strong&gt;Close the security loop:&lt;/strong&gt; remove the ingress-nginx admission webhooks and RBAC, and record the removal in your asset inventory — an unpatched controller that no longer routes traffic but still runs is still attack surface.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The final checklist
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;This week&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;[ ] CVE-2026-116472 stopgap applied (&lt;code&gt;annotations-risk-level&lt;/code&gt;, admission rule, or fork) on every cluster&lt;/li&gt;
&lt;li&gt;[ ] Every ingress-nginx install found, including staging and second installs&lt;/li&gt;
&lt;li&gt;[ ] Annotation inventory produced and sorted — the ❌ rows named&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Design&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;[ ] Gateway API implementation chosen; conformance checked for your exact features&lt;/li&gt;
&lt;li&gt;[ ] Every ❌ row (snippets, external auth, rate limits, body size) has an owner and a design&lt;/li&gt;
&lt;li&gt;[ ] Gateway ownership model decided (platform Gateway vs per-team)&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Build &amp;amp; verify&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;[ ] CRDs (v1.7.0 Standard channel) + controller installed alongside ingress-nginx&lt;/li&gt;
&lt;li&gt;[ ] ingress2gateway draft generated; every warning resolved or ticketed&lt;/li&gt;
&lt;li&gt;[ ] Parity tests pass, including the five behaviour traps and TLS checks&lt;/li&gt;
&lt;li&gt;[ ] Route status (&lt;code&gt;Accepted&lt;/code&gt;/&lt;code&gt;Programmed&lt;/code&gt;) gated in CI&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Cut over&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;[ ] Weighted traffic shift with bake periods and a written rollback trigger&lt;/li&gt;
&lt;li&gt;[ ] Old controller kept live through the bake period&lt;/li&gt;
&lt;li&gt;[ ] Decommission completed and recorded; webhooks/RBAC removed&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  FAQ
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Is ingress-nginx still safe to run in 2026?&lt;/strong&gt;&lt;br&gt;
It runs, and existing installs are not broken — but it receives no security fixes, and CVE-2026-116472 (November 2026) is the standing example: disclosed, rated, and deliberately unpatched by the Kubernetes project. "Safe" now means "you accept every future CVE as permanent." For an internet-facing data plane, that is not a defensible steady state.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Is the Kubernetes Ingress API deprecated too?&lt;/strong&gt;&lt;br&gt;
No. The Ingress API is GA, feature-frozen, and has no planned removal. Only the community ingress-nginx &lt;em&gt;controller&lt;/em&gt; retired. Staying on Ingress with a different maintained controller is a legitimate choice; Gateway API is the recommended one.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;I use F5's NGINX Ingress Controller. Am I affected?&lt;/strong&gt;&lt;br&gt;
No. F5's controller is a separate, still-maintained project. The retired one is the Kubernetes community's &lt;code&gt;kubernetes/ingress-nginx&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Can I just upgrade ingress-nginx to fix the November CVE?&lt;/strong&gt;&lt;br&gt;
There is nothing to upgrade to. The final upstream release is v1.15.1 (March 2026); the CVE was published eight months later with an explicit "will not fix." Only vendor forks (e.g. SUSE's Rancher Prime build) carry patches.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;How long does the migration take?&lt;/strong&gt;&lt;br&gt;
As planning guidance, not a promise: a host with plain prefix routing and TLS converts in hours. An estate with snippets, external auth, and regex rewrites is measured in weeks, and the long pole is always those annotation rows — which is why the inventory comes before any date is announced. (Cluster-wide change discipline in our &lt;a href="https://www.devopsenginer.com/blog/kubernetes-upgrades-guide-2026" rel="noopener noreferrer"&gt;Kubernetes upgrades guide&lt;/a&gt; applies unchanged.)&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What about InGate, the planned replacement?&lt;/strong&gt;&lt;br&gt;
Retired before it matured — the November 2025 announcement retires it alongside ingress-nginx. Do not plan a migration onto it.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Sources: Kubernetes blog, "Ingress NGINX Retirement: What You Need to Know" (11 Nov 2025); Kubernetes blog, "Before You Migrate: Five Surprising Ingress-NGINX Behaviors You Need to Know" (27 Feb 2026); Kubernetes Security Response Committee advisory, CVE-2026-116472 (5 Nov 2026); Gateway API v1.7.0 release notes (kubernetes-sigs/gateway-api); ingress2gateway project (kubernetes-sigs). Version and CVE facts checked against these sources on 12 Nov 2026.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>kubernetes</category>
      <category>devops</category>
      <category>gatewayapi</category>
      <category>networking</category>
    </item>
    <item>
      <title>I Built Three Free DevOps Tools: A Cron Translator, a VPC Subnet Planner and a Kubernetes YAML Checker</title>
      <dc:creator>Rabin Adhikari</dc:creator>
      <pubDate>Tue, 10 Nov 2026 08:48:12 +0000</pubDate>
      <link>https://dev.to/softverseadmin/i-built-three-free-devops-tools-a-cron-translator-a-vpc-subnet-planner-and-a-kubernetes-yaml-9j0</link>
      <guid>https://dev.to/softverseadmin/i-built-three-free-devops-tools-a-cron-translator-a-vpc-subnet-planner-and-a-kubernetes-yaml-9j0</guid>
      <description>&lt;h1&gt;
  
  
  I Built Three Free DevOps Tools: A Cron Translator, a VPC Subnet Planner and a Kubernetes YAML Checker
&lt;/h1&gt;

&lt;p&gt;This site has 51 guides, and a pattern I did not expect when I started writing them: the questions that come back — in comments, in my own day work, in the middle of incidents — are usually not knowledge problems. Nobody forgets what a subnet &lt;em&gt;is&lt;/em&gt; at 2 a.m. What they need at 2 a.m. is the arithmetic done correctly, a schedule converted without a dialect error, and a manifest checked against the ten mistakes that always slip review. Those are tool-shaped problems, not guide-shaped ones.&lt;/p&gt;

&lt;p&gt;So instead of guide number 52, I built the tools. There is now a &lt;a href="https://www.devopsenginer.com/tools" rel="noopener noreferrer"&gt;free tools section&lt;/a&gt; on this site — also in the navigation, under Tools — with three of them. Each one is the fast path for a friction the guides describe in depth, and each one links to its guide for when you want the deep path.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;TL;DR:&lt;/strong&gt; Three free tools, all running entirely in your browser — a &lt;a href="https://www.devopsenginer.com/tools/cron-translator" rel="noopener noreferrer"&gt;Cron Translator&lt;/a&gt; (one expression in, crontab / Kubernetes CronJob / GitHub Actions / EventBridge out, with the next 10 run times in UTC and JST), an &lt;a href="https://www.devopsenginer.com/tools/subnet-planner" rel="noopener noreferrer"&gt;AWS VPC Subnet Planner&lt;/a&gt; (carve a VPC CIDR across AZs with AWS-usable host counts — a /24 is 251, not 256), and a &lt;a href="https://www.devopsenginer.com/tools/kubernetes-yaml-checker" rel="noopener noreferrer"&gt;Kubernetes YAML Checker&lt;/a&gt; (pass/warn/fail on &lt;code&gt;:latest&lt;/code&gt; images, missing requests and limits, missing probes, privileged containers and more). No sign-up, nothing uploaded, no input stored.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;&lt;em&gt;Originally published on &lt;a href="https://www.devopsenginer.com/blog/i-built-three-free-devops-tools" rel="noopener noreferrer"&gt;DevOps Enginer&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  1. Cron Translator — the same schedule is four different schedules
&lt;/h2&gt;

&lt;p&gt;Paste a standard five-field cron expression and the &lt;a href="https://www.devopsenginer.com/tools/cron-translator" rel="noopener noreferrer"&gt;Cron Translator&lt;/a&gt; returns the correct form for Linux crontab, a Kubernetes CronJob, GitHub Actions &lt;code&gt;on.schedule&lt;/code&gt; and AWS EventBridge — plus a plain-English reading of the schedule, the next 10 run times in UTC and JST side by side, and a warning wherever a platform cannot express what you wrote.&lt;/p&gt;

&lt;p&gt;The worked example that justifies the tool's existence: &lt;code&gt;0 9 * * 1-5&lt;/code&gt; — 09:00, Monday to Friday. On a Tokyo server's crontab it fires at 09:00 JST, exactly as intended. Now carry it to the other platforms unchanged:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;EventBridge rejects the shape and renumbers the days.&lt;/strong&gt; EventBridge cron has six fields (a year field is added), requires exactly one of day-of-month or day-of-week to be &lt;code&gt;?&lt;/code&gt;, and counts days of the week 1–7 starting at Sunday instead of 0–6. The correct EventBridge form of the same schedule is &lt;code&gt;cron(0 9 ? * 2-6 *)&lt;/code&gt;. Copy-paste the crontab line and the rule fires on the wrong days — or never.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;GitHub Actions accepts the string and changes its meaning.&lt;/strong&gt; &lt;code&gt;on.schedule&lt;/code&gt; is always UTC and has no timezone setting, so your 09:00 job fires at 18:00 JST. It also enforces a minimum interval of once every 5 minutes, whatever you write.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Kubernetes is UTC too&lt;/strong&gt;, unless you set &lt;code&gt;spec.timeZone&lt;/code&gt; on the CronJob (Kubernetes 1.27+) — otherwise the schedule follows the kube-controller-manager's timezone, which is UTC in almost every cluster.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;And one trap that lives inside standard cron itself: when both day-of-month and day-of-week are restricted, cron runs when &lt;em&gt;either&lt;/em&gt; matches — &lt;code&gt;0 9 1 * 1&lt;/code&gt; is the 1st of the month &lt;em&gt;and&lt;/em&gt; every Monday. EventBridge cannot express that at all; it needs two separate rules, and the translator says so instead of mistranslating it. Quartz syntax (&lt;code&gt;?&lt;/code&gt;, &lt;code&gt;L&lt;/code&gt;, &lt;code&gt;W&lt;/code&gt;, &lt;code&gt;#&lt;/code&gt;) and 6- or 7-field expressions are flagged as unsupported, for the same reason.&lt;/p&gt;

&lt;p&gt;The deep version of all of this is the &lt;a href="https://www.devopsenginer.com/blog/kubernetes-jobs-cronjobs-guide-2026" rel="noopener noreferrer"&gt;Kubernetes Jobs and CronJobs guide&lt;/a&gt; — the full CronJob contract, deadlines and concurrency policies included.&lt;/p&gt;

&lt;h2&gt;
  
  
  2. AWS VPC Subnet Planner — a /24 is 251, and you cannot resize later
&lt;/h2&gt;

&lt;p&gt;Generic subnet calculators do one subnet at a time. A VPC is a &lt;em&gt;plan&lt;/em&gt;, so the &lt;a href="https://www.devopsenginer.com/tools/subnet-planner" rel="noopener noreferrer"&gt;Subnet Planner&lt;/a&gt; works at plan level: enter the VPC CIDR, choose a subnet size and count, set how many availability zones to spread across, and it carves the block up — every subnet's CIDR, network address, first usable, last usable, broadcast, total addresses and AWS-usable hosts, plus the leftover free space decomposed into clean CIDR blocks for the tiers you have not built yet.&lt;/p&gt;

&lt;p&gt;The number everyone gets wrong once: &lt;strong&gt;a /24 has 251 usable addresses, not 256.&lt;/strong&gt; AWS reserves five addresses in every subnet — &lt;code&gt;.0&lt;/code&gt; the network address, &lt;code&gt;.1&lt;/code&gt; the VPC router, &lt;code&gt;.2&lt;/code&gt; the DNS server, &lt;code&gt;.3&lt;/code&gt; reserved for future use, and &lt;code&gt;.255&lt;/code&gt; the broadcast address. That is also why the first usable address in an AWS subnet is &lt;code&gt;.4&lt;/code&gt;, and why a /28 — the smallest subnet AWS allows — gives you 11 usable hosts, not 16.&lt;/p&gt;

&lt;p&gt;The reason a planner earns its keep is permanence. A VPC's primary CIDR block never changes, and a subnet's CIDR is fixed at creation; you can bolt on secondary blocks later, but the original layout is the one you live with. Two minutes of carving before you create anything beats discovering at subnet nine that the block does not divide the way the whiteboard said it did.&lt;/p&gt;

&lt;p&gt;The design reasoning behind a layout — tiers, route tables, NAT economics — is in the &lt;a href="https://www.devopsenginer.com/blog/aws-vpc-guide-2026" rel="noopener noreferrer"&gt;AWS VPC guide&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  3. Kubernetes YAML Checker — valid YAML, broken production
&lt;/h2&gt;

&lt;p&gt;Most manifests that fail in production were syntactically valid when they shipped. The &lt;a href="https://www.devopsenginer.com/tools/kubernetes-yaml-checker" rel="noopener noreferrer"&gt;YAML Checker&lt;/a&gt; pastes or uploads a manifest and reports in two layers: YAML syntax errors &lt;em&gt;with line numbers&lt;/em&gt;, then, per document, pass/warn/fail on the production-readiness mistakes that pass &lt;code&gt;kubectl apply&lt;/code&gt; and page you later:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Image pinning&lt;/strong&gt; — &lt;code&gt;:latest&lt;/code&gt;, and untagged images, which Kubernetes treats as &lt;code&gt;:latest&lt;/code&gt; (an untagged reference also defaults &lt;code&gt;imagePullPolicy&lt;/code&gt; to &lt;code&gt;Always&lt;/code&gt;, so a restart can pull different code than the one you tested).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Missing CPU/memory requests&lt;/strong&gt; — the scheduler's contract; without them, placement and autoscaling math are fiction.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Missing limits&lt;/strong&gt; — the cgroup ceiling; without them, one container starves its neighbours.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Missing liveness and readiness probes.&lt;/strong&gt;&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;privileged: true&lt;/code&gt;&lt;/strong&gt;, &lt;strong&gt;&lt;code&gt;runAsUser: 0&lt;/code&gt;&lt;/strong&gt;, and &lt;code&gt;runAsNonRoot&lt;/code&gt; not set — worded honestly: "not set" does not prove the container runs as root, it means the manifest guarantees nothing and the image decides.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;hostNetwork&lt;/code&gt; / &lt;code&gt;hostPID&lt;/code&gt; / &lt;code&gt;hostIPC&lt;/code&gt;&lt;/strong&gt;, a missing namespace (how a test Deployment ends up in the production namespace), missing labels, and a missing container &lt;code&gt;securityContext&lt;/code&gt;. Pod specs inside Deployments, Jobs and CronJobs are all checked.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Two honest framings, printed on the tool page too. First: &lt;strong&gt;this is a common-mistakes checker, not a schema validator.&lt;/strong&gt; It does not run the Kubernetes OpenAPI schema, so it cannot catch every misspelled field — the authoritative check remains &lt;code&gt;kubectl apply --dry-run=server&lt;/code&gt;, which asks the API server itself. Second: &lt;strong&gt;your manifest never leaves your machine.&lt;/strong&gt; The checker is client-side JavaScript; nothing is uploaded, logged or stored, which is why it is safe to paste real manifests. Still redact live Secret values before pasting them into &lt;em&gt;any&lt;/em&gt; online tool — redacted examples check exactly the same way.&lt;/p&gt;

&lt;p&gt;For the configuration side of the same discipline, see the &lt;a href="https://www.devopsenginer.com/blog/kubernetes-configmaps-secrets-guide-2026" rel="noopener noreferrer"&gt;Kubernetes ConfigMaps and Secrets guide&lt;/a&gt; and the &lt;a href="https://www.devopsenginer.com/blog/kubernetes-resource-requests-limits-guide-2026" rel="noopener noreferrer"&gt;Requests and Limits guide&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why free, and why in the browser
&lt;/h2&gt;

&lt;p&gt;Free is the short answer; the longer one is that these tools are the front door, not the shop. This site earns from the work around them — the guides' audience, and products like the Backup Verification Kit — so the tools themselves carry no price, no account wall and no trial clock.&lt;/p&gt;

&lt;p&gt;Browser-only is a design decision, not a shortcut. A subnet plan and a production manifest are exactly the kind of input you should not have to send to a stranger's server to get arithmetic done. There is no server side to these tools: open the page, paste, close the tab, and nothing about your input exists anywhere. If a tool like this ever asks you to upload your manifests to &lt;em&gt;its&lt;/em&gt; cloud, that is the moment to close the tab.&lt;/p&gt;

&lt;h2&gt;
  
  
  The honest limits
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Cron Translator:&lt;/strong&gt; standard 5-field expressions and the common &lt;code&gt;@daily&lt;/code&gt;/&lt;code&gt;@weekly&lt;/code&gt;-style descriptors. Quartz extensions and 6/7-field expressions are flagged, not translated. Scheduled GitHub Actions runs can also be delayed under load — no translator can promise you the minute, only the meaning.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Subnet Planner:&lt;/strong&gt; IPv4 and AWS rules — VPC and subnet prefixes between /16 and /28. It carves one tier at a time; for a three-tier layout, split the VPC into per-tier blocks first (the &lt;a href="https://www.devopsenginer.com/blog/aws-vpc-guide-2026" rel="noopener noreferrer"&gt;VPC guide&lt;/a&gt; walks that layout), then plan each tier.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;YAML Checker:&lt;/strong&gt; a production-readiness lint, not a validator (above). It checks Pod, Deployment, Job and CronJob pod specs; it does not know your cluster's admission policies, and passing it does not mean a manifest is &lt;em&gt;right&lt;/em&gt; — only that the usual suspects are absent.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  FAQ
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Are the tools really free? Do I need an account?&lt;/strong&gt;&lt;br&gt;
Yes — free, no account, no sign-up. All three run in your browser; there is nothing to install and nothing to log in to.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Is my schedule, CIDR plan or YAML sent to a server?&lt;/strong&gt;&lt;br&gt;
No. All three tools are client-side. Your input is never uploaded, logged or stored — close the tab and it is gone.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What is the EventBridge cron expression for 9 AM on weekdays?&lt;/strong&gt;&lt;br&gt;
&lt;code&gt;cron(0 9 ? * 2-6 *)&lt;/code&gt; — six fields, &lt;code&gt;?&lt;/code&gt; in day-of-month, and weekday numbers shifted by one because EventBridge counts 1 as Sunday where standard cron counts 0 (or 7). It also runs in UTC, like GitHub Actions.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;How many usable IP addresses does an AWS /24 subnet have?&lt;/strong&gt;&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;A /24 holds 256 addresses and AWS reserves five in every subnet: network, VPC router, DNS server, one for future use, and broadcast.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;&lt;strong&gt;Does the YAML checker replace &lt;code&gt;kubectl apply --dry-run=server&lt;/code&gt;?&lt;/strong&gt;&lt;br&gt;
No. The checker catches common production-readiness mistakes — image pinning, requests, limits, probes, privilege. The server dry-run is the authoritative schema validation. Use both: the checker in the editor, the dry-run in CI.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Can I paste a real production manifest into the checker?&lt;/strong&gt;&lt;br&gt;
The manifest never leaves your machine, so the check itself is safe. Redact live Secret values first anyway — no online tool needs them, and redacted manifests check identically.&lt;/p&gt;

&lt;h2&gt;
  
  
  A closing question
&lt;/h2&gt;

&lt;p&gt;These three tools exist because the same frictions kept recurring. So: &lt;strong&gt;what small DevOps task do you still do by hand, badly, once a week?&lt;/strong&gt; A conversion you re-derive, a checklist you re-type, arithmetic you re-check three times. Tell me in the comments — that is precisely where tool number four will come from.&lt;/p&gt;

</description>
      <category>devops</category>
      <category>kubernetes</category>
      <category>aws</category>
      <category>productivity</category>
    </item>
    <item>
      <title>The Backup Verification Kit Is Live: Three Posts, a Public Argument and 21 Tests Later</title>
      <dc:creator>Rabin Adhikari</dc:creator>
      <pubDate>Thu, 05 Nov 2026 11:22:45 +0000</pubDate>
      <link>https://dev.to/softverseadmin/the-backup-verification-kit-is-live-three-posts-a-public-argument-and-21-tests-later-3g6p</link>
      <guid>https://dev.to/softverseadmin/the-backup-verification-kit-is-live-three-posts-a-public-argument-and-21-tests-later-3g6p</guid>
      <description>&lt;p&gt;This site's backup series did not start as a product. It started with somebody else's bad month. On dev.to, Renhua Wei — who writes as dafili — published what remains the most honest backup post I have read this year: &lt;a href="https://dev.to/dafili/my-backup-exited-0-for-three-weeks-and-nothing-paged-me-5ce6"&gt;"My Backup Exited 0 for Three Weeks and Nothing Paged Me"&lt;/a&gt;. His backup job reported success, night after night, while producing output he could not have restored from. Nothing alerted, because every check he had was asking the backup whether the backup was fine — and the backup, broken in a way that still exited 0, kept saying yes.&lt;/p&gt;

&lt;p&gt;I wrote &lt;a href="https://www.devopsenginer.com/blog/backup-verification-in-2026-exit-code-0-is-not-a-backup" rel="noopener noreferrer"&gt;the first response on this site&lt;/a&gt; — the four levels of backup confidence, and why "the job succeeded" is not one of them. Then dafili did the thing that makes public writing worth doing: he corrected me, in public, where I was wrong. My restic guidance had recommended a &lt;code&gt;--read-data-subset=5%&lt;/code&gt; sample; he pointed out that the subset is chosen &lt;em&gt;randomly&lt;/em&gt; each run, so some packs might never be read at all — the deterministic form is the &lt;code&gt;n/t&lt;/code&gt; rotation. He was right: I verified it against the restic man page, fixed the post the same day on every copy, and credited him.&lt;/p&gt;

&lt;p&gt;That exchange became two more posts. In &lt;a href="https://www.devopsenginer.com/blog/restore-drills-in-2026-you-cant-automate-the-declaration" rel="noopener noreferrer"&gt;Restore Drills&lt;/a&gt;, his doctrine got its name: &lt;strong&gt;you cannot automate the declaration, only the comparison&lt;/strong&gt; — a human writes down, by hand, what a correct restore must contain; automation compares against it nightly. In &lt;a href="https://www.devopsenginer.com/blog/declared-not-derived-backup-checks" rel="noopener noreferrer"&gt;Declared, Not Derived&lt;/a&gt;, the full placement was rig-tested before publication: a declared floor inside the backup run, a rolling median in monitoring, the fingerprint outside the backup tree, a restore-recency gauge. And the thread kept going after the posts shipped — his corrections shaped the design three times: the restic &lt;code&gt;n/t&lt;/code&gt; rotation, a checksum-recursion flaw, and the bootstrap rule that an undeclared state must SKIP instead of crashing or, worse, passing.&lt;/p&gt;

&lt;p&gt;A doctrine that survives an argument like that deserves to be more than three posts a reader has to reassemble by hand. So I packaged it. &lt;strong&gt;The Backup Verification Kit is live on Gumroad today&lt;/strong&gt; — the checks from the series as a dependency-free toolkit, plus a restore-drill runbook, plus a Mac app that shows the same checks as a dashboard.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;TL;DR:&lt;/strong&gt; The Backup Verification Kit — built from this site's backup series (#47–#49) and dafili's declared-not-derived doctrine — is live on Gumroad: five nightly checks (declared size floor, rolling-median drift, heartbeat freshness, hand-written fingerprint comparison, restore-drill gauge) in bash + Python 3, no agents, no SaaS. The release-gate suite passes 21/21, including an empty-state phase and five failure modes; it caught one real defect before shipping, since fixed and re-verified. Complete adds the restore-drill runbook and a Mac app (14/14 checks on our test Mac). Scripts $39, Complete $79, 30-day refund.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;&lt;em&gt;Originally published on &lt;a href="https://www.devopsenginer.com/blog/i-built-a-backup-verification-kit" rel="noopener noreferrer"&gt;DevOps Enginer&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What the kit does, one night at a time
&lt;/h2&gt;

&lt;p&gt;The kit installs beside the backups you already make. It creates nothing and restores nothing — it verifies, in exit codes your existing monitoring already understands.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Declared size floor&lt;/strong&gt; (&lt;code&gt;floor_check.sh&lt;/code&gt;). You declare, by hand, the size below which a dump is not the dump — "catastrophically wrong", not "a bit small". It runs inside the backup run and fails it on the spot. Five bad nights cannot move it: it is not computed from your backups; you wrote it down.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Rolling-median drift&lt;/strong&gt; (&lt;code&gt;median_check.py&lt;/code&gt;). Tonight's size against the median of the last 30 nights. Slow shrinkage — the partition that quietly stops being included — alerts as DRIFT below 80% of the median.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Heartbeat freshness&lt;/strong&gt; (&lt;code&gt;check_monitor_freshness.sh&lt;/code&gt;). The median check leaves a heartbeat, and the heartbeat itself is watched: a dead checker must never look like a healthy backup, so a stale or missing heartbeat alerts too.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Fingerprint comparison&lt;/strong&gt; (&lt;code&gt;fingerprint_compare.py&lt;/code&gt; + &lt;code&gt;declare_fingerprint.sh&lt;/code&gt;). The core of the doctrine. Your restore-to-scratch is compared against a fingerprint a human wrote by hand — expected files, one sha256, a row count, a config value, one assertion from the live system — stored outside the backup tree, protected by a checksum and an append-only declaration history. No fingerprint yet? The comparison prints &lt;strong&gt;SKIP and exits 3&lt;/strong&gt;. Day one does not get to be green for a reason unrelated to your data being intact.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Restore-drill gauge&lt;/strong&gt; (&lt;code&gt;restore_gauge.sh&lt;/code&gt;). A daily count of days since the last &lt;em&gt;successful manual&lt;/em&gt; restore drill. Past roughly 120 days without a human proving a restore, it pages: backup systems rarely die loudly, but restore &lt;em&gt;practices&lt;/em&gt; die quietly.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;A worked night, from the test rig: a dump came out at 118,500 bytes against a 150,000-byte rolling median — 79.0% — because a partition had stopped being included. The median check called DRIFT and exited 1 that night. The declared floor (60,000 bytes in that setup) passed the same dump, &lt;em&gt;by design&lt;/em&gt;: the floor is the catastrophe line, the median is the shrink detector, and neither grades its own homework. A restic helper (&lt;code&gt;restic_rotation.sh&lt;/code&gt;) implements dafili's &lt;code&gt;n/t&lt;/code&gt; correction: a deterministic &lt;code&gt;--read-data-subset=n/7&lt;/code&gt; weekday rotation, so every pack is read within a week.&lt;/p&gt;

&lt;h2&gt;
  
  
  What was tested — and what the testing caught
&lt;/h2&gt;

&lt;p&gt;This site's rule is test-first: nothing untested gets published as fact, and a product gets a harder gate than a post. Before listing, the kit went through a senior review (every script read line by line, the shipped zip extracted fresh, the README quickstart followed literally), then an evidence-grade release suite: 21 cases, fabricated sample data only, each asserting the documented behaviour — output wording &lt;em&gt;and&lt;/em&gt; exit code.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The suite passes 21/21.&lt;/strong&gt; The shape of it matters more than the number:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Phase 0 — the empty state (5 cases).&lt;/strong&gt; Point every check at a folder containing &lt;em&gt;nothing&lt;/em&gt;: no backup file, no history, no heartbeat, no fingerprint, no drill record. Absence must be reported honestly, in words — &lt;code&gt;FLOOR ERROR&lt;/code&gt;, &lt;code&gt;MEDIAN CHECK ERROR&lt;/code&gt;, &lt;code&gt;SKIP&lt;/code&gt; (exit 3), &lt;code&gt;MISSING METRIC&lt;/code&gt;, &lt;code&gt;GAUGE ALERT&lt;/code&gt; — never as a pass, never as a traceback. It guards the bootstrap rule dafili's comment produced: an undeclared state SKIPs — it never crashes, never passes.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Phase 1 — healthy data (6 cases).&lt;/strong&gt; A 150,300-byte sample dump against a 60,000-byte floor and a 150,000-byte median (100.2% — ok), heartbeat fresh, fingerprint declared and matched entry by entry, drill gauge at 41 days against the 120-day threshold. All exit 0.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Phase 2 — failure modes (5 cases, all caught).&lt;/strong&gt; A 0-byte dump: &lt;code&gt;FLOOR FAIL&lt;/code&gt;, exit 1. The 79.0% shrink: DRIFT, exit 1. A restore whose row count was cut from 412,308 to 350,000 after declaration: fingerprint &lt;code&gt;ALERT&lt;/code&gt;, exit 1. A heartbeat 259,200 seconds old against an 86,400 limit: &lt;code&gt;MISSING METRIC — the checker itself is down&lt;/code&gt;. A last drill 200 days ago: &lt;code&gt;GAUGE PAGE&lt;/code&gt;, exit 1.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Phase 3 — the demo regression (5 cases).&lt;/strong&gt; The bundled demo reproduces its documented numbers exactly: the declaration checksum &lt;code&gt;228fd3b095f2234bfb2679c1582bf2196faf51f8e822f277ddd44f461cc92cfb&lt;/code&gt; — content-fixed, so your machine prints the same one, which is how you check that what you downloaded is what we tested — and the median-poisoning demonstration, where five truncated nights collapse the last-5 median to 80,905 bytes, a healthy backup then alarms at &lt;strong&gt;3,403.4%&lt;/strong&gt; of the median, and a truncated backup &lt;em&gt;passes&lt;/em&gt; a fully poisoned 30-night median at &lt;strong&gt;99.1%&lt;/strong&gt; — while the declared floor's verdict never moves. That demo is the doctrine in one screen: derived checks poison; declared ones don't.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;And the suite earned its keep: &lt;strong&gt;it caught a defect the review had missed.&lt;/strong&gt; Fed an empty size argument — exactly what happens when the backup file is missing and &lt;code&gt;stat&lt;/code&gt; returns nothing — &lt;code&gt;floor_check.sh&lt;/code&gt; died on bash's raw &lt;code&gt;${2:?}&lt;/code&gt; usage error with exit 1, making its own documented branch (&lt;code&gt;FLOOR ERROR … the size pipeline is broken&lt;/code&gt;, exit 2) unreachable. Still fail-closed, but the wrong error class and words for the 3 a.m. reader. The fix was small (an explicit argument guard), changed nothing else, and the suite was re-run from scratch: 21/21, the fixed script verified byte-identical inside the rebuilt buyer zip. The earlier review had already found and fixed eight major defects of the same family — fail-open edges in a product whose reason to exist is failing loud, like a non-numeric dump size falling through to "floor ok", and a typo'd fingerprint entry kind silently ignored while the kit reported PASS. Both are ALERTs now. Two honesty footnotes: the scripts are syntax-checked (&lt;code&gt;bash -n&lt;/code&gt;, &lt;code&gt;py_compile&lt;/code&gt;) and rig-run, but shellcheck is not installed on our machine, so they are not shellchecked — the docs say so; and the doctrine's pipeline-masking demonstration was reproduced on our rig with a stand-in producer: a file holding 16.3% of the rows (481,800 of 2,950,561), &lt;code&gt;gzip -t&lt;/code&gt; still passing it.&lt;/p&gt;

&lt;p&gt;The &lt;strong&gt;Complete tier's Mac app&lt;/strong&gt; — the same five checks as a visual dashboard for backups on a Mac or its mounted volumes — passed its own 14-check run on our test machine: the Healthy dashboard (PROTECTED, drill ring at 41 days, next due 79), the Drift dataset flipping the header to ATTENTION NEEDED ("shrank to 79% of its normal size — 3rd night in a row"), the first-run empty state (no data folder: it builds its demo state and opens clean instead of crashing), and the trial-and-license gate — day one shows "Trial — 3 days left" with full features; with the trial start backdated past 72 hours it opens on the license screen, checks paused, data untouched. Fed an unseen dataset — a sample SQLite shop database grown nightly for 12 nights, plus website-files and documents backups — its detail screen showed the real numbers: tonight's dump at 79,567 actual bytes, orders ≥ 410, customers ≥ 101. The installer is DMG v4 (2,799,616 bytes, sha256 &lt;code&gt;b771e25e…&lt;/code&gt;, in the build notes), verified from the mounted volume.&lt;/p&gt;

&lt;h2&gt;
  
  
  The honest limits, printed on the listing too
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;It is not a backup tool.&lt;/strong&gt; It makes nothing and restores nothing; it verifies the backups and restore-to-scratch runs you already have.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The restic rotation helper is documented, not rig-tested.&lt;/strong&gt; A thin wrapper around documented restic commands; it was not part of the logged rig runs, and the listing says so too.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The PostgreSQL figures in posts #47–#49 are dafili's published measurements&lt;/strong&gt;, credited to him there and here. PostgreSQL was never installed on our rig; we reproduced the masking &lt;em&gt;mechanics&lt;/em&gt; with a stand-in producer.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The Mac app is unsigned v1.&lt;/strong&gt; No Apple Developer certificate behind it yet, so macOS blocks the first open: drag to Applications, open once, click Done, then System Settings → Privacy &amp;amp; Security → "Open Anyway". One time, documented on the listing and in the download — but it is friction, and you should know before you pay, not after.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;It was tested on our rig, not your systems&lt;/strong&gt; (Ubuntu 24.04, bash, Python 3.12; logged runs dated 3 Nov 2026; the app on macOS with sample data). The demo ships in the kit so you can watch it work before trusting it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;It is not a guarantee.&lt;/strong&gt; Verification shrinks the chance of discovering a bad backup mid-incident. The kit's point is the opposite of a guarantee: it makes you restore, on a schedule, with evidence.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The two versions
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Scripts — $39.&lt;/strong&gt; The full toolkit: the five checks, the restic rotation helper, &lt;code&gt;kit.conf&lt;/code&gt;, the demo rig, worked examples, cron lines and systemd units.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Complete — $79.&lt;/strong&gt; Everything above, plus the restore-drill runbook (the quarterly manual drill step by step: before/during/after, the re-declaration checklist, a triage guide for every alert the kit can raise) and the Mac app. The app runs full-featured for 3 days from first launch, no card needed; the license key from a Complete purchase activates it permanently — one online check at activation, then it works offline.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Both are on Gumroad, which handles payment and taxes: &lt;strong&gt;&lt;a href="https://devopsenginer.gumroad.com/l/backup-verification-kit" rel="noopener noreferrer"&gt;Backup Verification Kit on Gumroad&lt;/a&gt;&lt;/strong&gt;. Refunds are 30 days, no questions asked — if the kit doesn't fit your setup, I would rather learn that from a refund than from your silence.&lt;/p&gt;

&lt;h2&gt;
  
  
  FAQ
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;What do I need to run the scripts?&lt;/strong&gt;&lt;br&gt;
Linux, bash, Python 3.8+, coreutils. No pip installs, no agents, no accounts, nothing phones home. The optional Mac app needs macOS 14 or later.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Which backup tools does it work with?&lt;/strong&gt;&lt;br&gt;
Any tool that leaves you a dump or snapshot file and a way to restore-to-scratch. The size checks judge the artifact; the fingerprint judges the restored tree. Only the restic helper is tool-specific — and that one is documented rather than rig-tested.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What does exit code 3 mean?&lt;/strong&gt;&lt;br&gt;
SKIP: no fingerprint has been declared yet. An undeclared state must not come back green, so the check refuses to call day one a pass. Declaring takes about ten minutes by hand; from the first night you get real PASS/ALERT results.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Why is the fingerprint written by hand? Isn't that the opposite of automation?&lt;/strong&gt;&lt;br&gt;
Deliberately, yes. Expectations generated from the system under test grade that system's own homework — they agree with it exactly when it is wrong, which is how a backup exits 0 for three weeks. A human writes down what a correct restore must contain, quarterly, with dated reasons; automation compares nightly. The hand-written part is the product.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;How does the Mac app's trial and license work?&lt;/strong&gt;&lt;br&gt;
Full features for 3 days from first launch, no card. After that, checks pause — your data untouched — until you paste the license key from your Complete-tier purchase. Activation checks the key once against Gumroad; then the app works offline. And yes, the app is unsigned in v1: first open needs the "Open Anyway" step above.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What if it doesn't fit my setup?&lt;/strong&gt;&lt;br&gt;
Thirty-day refund, no questions asked. Email within 30 days of purchase for a full refund.&lt;/p&gt;

&lt;h2&gt;
  
  
  A closing question
&lt;/h2&gt;

&lt;p&gt;This kit exists because one engineer wrote honestly about his backup failing, and another got corrected in public and fixed it. The thread is still the best specification the kit has. So, in that spirit: &lt;strong&gt;when did you last actually restore — not back up, restore — and what did you compare the result against?&lt;/strong&gt; If the honest answer is "the job said success", you are exactly who the last three posts, and this kit, were written for. Tell me in the comments — especially if your answer is a check I haven't thought of. Those have a habit of becoming the next post.&lt;/p&gt;

</description>
      <category>devops</category>
      <category>backup</category>
      <category>sre</category>
      <category>testing</category>
    </item>
    <item>
      <title>Linux for DevOps in 2026: Commands, Permissions, Processes and systemd You Must Know</title>
      <dc:creator>Rabin Adhikari</dc:creator>
      <pubDate>Wed, 04 Nov 2026 03:21:04 +0000</pubDate>
      <link>https://dev.to/softverseadmin/linux-for-devops-in-2026-commands-permissions-processes-and-systemd-you-must-know-5b8b</link>
      <guid>https://dev.to/softverseadmin/linux-for-devops-in-2026-commands-permissions-processes-and-systemd-you-must-know-5b8b</guid>
      <description>&lt;h1&gt;
  
  
  Linux for DevOps in 2026: Commands, Permissions, Processes and systemd You Must Know
&lt;/h1&gt;

&lt;p&gt;Every other step on the DevOps path quietly assumes you can drive a Linux machine. Kubernetes nodes are Linux. CI runners are Linux. The cloud instance behind every tutorial — including the ones on this site — is Linux, and when something breaks at 2 a.m., nobody hands you a graphical interface: you get an SSH prompt and a blinking cursor. Linux is not one topic on the roadmap among many. It is the floor the rest of the roadmap stands on.&lt;/p&gt;

&lt;p&gt;This post is the Operating Systems node of our &lt;a href="https://www.devopsenginer.com/devops" rel="noopener noreferrer"&gt;DevOps Roadmap&lt;/a&gt; — if you are following the full path, start there and work through the nodes in order; this is the first deep-dive the prerequisites point at. It is organised as a learning path, not a reference dump: the topics are in the order to learn them, and every section ends with a short &lt;strong&gt;"you know enough when…"&lt;/strong&gt; checklist, so you can tell the difference between &lt;em&gt;having read about it&lt;/em&gt; and &lt;em&gt;being able to do it&lt;/em&gt;. When a section is genuinely enough for now, it says so — the goal is the Linux a working DevOps engineer uses daily, not a kernel-internals degree.&lt;/p&gt;

&lt;p&gt;A note on honesty, because this site's backup series earned its readers the hard way: every command below is standard practice, and the claims that could surprise you — exit codes, permission arithmetic, SSH refusing a badly-permissioned key — were &lt;strong&gt;run on an Ubuntu 24.04 machine before they were written down&lt;/strong&gt;.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;TL;DR:&lt;/strong&gt; Learn Linux in this order: (1) get a free practice machine — WSL, a free-tier VPS, or a container — and live in it; (2) files and permissions until &lt;code&gt;ls -l&lt;/code&gt; reads like a sentence; (3) processes and signals, because restarting things is the job; (4) shell mechanics — pipes, redirection, exit codes — because every pipeline and CI log speaks them; (5) the networking commands that answer "is it listening, can I reach it, does the name resolve"; (6) systemd and the journal, because services and logs are where servers explain themselves; (7) disk, SSH and package management as the daily-driver finish. Each step has a "you know enough when…" bar. Clear the bar, move on — and know that the next roadmap nodes (scripting, Git, containers, Kubernetes) all stand on this floor.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;&lt;em&gt;Originally published on &lt;a href="https://www.devopsenginer.com/blog/linux-for-devops-2026" rel="noopener noreferrer"&gt;DevOps Enginer&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Your free practice lab
&lt;/h2&gt;

&lt;p&gt;You cannot learn Linux by reading about Linux — including this post. You need a machine you are allowed to break. Three free options, in the order most learners should pick them:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;WSL 2 (Windows Subsystem for Linux)&lt;/strong&gt; — if your daily machine runs Windows, this puts a real Ubuntu one command away (&lt;code&gt;wsl --install&lt;/code&gt; from an admin terminal, then reboot). It runs a genuine Linux kernel in a lightweight virtual machine, which is why it is good enough for everything in this post except deep systemd work — and recent WSL versions can enable systemd too, via a &lt;code&gt;systemd=true&lt;/code&gt; line under &lt;code&gt;[boot]&lt;/code&gt; in &lt;code&gt;/etc/wsl.conf&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A free-tier cloud VPS&lt;/strong&gt; — every major cloud offers some free tier or trial credit, and a small always-free instance is the most &lt;em&gt;realistic&lt;/em&gt; lab of the three: a real server, reached over real SSH, with real systemd — the exact shape of the machines you will operate at work. It is also the option where the cleanup lesson applies: a forgotten test instance is how people meet their first surprise bill, so learn &lt;code&gt;stop&lt;/code&gt; and &lt;code&gt;delete&lt;/code&gt; the same day you learn &lt;code&gt;create&lt;/code&gt;. I learned that lesson on my own wallet while teaching myself AWS: the resources you forget do not forget you, and the end-of-month bill is a strict teacher. I have stopped and deleted on the same day ever since.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Containers on any machine you already have&lt;/strong&gt; — &lt;code&gt;docker run -it ubuntu:24.04 bash&lt;/code&gt; gives you a disposable Ubuntu shell in seconds. Perfect for practising files, permissions and shell mechanics; not a place to learn systemd, because a container normally has no systemd running as PID 1 — it has your shell, or the one process the container was started with.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Whichever you choose: use it daily, in short sessions, and do your real work in it where you can. Fluency comes from the hundredth &lt;code&gt;ls&lt;/code&gt;, not the first.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;You know enough when…&lt;/strong&gt; you can open a shell on your practice machine without thinking about how, and you have broken something in it at least once and fixed it or rebuilt it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Files, paths and the shape of the system
&lt;/h2&gt;

&lt;p&gt;Everything in Linux is a file — a slogan you will hear constantly, which becomes practical the moment you learn the handful of places files live:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;/etc&lt;/code&gt; — configuration. When a service behaves strangely, its config is almost always here (&lt;code&gt;/etc/nginx/&lt;/code&gt;, &lt;code&gt;/etc/ssh/&lt;/code&gt;).&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;/var&lt;/code&gt; — things that &lt;em&gt;vary&lt;/em&gt;: logs (&lt;code&gt;/var/log&lt;/code&gt;), caches, spool, and the data directories of many services.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;/home&lt;/code&gt; — user home directories; your work lives in &lt;code&gt;/home/&amp;lt;you&amp;gt;&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;/usr&lt;/code&gt; — the bulk of installed programs and libraries (&lt;code&gt;/usr/bin&lt;/code&gt; is where most commands live).&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;/proc&lt;/code&gt; and &lt;code&gt;/sys&lt;/code&gt; — virtual filesystems: not real files on disk, but windows into the running kernel. &lt;code&gt;/proc/loadavg&lt;/code&gt;, &lt;code&gt;/proc/meminfo&lt;/code&gt; and &lt;code&gt;/proc/&amp;lt;pid&amp;gt;/&lt;/code&gt; are genuinely useful, as the processes section shows.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;/tmp&lt;/code&gt; — scratch space, world-writable, cleaned on a schedule. Never store anything you would miss.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The working kit is small: &lt;code&gt;pwd&lt;/code&gt; (where am I), &lt;code&gt;ls -la&lt;/code&gt; (what is here, including dotfiles, with permissions), &lt;code&gt;cd&lt;/code&gt;, &lt;code&gt;file &amp;lt;name&amp;gt;&lt;/code&gt; (what &lt;em&gt;kind&lt;/em&gt; of file is this — run it before assuming), &lt;code&gt;less&lt;/code&gt; (read without editing), &lt;code&gt;head&lt;/code&gt;/&lt;code&gt;tail&lt;/code&gt; (edges of a file), &lt;code&gt;cp -a&lt;/code&gt;, &lt;code&gt;mv&lt;/code&gt;, &lt;code&gt;mkdir -p&lt;/code&gt;, &lt;code&gt;touch&lt;/code&gt;, and &lt;code&gt;rm&lt;/code&gt; — plus the respect &lt;code&gt;rm&lt;/code&gt; deserves: &lt;strong&gt;there is no trash can&lt;/strong&gt;, and &lt;code&gt;rm -rf&lt;/code&gt; does exactly what it says, instantly, as whatever user you are. Two habits separate careful engineers early: read the command once more before pressing Enter on anything destructive, and prefer &lt;code&gt;mv&lt;/code&gt;-ing a file to a backup name (&lt;code&gt;.bak&lt;/code&gt;) over deleting it until you are sure.&lt;/p&gt;

&lt;p&gt;Links deserve thirty seconds now because they confuse everyone once: a &lt;strong&gt;symbolic link&lt;/strong&gt; (&lt;code&gt;ln -s target name&lt;/code&gt;) is a tiny file that points at a path — delete the target and the link dangles. A &lt;strong&gt;hard link&lt;/strong&gt; (&lt;code&gt;ln target name&lt;/code&gt;) is a second name for the &lt;em&gt;same&lt;/em&gt; file content — the same inode, which you can verify with &lt;code&gt;ls -li&lt;/code&gt;: both names show the identical inode number, and the content survives until the &lt;em&gt;last&lt;/em&gt; name is removed.&lt;/p&gt;

&lt;p&gt;And finding things: &lt;code&gt;find /var -name "*.log" -type f&lt;/code&gt; walks a tree; add &lt;code&gt;-size +100M&lt;/code&gt; for big files, &lt;code&gt;-mtime -1&lt;/code&gt; for files changed in the last day. &lt;code&gt;find&lt;/code&gt; is slower and more honest than your memory of where you put things.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;You know enough when…&lt;/strong&gt; you can look at an unfamiliar server and, in under a minute, answer: where are this app's config, logs and data — using &lt;code&gt;ls&lt;/code&gt;, &lt;code&gt;file&lt;/code&gt; and &lt;code&gt;find&lt;/code&gt;, not guesses.&lt;/p&gt;

&lt;h2&gt;
  
  
  Permissions and ownership: the section that unlocks half of all errors
&lt;/h2&gt;

&lt;p&gt;A remarkable share of "Linux is broken" moments are permissions, wearing a disguise. Read &lt;code&gt;ls -l&lt;/code&gt; output until it parses at a glance:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;-rwxr-xr--  1  deploy  deploy  4096  Nov  3 10:12  deploy.sh
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Left to right: file type and permissions (&lt;code&gt;-&lt;/code&gt; regular file, &lt;code&gt;d&lt;/code&gt; directory, &lt;code&gt;l&lt;/code&gt; link), then three triplets — &lt;strong&gt;owner, group, everyone else&lt;/strong&gt; — each &lt;code&gt;r&lt;/code&gt;ead, &lt;code&gt;w&lt;/code&gt;rite, e&lt;code&gt;x&lt;/code&gt;ecute. Then link count, owner, group, size, date, name. The octal shorthand counts r=4, w=2, x=1 per triplet: &lt;code&gt;644&lt;/code&gt; is the everyday file (owner can write, everyone can read), &lt;code&gt;755&lt;/code&gt; the everyday directory or executable. On directories, &lt;code&gt;x&lt;/code&gt; means &lt;em&gt;enter/traverse&lt;/em&gt; — which is why a directory can be readable yet unusable without &lt;code&gt;x&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;chmod 640 app.log&lt;/code&gt;, &lt;code&gt;chmod u+x deploy.sh&lt;/code&gt;, &lt;code&gt;chown deploy:deploy app.log&lt;/code&gt; are the three verbs; &lt;code&gt;stat -c '%a %U %G' file&lt;/code&gt; prints the facts without the noise; &lt;code&gt;id&lt;/code&gt; shows who &lt;em&gt;you&lt;/em&gt; are and which groups you carry. New files get their permissions from the process's &lt;strong&gt;umask&lt;/strong&gt;: files start from 666, directories from 777, and the umask is subtracted. Verified on our rig: with &lt;code&gt;umask 022&lt;/code&gt;, a new file lands at &lt;code&gt;644&lt;/code&gt; and a new directory at &lt;code&gt;755&lt;/code&gt;. Run &lt;code&gt;umask&lt;/code&gt; on your own machine — the value differs between systems, and the arithmetic, not any specific number, is the thing to learn.&lt;/p&gt;

&lt;p&gt;Three special bits, one line each: &lt;strong&gt;setuid&lt;/strong&gt; (the &lt;code&gt;s&lt;/code&gt; in &lt;code&gt;-rwsr-xr-x&lt;/code&gt; on &lt;code&gt;/usr/bin/passwd&lt;/code&gt; and &lt;code&gt;/usr/bin/sudo&lt;/code&gt; — the program runs as its &lt;em&gt;owner&lt;/em&gt;, root, which is how a normal user may change a password at all); &lt;strong&gt;setgid&lt;/strong&gt; on a shared directory makes new files inherit the directory's group; the &lt;strong&gt;sticky bit&lt;/strong&gt; on &lt;code&gt;/tmp&lt;/code&gt; is why you cannot delete other people's files there despite it being world-writable. And &lt;code&gt;sudo&lt;/code&gt; itself: it runs one command as root, it is logged, and the professional habit is to use it per-command — not to live in a root shell where one typo has no ceiling.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;You know enough when…&lt;/strong&gt; you can predict, before running it, what &lt;code&gt;chmod 750&lt;/code&gt; does to a directory and who can then do what inside it — and when a service says "permission denied", your first move is &lt;code&gt;ls -l&lt;/code&gt; and &lt;code&gt;id&lt;/code&gt;, not a search engine.&lt;/p&gt;

&lt;h2&gt;
  
  
  Processes: what is actually running
&lt;/h2&gt;

&lt;p&gt;A process is a running program: it has an ID (&lt;strong&gt;PID&lt;/strong&gt;), a parent (&lt;strong&gt;PPID&lt;/strong&gt;), an owner, a state, and a lifetime. On Ubuntu, PID 1 is &lt;strong&gt;systemd&lt;/strong&gt; — every other process descends from it, which is why the services section later is really this section continued.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;ps aux | &lt;span class="nb"&gt;head&lt;/span&gt;                  &lt;span class="c"&gt;# every process, user-oriented view&lt;/span&gt;
ps &lt;span class="nt"&gt;-eo&lt;/span&gt; pid,ppid,stat,comm      &lt;span class="c"&gt;# the compact truth: pid, parent, STATE, name&lt;/span&gt;
top                            &lt;span class="c"&gt;# the live view (q to quit; htop is nicer but usually not installed)&lt;/span&gt;
pstree &lt;span class="nt"&gt;-p&lt;/span&gt;                      &lt;span class="c"&gt;# the family tree, with PIDs&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;STAT&lt;/code&gt; column is worth learning properly, because two of its letters diagnose real incidents: &lt;strong&gt;R&lt;/strong&gt; running, &lt;strong&gt;S&lt;/strong&gt; sleeping (most processes, most of the time — normal), &lt;strong&gt;T&lt;/strong&gt; stopped (someone sent SIGSTOP or pressed Ctrl+Z), &lt;strong&gt;Z&lt;/strong&gt; &lt;em&gt;zombie&lt;/em&gt; — the process has finished but its parent has not collected its exit status yet. A zombie consumes no CPU or memory; it is a bookkeeping entry, and the fix is at the parent, not the zombie. Both states are easy to meet on purpose in your lab: &lt;code&gt;kill -STOP &amp;lt;pid&amp;gt;&lt;/code&gt; freezes a process into &lt;strong&gt;T&lt;/strong&gt;, and a parent that forks a child and sleeps without waiting leaves a &lt;strong&gt;Z&lt;/strong&gt; behind — we produced both on the rig to check this paragraph.&lt;/p&gt;

&lt;p&gt;Processes are controlled by &lt;strong&gt;signals&lt;/strong&gt;. The working set:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Signal&lt;/th&gt;
&lt;th&gt;Number&lt;/th&gt;
&lt;th&gt;Default effect&lt;/th&gt;
&lt;th&gt;Sent by&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;SIGTERM&lt;/td&gt;
&lt;td&gt;15&lt;/td&gt;
&lt;td&gt;Exit, after cleanup&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;kill &amp;lt;pid&amp;gt;&lt;/code&gt; — the polite request&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;SIGKILL&lt;/td&gt;
&lt;td&gt;9&lt;/td&gt;
&lt;td&gt;Die immediately, no cleanup&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;kill -9 &amp;lt;pid&amp;gt;&lt;/code&gt; — the last resort&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;SIGINT&lt;/td&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;Interrupt&lt;/td&gt;
&lt;td&gt;Ctrl+C in the terminal&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;SIGHUP&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;Hang up / often "reload config"&lt;/td&gt;
&lt;td&gt;Terminal closing; &lt;code&gt;systemctl reload&lt;/code&gt; sends it to some daemons&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;SIGSTOP / SIGCONT&lt;/td&gt;
&lt;td&gt;19 / 18&lt;/td&gt;
&lt;td&gt;Freeze / resume&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;kill -STOP&lt;/code&gt;, Ctrl+Z sends SIGTSTP (20), a close cousin&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;kill&lt;/code&gt; sends SIGTERM unless you name another signal — start polite, because a process that gets SIGTERM can close files and release locks, and one that gets SIGKILL cannot. An exited process reports an &lt;strong&gt;exit code&lt;/strong&gt;: 0 is success, and death-by-signal shows up in shells as &lt;strong&gt;128 + signal number&lt;/strong&gt; — verified on the rig: killed with SIGKILL a process reports &lt;strong&gt;137&lt;/strong&gt;, SIGTERM &lt;strong&gt;143&lt;/strong&gt;, SIGINT &lt;strong&gt;130&lt;/strong&gt;. When a CI log or a supervisor says "exit code 137", it is telling you &lt;em&gt;something killed it&lt;/em&gt; (very often the out-of-memory killer) — that decoding is a debugging skill, not trivia.&lt;/p&gt;

&lt;p&gt;Day-to-day survival kit: run things in the foreground until they behave; &lt;code&gt;Ctrl+Z&lt;/code&gt;, &lt;code&gt;jobs&lt;/code&gt;, &lt;code&gt;bg&lt;/code&gt;/&lt;code&gt;fg&lt;/code&gt; to juggle; &lt;code&gt;nohup command &amp;amp;&lt;/code&gt; (or a tmux session) for work that must survive your SSH connection closing; &lt;code&gt;nice -n 10 command&lt;/code&gt; to start something politely deprioritised; &lt;code&gt;pkill -f pattern&lt;/code&gt; when you know the shape of the thing but not its PID.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;You know enough when…&lt;/strong&gt; given a runaway or wedged program, you can find it with &lt;code&gt;ps&lt;/code&gt;/&lt;code&gt;top&lt;/code&gt;, read its state, stop it politely, escalate only if it refuses — and decode whatever exit code it leaves behind.&lt;/p&gt;

&lt;h2&gt;
  
  
  The shell and Bash: pipes, redirection and exit codes
&lt;/h2&gt;

&lt;p&gt;The shell is not a passive prompt; it is a small programming language whose three core mechanics run everything else in DevOps — every CI pipeline, Dockerfile and remote command is these three ideas wearing a uniform.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Pipes&lt;/strong&gt; connect programs: &lt;code&gt;command1 | command2&lt;/code&gt; feeds one's stdout into the next's stdin. The classic log-triage shape is worth memorising as a pattern, not a command:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;cat &lt;/span&gt;access.log | &lt;span class="nb"&gt;cut&lt;/span&gt; &lt;span class="nt"&gt;-d&lt;/span&gt;&lt;span class="s1"&gt;' '&lt;/span&gt; &lt;span class="nt"&gt;-f1&lt;/span&gt; | &lt;span class="nb"&gt;sort&lt;/span&gt; | &lt;span class="nb"&gt;uniq&lt;/span&gt; &lt;span class="nt"&gt;-c&lt;/span&gt; | &lt;span class="nb"&gt;sort&lt;/span&gt; &lt;span class="nt"&gt;-nr&lt;/span&gt; | &lt;span class="nb"&gt;head&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;(Read: take the first field of every line, count occurrences, sort biggest first, show the top. Swap in your own fields and files — the &lt;em&gt;shape&lt;/em&gt; is the skill.)&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Redirection&lt;/strong&gt; moves streams to files: &lt;code&gt;&amp;gt;&lt;/code&gt; truncates and writes, &lt;code&gt;&amp;gt;&amp;gt;&lt;/code&gt; appends, &lt;code&gt;2&amp;gt;&lt;/code&gt; redirects stderr, and &lt;code&gt;2&amp;gt;&amp;amp;1&lt;/code&gt; means "send stderr wherever stdout is currently going". Order matters — this was verified on the rig, because it surprises everyone once: &lt;code&gt;cmd &amp;gt;file 2&amp;gt;&amp;amp;1&lt;/code&gt; puts &lt;em&gt;both&lt;/em&gt; streams in the file, while &lt;code&gt;cmd 2&amp;gt;&amp;amp;1 &amp;gt;file&lt;/code&gt; sends stderr to your &lt;em&gt;terminal&lt;/em&gt; and only stdout to the file. When a runbook says "capture all output", the first form is the one it means.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Exit codes&lt;/strong&gt; are how programs report truth. Every command ends with one (&lt;code&gt;echo $?&lt;/code&gt; shows the last one), 0 means success, and anything else is a failure &lt;em&gt;flavour&lt;/em&gt; — the signal deaths from the previous section being one family. Two consequences run your professional life. First, &lt;strong&gt;a pipeline's exit status is its last command's status&lt;/strong&gt;: on the rig, &lt;code&gt;false | true&lt;/code&gt; reports 0, because &lt;code&gt;true&lt;/code&gt; succeeded — which is how a failing step in the middle of a pipeline can pass silently. Bash's answer is &lt;code&gt;set -o pipefail&lt;/code&gt;, which makes the pipeline fail if &lt;em&gt;any&lt;/em&gt; stage fails (the same run reports 1 with pipefail on). Second, success is not always 0-or-nothing in the way scripts expect: &lt;code&gt;grep&lt;/code&gt; exits 1 when it finds &lt;em&gt;no match&lt;/em&gt; — a normal, honest answer that will murder a script running under &lt;code&gt;set -e&lt;/code&gt; if you did not know it was coming. Readers of this site's backup series have seen where ignoring these two facts leads; pipelines that report success over nothing are a genre.&lt;/p&gt;

&lt;p&gt;Your first real script needs little more than these bones:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;#!/usr/bin/env bash&lt;/span&gt;
&lt;span class="nb"&gt;set&lt;/span&gt; &lt;span class="nt"&gt;-euo&lt;/span&gt; pipefail

&lt;span class="nv"&gt;name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;1&lt;/span&gt;:?usage:&lt;span class="p"&gt; greet.sh &amp;lt;name&amp;gt;&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;
&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"hello, &lt;/span&gt;&lt;span class="nv"&gt;$name&lt;/span&gt;&lt;span class="s2"&gt; — running on &lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;hostname&lt;/span&gt;&lt;span class="si"&gt;)&lt;/span&gt;&lt;span class="s2"&gt;, as &lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;whoami&lt;/span&gt;&lt;span class="si"&gt;)&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The shebang picks the interpreter; &lt;code&gt;set -euo pipefail&lt;/code&gt; makes the script fail loudly (on errors, on unset variables, on any failed pipeline stage) instead of drifting on in a broken state; &lt;code&gt;${1:?...}&lt;/code&gt; refuses to run without its argument. Variables, &lt;code&gt;export&lt;/code&gt; (make it visible to child processes), &lt;code&gt;$PATH&lt;/code&gt; (the colon-separated list of directories searched for commands — &lt;code&gt;command -v kubectl&lt;/code&gt; tells you which one actually runs), and reading arguments are genuinely most of the Bash a DevOps engineer writes. Deeper scripting — the loops-over-servers, the cleanup jobs — is its own roadmap node (Python and Bash for DevOps, the next prerequisite on the roadmap), and &lt;a href="https://www.devopsenginer.com/blog/how-to-build-cicd-pipeline-github-actions-2025" rel="noopener noreferrer"&gt;Build a CI/CD Pipeline with GitHub Actions&lt;/a&gt; is where you will meet these mechanics again inside CI logs, which are shell transcripts whether they admit it or not.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;You know enough when…&lt;/strong&gt; you can write a five-line script with &lt;code&gt;set -euo pipefail&lt;/code&gt;, explain what each flag saves you from, and look at a failing pipeline and say &lt;em&gt;which stage&lt;/em&gt; failed and what its exit code means.&lt;/p&gt;

&lt;h2&gt;
  
  
  Networking commands: is it listening, can I reach it, does the name resolve
&lt;/h2&gt;

&lt;p&gt;Nearly every network problem you will ever debug reduces to three questions, and Linux has a direct instrument for each.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Is it listening?&lt;/strong&gt; &lt;code&gt;ss -tlnp&lt;/code&gt; lists TCP listeners (&lt;code&gt;-t&lt;/code&gt; TCP, &lt;code&gt;-l&lt;/code&gt; listening, &lt;code&gt;-n&lt;/code&gt; numeric, &lt;code&gt;-p&lt;/code&gt; the owning process — root or the owner sees names). Verified end-to-end on the rig: after starting a throwaway web server (&lt;code&gt;python3 -m http.server 8901&lt;/code&gt;), &lt;code&gt;ss -tlnp&lt;/code&gt; showed &lt;code&gt;python3&lt;/code&gt; listening on port 8901, &lt;code&gt;lsof -i :8901&lt;/code&gt; named the same PID, and &lt;code&gt;curl&lt;/code&gt; got its HTTP 200 reply. That trio — &lt;code&gt;ss&lt;/code&gt;, &lt;code&gt;lsof&lt;/code&gt;, &lt;code&gt;curl&lt;/code&gt; — settles most "the service is down" claims in a minute. (&lt;code&gt;netstat&lt;/code&gt; is &lt;code&gt;ss&lt;/code&gt;'s ancestor; learn &lt;code&gt;ss&lt;/code&gt;, and recognise &lt;code&gt;netstat&lt;/code&gt; when old runbooks use it.)&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Can I reach it?&lt;/strong&gt; &lt;code&gt;ping -c 3 host&lt;/code&gt; tests basic reachability (and plenty of networks block ping — silence is a clue, not a verdict); &lt;code&gt;curl -sI https://example.com&lt;/code&gt; fetches just the headers and is the single most useful reachability probe for anything HTTP; &lt;code&gt;curl -v&lt;/code&gt; when you need to watch the connection happen. Know your own address first: &lt;code&gt;ip -brief a&lt;/code&gt; for interfaces, &lt;code&gt;ip route&lt;/code&gt; for the default gateway — the first hop every packet takes.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Does the name resolve?&lt;/strong&gt; &lt;code&gt;getent hosts example.com&lt;/code&gt; asks the system's own resolver — the same path your applications take, through &lt;code&gt;/etc/hosts&lt;/code&gt; and NSS — and it exists on every Ubuntu because it ships with the base system. &lt;code&gt;dig +short example.com&lt;/code&gt; gives the DNS specialist's view; note that &lt;code&gt;dig&lt;/code&gt; belongs to an add-on package (verified on the rig: it is owned by &lt;code&gt;bind9-dnsutils&lt;/code&gt;), which is why it is sometimes missing on minimal servers while &lt;code&gt;getent&lt;/code&gt; never is. When the two disagree, you have found the classic split-brain: a stale &lt;code&gt;/etc/hosts&lt;/code&gt; entry shadowing real DNS.&lt;/p&gt;

&lt;p&gt;One layer up, this is exactly the mental model the cloud networking builds on: a listener that is not there, a route that does not reach, a name that resolves wrong — the &lt;a href="https://www.devopsenginer.com/blog/aws-vpc-guide-2026" rel="noopener noreferrer"&gt;AWS VPC guide&lt;/a&gt; is subnets and route tables expressing these same three questions at cloud scale.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;You know enough when…&lt;/strong&gt; told "the app is down", you check — in order — listener, reachability, name resolution, and can point at the layer that is actually failing before touching any config.&lt;/p&gt;

&lt;h2&gt;
  
  
  Services and systemd: starting things properly
&lt;/h2&gt;

&lt;p&gt;Hand-started processes die with your terminal, your session, or the reboot. Servers run &lt;strong&gt;services&lt;/strong&gt;, and on Ubuntu the manager is systemd: PID 1 from the processes section, wearing its other hat. A service is described by a &lt;strong&gt;unit file&lt;/strong&gt; — a small declarative text file saying what to run, as whom, and what to do when it dies. You read them with &lt;code&gt;systemctl cat &amp;lt;unit&amp;gt;&lt;/code&gt;; you will rarely write one from scratch as a beginner, but you must be fluent in the verbs:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;sudo &lt;/span&gt;systemctl start nginx        &lt;span class="c"&gt;# start it now&lt;/span&gt;
&lt;span class="nb"&gt;sudo &lt;/span&gt;systemctl stop nginx         &lt;span class="c"&gt;# stop it now&lt;/span&gt;
&lt;span class="nb"&gt;sudo &lt;/span&gt;systemctl restart nginx      &lt;span class="c"&gt;# stop + start&lt;/span&gt;
&lt;span class="nb"&gt;sudo &lt;/span&gt;systemctl &lt;span class="nb"&gt;enable &lt;/span&gt;nginx       &lt;span class="c"&gt;# also start at boot (enable --now does both)&lt;/span&gt;
systemctl status nginx            &lt;span class="c"&gt;# is it running, since when, last log lines&lt;/span&gt;
systemctl is-active nginx         &lt;span class="c"&gt;# the script-friendly question&lt;/span&gt;
systemctl list-units &lt;span class="nt"&gt;--failed&lt;/span&gt;     &lt;span class="c"&gt;# what is broken on this machine right now&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;status&lt;/code&gt; is the first command to run on any "the service won't…" report: it shows active state, the main PID, memory, and the most recent log lines in one screen. In scripts, prefer &lt;code&gt;systemctl is-active --quiet nginx&lt;/code&gt; and branch on the exit code — non-zero means &lt;em&gt;not active&lt;/em&gt; — rather than parsing status prose meant for humans. (Exact exit codes vary by situation — an unknown unit name earns you a 4 and a "not-found" from &lt;code&gt;is-enabled&lt;/code&gt;, both verified on the rig; the robust habit is testing zero vs non-zero and reading &lt;code&gt;status&lt;/code&gt; for the story.)&lt;/p&gt;

&lt;p&gt;Two systemd habits complete the picture. &lt;strong&gt;Timers&lt;/strong&gt; are systemd's cron: &lt;code&gt;systemctl list-timers&lt;/code&gt; shows what is scheduled on a modern Ubuntu — when it last ran, when it runs next — and units named &lt;code&gt;*.timer&lt;/code&gt; in that list are behind most housekeeping. And when a service misbehaves, its confession is in the journal, which is the next section.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;You know enough when…&lt;/strong&gt; you can take a service from stopped to running-at-boot, prove its state three ways (&lt;code&gt;status&lt;/code&gt;, &lt;code&gt;is-active&lt;/code&gt;, its port in &lt;code&gt;ss&lt;/code&gt;), and find its unit file and read what it actually executes.&lt;/p&gt;

&lt;h2&gt;
  
  
  Logs: where servers explain themselves
&lt;/h2&gt;

&lt;p&gt;When something breaks, the log is the difference between debugging and guessing. On current Ubuntu, the system's central log is the &lt;strong&gt;journal&lt;/strong&gt;, kept by systemd's journald and read with &lt;code&gt;journalctl&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;journalctl &lt;span class="nt"&gt;-u&lt;/span&gt; nginx               &lt;span class="c"&gt;# everything for one service&lt;/span&gt;
journalctl &lt;span class="nt"&gt;-u&lt;/span&gt; nginx &lt;span class="nt"&gt;-f&lt;/span&gt;            &lt;span class="c"&gt;# follow it live (the modern tail -f)&lt;/span&gt;
journalctl &lt;span class="nt"&gt;-u&lt;/span&gt; nginx &lt;span class="nt"&gt;--since&lt;/span&gt; &lt;span class="s2"&gt;"1 hour ago"&lt;/span&gt;
journalctl &lt;span class="nt"&gt;-p&lt;/span&gt; err                 &lt;span class="c"&gt;# errors and worse, across all units&lt;/span&gt;
journalctl &lt;span class="nt"&gt;--disk-usage&lt;/span&gt;           &lt;span class="c"&gt;# how much log you are keeping&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Not everything lives in the journal: applications also write plain files, overwhelmingly under &lt;code&gt;/var/log&lt;/code&gt; — our rig's own &lt;code&gt;/var/log&lt;/code&gt; holds things like &lt;code&gt;dpkg.log&lt;/code&gt; (package history) alongside the journal directory — and the universal tools apply: &lt;code&gt;tail -f file&lt;/code&gt; to watch, &lt;code&gt;tail -n 100&lt;/code&gt; for the recent past, &lt;code&gt;grep -i error file&lt;/code&gt; to filter, &lt;code&gt;less&lt;/code&gt; to search a big file without loading your editor with it. Rotation (old logs compressed and eventually deleted) is handled by &lt;code&gt;logrotate&lt;/code&gt; on a timer; the working knowledge is simply that logs are finite, dated, and findable — and that "the log from last Tuesday" may legitimately no longer exist.&lt;/p&gt;

&lt;p&gt;Discipline beats tooling here: read logs &lt;em&gt;before&lt;/em&gt; restarting things (a restart often clears the evidence and occasionally the problem's only copy), read from the first error rather than the last (cascades make the final lines the least informative), and correlate by timestamp — which is why servers keep honest clocks and why log lines without timestamps are a code smell. The Kubernetes version of this exact discipline — &lt;code&gt;kubectl logs&lt;/code&gt;, aggregated pipelines, what to keep — is in our &lt;a href="https://www.devopsenginer.com/blog/kubernetes-logging-guide-2026" rel="noopener noreferrer"&gt;Kubernetes Logging guide&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;You know enough when…&lt;/strong&gt; handed a broken service, your first three moves are &lt;code&gt;systemctl status&lt;/code&gt;, &lt;code&gt;journalctl -u&lt;/code&gt;, and &lt;code&gt;tail&lt;/code&gt; of the app's own log — in that order, before changing anything.&lt;/p&gt;

&lt;h2&gt;
  
  
  Disk and storage: the full-disk fire drill
&lt;/h2&gt;

&lt;p&gt;Disks fail DevOps engineers in one boring way far more than all exotic ways combined: they fill up. The drill is three commands deep:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;df&lt;/span&gt; &lt;span class="nt"&gt;-h&lt;/span&gt;                     &lt;span class="c"&gt;# every mounted filesystem: size, used, available, where mounted&lt;/span&gt;
&lt;span class="nb"&gt;du&lt;/span&gt; &lt;span class="nt"&gt;-sh&lt;/span&gt; /var/&lt;span class="k"&gt;*&lt;/span&gt; | &lt;span class="nb"&gt;sort&lt;/span&gt; &lt;span class="nt"&gt;-h&lt;/span&gt;   &lt;span class="c"&gt;# which subtree is eating it&lt;/span&gt;
find / &lt;span class="nt"&gt;-xdev&lt;/span&gt; &lt;span class="nt"&gt;-type&lt;/span&gt; f &lt;span class="nt"&gt;-size&lt;/span&gt; +500M 2&amp;gt;/dev/null   &lt;span class="c"&gt;# the individual big files&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;df&lt;/code&gt; answers at the filesystem level, &lt;code&gt;du&lt;/code&gt; at the directory level — when &lt;code&gt;df&lt;/code&gt; says full and &lt;code&gt;du&lt;/code&gt; cannot find the bytes, the classic culprit is a &lt;em&gt;deleted file still held open&lt;/em&gt; by a running process (the space returns when the process restarts), findable via &lt;code&gt;lsof +L1&lt;/code&gt;. Two supporting facts: &lt;code&gt;lsblk&lt;/code&gt; shows the block devices and their mount points (the map of what disk is even attached), and filesystems can run out of &lt;strong&gt;inodes&lt;/strong&gt; — file &lt;em&gt;slots&lt;/em&gt; — before bytes, which &lt;code&gt;df -i&lt;/code&gt; checks; millions of tiny files are how that happens. The usual suspects for growth are logs (previous section), container images (next paragraph's reading), and package caches — which is why the cleanup order is almost always: logs, images, caches, &lt;em&gt;then&lt;/em&gt; real data, carefully.&lt;/p&gt;

&lt;p&gt;Containers turn this from theory into next month's problem: images are layered filesystems stacked on exactly these mechanics, and the &lt;a href="https://www.devopsenginer.com/blog/how-to-optimize-docker-images-production-2025" rel="noopener noreferrer"&gt;Docker image optimization guide&lt;/a&gt; is this section's ideas applied to image layers — smaller layers, fewer surprises.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;You know enough when…&lt;/strong&gt; on a "disk full" alert you can name the guilty directory within two commands, say whether it is bytes or inodes, and clean the safe categories without touching data you cannot account for.&lt;/p&gt;

&lt;h2&gt;
  
  
  SSH: the front door, and how not to lock yourself out
&lt;/h2&gt;

&lt;p&gt;SSH is how you will reach every server in your career, so its basics are not optional polish — they are the job's front door. The model: your &lt;strong&gt;key pair&lt;/strong&gt; is a private key (a file you guard, conventionally &lt;code&gt;~/.ssh/id_ed25519&lt;/code&gt;) and a public key you copy onto servers into &lt;code&gt;~/.ssh/authorized_keys&lt;/code&gt;; the server challenges, your client proves it holds the private half, no password crosses the wire.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;ssh-keygen &lt;span class="nt"&gt;-t&lt;/span&gt; ed25519                 &lt;span class="c"&gt;# create the pair (with a passphrase)&lt;/span&gt;
ssh-copy-id user@server               &lt;span class="c"&gt;# install your public key on a server&lt;/span&gt;
ssh &lt;span class="nt"&gt;-i&lt;/span&gt; ~/.ssh/id_ed25519 user@server  &lt;span class="c"&gt;# connect with a specific key&lt;/span&gt;
scp file user@server:/path/           &lt;span class="c"&gt;# copy a file over SSH&lt;/span&gt;
rsync &lt;span class="nt"&gt;-avz&lt;/span&gt; &lt;span class="nb"&gt;dir&lt;/span&gt;/ user@server:/path/    &lt;span class="c"&gt;# sync a tree (resumable, incremental)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The rule that is not negotiable, verified on the rig so you can watch it bite: &lt;strong&gt;permissions&lt;/strong&gt;. Your &lt;code&gt;~/.ssh&lt;/code&gt; should be &lt;code&gt;700&lt;/code&gt;, private keys &lt;code&gt;600&lt;/code&gt;, &lt;code&gt;authorized_keys&lt;/code&gt; &lt;code&gt;600&lt;/code&gt; — and OpenSSH enforces it. We generated a key and deliberately set it world-readable (&lt;code&gt;644&lt;/code&gt;): &lt;code&gt;ssh-add&lt;/code&gt; refused it outright — &lt;em&gt;"WARNING: UNPROTECTED PRIVATE KEY FILE! … This private key will be ignored."&lt;/em&gt; — and accepted it the moment permissions were tightened to &lt;code&gt;600&lt;/code&gt;. Most "SSH suddenly ignores my key" mysteries are this, or an &lt;code&gt;authorized_keys&lt;/code&gt; pasted with wrong ownership. Debug with &lt;code&gt;ssh -v&lt;/code&gt; and read from the first line that surprises you.&lt;/p&gt;

&lt;p&gt;Ergonomics worth adopting on day one: a &lt;code&gt;~/.ssh/config&lt;/code&gt; with named hosts —&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ssh"&gt;&lt;code&gt;&lt;span class="k"&gt;Host&lt;/span&gt; staging
    &lt;span class="k"&gt;HostName&lt;/span&gt; &lt;span class="m"&gt;203&lt;/span&gt;.0.113.10
    &lt;span class="k"&gt;User&lt;/span&gt; deploy
    &lt;span class="k"&gt;IdentityFile&lt;/span&gt; ~/.ssh/id_ed25519
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;— after which &lt;code&gt;ssh staging&lt;/code&gt; is the whole command, and every tool that rides SSH (&lt;code&gt;scp&lt;/code&gt;, &lt;code&gt;rsync&lt;/code&gt;, &lt;code&gt;git&lt;/code&gt;) inherits the shortcut. On the server side, the beginner's checklist is short: key-based login working &lt;em&gt;before&lt;/em&gt; you consider disabling passwords, root login restricted, and a second session tested before you close the first — SSH misconfigurations lock out the careless, and the fix-it path is the lesson nobody wants first-hand.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;You know enough when…&lt;/strong&gt; you can set up key login on a fresh server from memory, explain why your private key is &lt;code&gt;600&lt;/code&gt;, and debug a refused connection with &lt;code&gt;ssh -v&lt;/code&gt; instead of reinstalling things at random.&lt;/p&gt;

&lt;h2&gt;
  
  
  Package management: installing software the Debian way
&lt;/h2&gt;

&lt;p&gt;Ubuntu is Debian-family: software arrives as &lt;strong&gt;packages&lt;/strong&gt;, installed from repositories by &lt;strong&gt;apt&lt;/strong&gt;, with &lt;strong&gt;dpkg&lt;/strong&gt; as the lower-level tool that manages the packages already present. The daily surface is five commands:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;sudo &lt;/span&gt;apt update                       &lt;span class="c"&gt;# refresh the catalogue (installs nothing)&lt;/span&gt;
&lt;span class="nb"&gt;sudo &lt;/span&gt;apt &lt;span class="nb"&gt;install&lt;/span&gt; &lt;span class="nt"&gt;-y&lt;/span&gt; htop              &lt;span class="c"&gt;# install (htop: the nicer top, absent by default — try it)&lt;/span&gt;
apt show nginx                        &lt;span class="c"&gt;# what is this package, which version, from where&lt;/span&gt;
dpkg &lt;span class="nt"&gt;-l&lt;/span&gt; | &lt;span class="nb"&gt;grep &lt;/span&gt;nginx                  &lt;span class="c"&gt;# is it installed here, exactly which build&lt;/span&gt;
dpkg &lt;span class="nt"&gt;-S&lt;/span&gt; /usr/bin/dig                  &lt;span class="c"&gt;# which package owns this file → bind9-dnsutils (rig-verified)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That last trick — &lt;code&gt;dpkg -S&lt;/code&gt; answering "who owns this file" — is the one that pays off monthly, turning an unknown binary into a documented package. The habits that matter: &lt;code&gt;apt update&lt;/code&gt; before installs (a stale catalogue produces baffling version errors), remove with &lt;code&gt;apt remove&lt;/code&gt; (config stays) vs &lt;code&gt;apt purge&lt;/code&gt; (config goes), and know that servers patch themselves via the &lt;code&gt;unattended-upgrades&lt;/code&gt; mechanism on most Ubuntu installs — check it exists before assuming a box is current. Software that does not come as a package — a tarball in &lt;code&gt;/usr/local&lt;/code&gt;, a vendor script — is exactly where the permissions and PATH knowledge from earlier sections stops being academic.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;You know enough when…&lt;/strong&gt; you can install, identify, locate and cleanly remove a package, and answer "what version is running here and where did it come from" for anything on the box.&lt;/p&gt;

&lt;h2&gt;
  
  
  What to learn next
&lt;/h2&gt;

&lt;p&gt;Linux is the first prerequisite, not the last. Standing on it, the roadmap's order is: &lt;strong&gt;scripting&lt;/strong&gt; (Python and Bash — the sibling prerequisite node on the roadmap), &lt;strong&gt;Git&lt;/strong&gt; (the Version Control node), then the container layer — Docker, where the &lt;a href="https://www.devopsenginer.com/blog/kubectl-commands-cheat-sheet-2026" rel="noopener noreferrer"&gt;kubectl Commands Cheat Sheet&lt;/a&gt; marks the start of the Kubernetes path once containers feel ordinary. Keep the full map open as you go: &lt;a href="https://www.devopsenginer.com/devops" rel="noopener noreferrer"&gt;the DevOps Roadmap&lt;/a&gt; shows which nodes this post just lit up, and which one is waiting next. If your interest is the Japan route specifically, the career side of standing out as an engineer here is covered in &lt;a href="https://www.devopsenginer.com/blog/how-to-become-devops-engineer-japan-2025-career-guide" rel="noopener noreferrer"&gt;How to Become a DevOps Engineer in Japan&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  FAQ
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Which distribution should I learn?&lt;/strong&gt;&lt;br&gt;
Ubuntu — specifically a current LTS — unless your workplace standardises on something else. The reasons are boring and decisive: it is what most tutorials assume, what most cloud images default to, and what the commands in this post were verified on. The transferable skills (systemd, permissions, the shell) are the same across the Debian and RHEL families; the differences are mostly package managers and some file locations. Learn one deeply, then translate.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Do I need to learn vi/vim before anything else?&lt;/strong&gt;&lt;br&gt;
No — but you need &lt;em&gt;an&lt;/em&gt; editor you can operate over SSH without thinking, because config files will not wait for you to get comfortable. &lt;code&gt;nano&lt;/code&gt; is enough to start; learn &lt;code&gt;vim&lt;/code&gt;'s survival subset (open, search, edit, save-and-quit, quit-without-saving) soon after, because &lt;code&gt;vim&lt;/code&gt; (or &lt;code&gt;vi&lt;/code&gt;) is the editor that is present on systems where your favourite is not — including, we noticed, this very test machine, which has neither &lt;code&gt;nano&lt;/code&gt; nor &lt;code&gt;vim&lt;/code&gt; installed and still got administered fine. The editor is a means; the config file is the point.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;How long does it take to "know Linux" for DevOps?&lt;/strong&gt;&lt;br&gt;
To the bar this post sets — the "you know enough when…" checklists — a focused few weeks of daily practice in a lab is a realistic frame for someone already technical, and the honest answer is that the learning never completes: it stratifies. The daily-driver layer in this post becomes reflex; deeper layers (performance tuning, kernel behaviour, complex storage) are learned incident by incident over years. Beware any course promising the whole thing by Friday.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Is the command line really necessary in the age of cloud consoles and AI assistants?&lt;/strong&gt;&lt;br&gt;
Yes, for the same reason pilots still learn manual flying: the console and the assistant are at their best exactly when nothing is wrong, and the failures that define this job happen underneath both. The shell is also where the &lt;em&gt;evidence&lt;/em&gt; lives — logs, exit codes, listener tables — and reading evidence directly is a different skill from being told a summary of it.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Should I dual-boot or replace my OS with Linux?&lt;/strong&gt;&lt;br&gt;
Neither, to start. WSL or a free-tier VPS (the lab section) gives you the full learning surface with an undo button. Run your daily machine on whatever it runs; keep your Linux in the lab until the lab starts feeling like the more natural home — that feeling, when it arrives, is the real graduation signal.&lt;/p&gt;

&lt;h2&gt;
  
  
  Summary
&lt;/h2&gt;

&lt;p&gt;Linux for DevOps is not a hundred commands; it is a small set of load-bearing ideas, each with a short working kit: files and where they live; permissions you can read at a glance and a umask you can compute; processes, their states, and signals sent politely first; shell mechanics — pipes, redirection, exit codes — with &lt;code&gt;pipefail&lt;/code&gt; turned on and grep's honest 1 understood; three networking questions with an instrument each; systemd's verbs and the journal's memory; disks measured before they are full; SSH keys guarded at &lt;code&gt;600&lt;/code&gt;; and packages whose provenance you can name. Every other node on the &lt;a href="https://www.devopsenginer.com/devops" rel="noopener noreferrer"&gt;DevOps Roadmap&lt;/a&gt; — the scripting and Git prerequisites next, then containers, Kubernetes, CI/CD and cloud — assumes this floor and builds on it. Build the floor first, verify each "you know enough when…" in your lab rather than in your head, and the rest of the path gets noticeably less mysterious.&lt;/p&gt;

&lt;p&gt;Every behaviour claim in this post that could be checked was checked first: tested before written, or phrased without the number.&lt;/p&gt;

</description>
      <category>linux</category>
      <category>devops</category>
      <category>tutorial</category>
      <category>systemd</category>
    </item>
    <item>
      <title>Declared, Not Derived: Backup Checks That Can't Grade Their Own Homework</title>
      <dc:creator>Rabin Adhikari</dc:creator>
      <pubDate>Tue, 03 Nov 2026 05:56:03 +0000</pubDate>
      <link>https://dev.to/softverseadmin/declared-not-derived-backup-checks-that-cant-grade-their-own-homework-3la8</link>
      <guid>https://dev.to/softverseadmin/declared-not-derived-backup-checks-that-cant-grade-their-own-homework-3la8</guid>
      <description>&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h1&gt;
  
  
  Declared, Not Derived: Backup Checks That Can't Grade Their Own Homework
&lt;/h1&gt;

&lt;p&gt;This series has an unusual origin story, and this post is where it pays off. &lt;a href="https://www.devopsenginer.com/blog/backup-verification-in-2026-exit-code-0-is-not-a-backup" rel="noopener noreferrer"&gt;Backup Verification in 2026&lt;/a&gt; grew out of a comment thread under &lt;strong&gt;dafili&lt;/strong&gt;'s dev.to article &lt;a href="https://dev.to/dafili/my-backup-exited-0-for-three-weeks-and-nothing-paged-me-5ce6"&gt;"My Backup Exited 0 for Three Weeks and Nothing Paged Me"&lt;/a&gt; — and then dafili corrected one of its claims in public, and the correction went in. &lt;a href="https://www.devopsenginer.com/blog/restore-drills-in-2026-you-cant-automate-the-declaration" rel="noopener noreferrer"&gt;Restore Drills in 2026&lt;/a&gt; took his fingerprint doctrine as far as it went, and ended with an honest open question: &lt;em&gt;where does the fingerprint live?&lt;/em&gt; I did not have his answer yet.&lt;/p&gt;

&lt;p&gt;Now I do. He answered that question, and the follow-ups behind it, across two threads — under the backup article and under his follow-up, &lt;a href="https://dev.to/dafili/three-checks-that-look-like-they-passed-but-validated-nothing-ejd"&gt;"Three checks that look like they passed but validated nothing"&lt;/a&gt;. This post is the answers, assembled into one approach. And because a reader of this series has already had to correct it once, every claim in this post that rests on a measurement was &lt;strong&gt;run on a test rig before it was written down&lt;/strong&gt;. Where a number is dafili's, it is credited to dafili. Where it is ours, the test that produced it is described with it.&lt;/p&gt;

&lt;p&gt;Underneath all three posts sits one distinction, and dafili kept arriving at it from different directions until it was impossible to unsee: &lt;strong&gt;declared vs derived&lt;/strong&gt;. A standard a human &lt;em&gt;declared&lt;/em&gt; — a floor, a fingerprint, a hash written down by hand — can disagree with the system forever. A standard &lt;em&gt;derived&lt;/em&gt; from the system — a median of the backup's own output, expectations read from the backup itself, a fingerprint stored where the pipeline can rewrite it — agrees with the system at exactly the moment the system is wrong. Every check that validates nothing is a derived standard wearing a declared one's clothes.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;TL;DR:&lt;/strong&gt; Place each backup check where its standard cannot be talked out of itself. Inside the backup script: nothing but a human-declared &lt;strong&gt;absolute floor per database&lt;/strong&gt;, set at "catastrophically wrong", failing fast. In monitoring: the &lt;strong&gt;rolling median&lt;/strong&gt;, whose job is the slow shrinkage the floor cannot see — and which, being in monitoring, fails as a &lt;em&gt;missing metric&lt;/em&gt; when it breaks, instead of failing silently as success. For the restore drill: the &lt;strong&gt;fingerprint&lt;/strong&gt; from the previous post, now with its storage rules answered — outside the tree it describes, where the backup job cannot write, human-written and script-read, guarded by a checksum, with an &lt;strong&gt;append-only history&lt;/strong&gt; that ends the checksum recursion, and one value asserted from the outside world. And the drill itself stays on a gauge: &lt;code&gt;days_since_last_successful_manual_restore&lt;/code&gt;, paging at about 120 days.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;&lt;em&gt;Originally published on &lt;a href="https://www.devopsenginer.com/blog/declared-not-derived-backup-checks" rel="noopener noreferrer"&gt;DevOps Enginer&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The questions this post answers
&lt;/h2&gt;

&lt;p&gt;Three questions were left open in the threads. They are the skeleton of everything below:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Where does the fingerprint live?&lt;/strong&gt; (Asked at the end of the restore-drills post.)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Does the size-vs-median guard belong in the backup script or in monitoring?&lt;/strong&gt; (Asked under "Three checks".)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The follow-ups those answers provoked:&lt;/strong&gt; is the floor set per database or per box — and if the fingerprint gets a checksum, where does the &lt;em&gt;checksum's&lt;/em&gt; protection live? Does the recursion ever end?&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  First, the trap: a pipeline that reports success over almost nothing
&lt;/h2&gt;

&lt;p&gt;Start with dafili's measured rig from "Three checks", because everything else in this post is a defence against its shape. He killed a &lt;code&gt;pg_dump&lt;/code&gt; mid-run on purpose and measured every check he could think of. Three of them passed:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The pipeline exit code was 0.&lt;/strong&gt; A shell pipeline reports the status of its &lt;em&gt;last&lt;/em&gt; command. &lt;code&gt;gzip&lt;/code&gt; was never killed — only its input died — so it finished cleanly, and the script rolled on to its next line, which is usually "backup succeeded", followed by a rotation step that deletes yesterday's copy.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;gzip -t&lt;/code&gt; passed.&lt;/strong&gt; The archive is genuinely valid gzip — correct trailer, proper EOF — wrapped around &lt;strong&gt;81,068 bytes of data where the complete dump was 2,753,525 bytes. 2.9%.&lt;/strong&gt; The compression is perfect; the content just isn't there.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;pg_restore -l&lt;/code&gt; passed.&lt;/strong&gt; For the custom format, the table of contents is written near the &lt;em&gt;start&lt;/em&gt; of the file, before the data blocks — so a dump truncated at 6% of its size still lists all seven TOC entries, exit 0.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;What caught it, in his measurements: making &lt;code&gt;pg_dump&lt;/code&gt; compress its own output (&lt;code&gt;pg_dump -Z6 -f&lt;/code&gt;) — killed the same way, it exits 137 and &lt;code&gt;gzip -t&lt;/code&gt; now &lt;em&gt;fails&lt;/em&gt;, because with no surviving second process, the gzip trailer is never written. And &lt;code&gt;pg_restore -f /dev/null&lt;/code&gt; — actually reading the data instead of inspecting the header — errors out. Those PostgreSQL numbers are dafili's, from his rig, and they are worth reading in his article directly.&lt;/p&gt;

&lt;p&gt;We could not install PostgreSQL on our test VM (the package install stalled and was abandoned), so we do &lt;strong&gt;not&lt;/strong&gt; present his pg numbers as reproduced. What we reproduced is the generic mechanics underneath them, with a plain producer process standing in for the dump:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;### CASE 1 — producer | gzip, producer killed mid-stream
producer exit status : 137  (137 = 128 + SIGKILL: it was killed)
pipeline exit status : 0  &amp;lt;- what a cron job / CI step reports: the LAST command's status
gzip -t exit status  : 0  (the partial file is 'valid gzip')

rows in killed (piped) file : 481800
rows in complete file       : 2950561
killed file holds 16.3% of the rows — and passed gzip -t

### CASE 2 — producer writes its own gzip file, killed mid-stream
producer exit status : 137  (137 = killed; nobody else's status can mask it)
gzip: fallback-direct.gz: unexpected end of file
gzip -t exit status  : 1  (non-zero: the trailer was never written)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Same shape, smaller stakes: kill the producer inside a pipeline and the pipeline reports 0 while &lt;code&gt;gzip -t&lt;/code&gt; blesses a file holding 16.3% of the rows. Let the producer compress its own output and the same kill produces an honest non-zero exit and a &lt;code&gt;gzip -t&lt;/code&gt; failure. The structural fix dafili measured on PostgreSQL — &lt;em&gt;stop using a pipeline; don't let a second process's success mask the first one's death&lt;/em&gt; — is not a PostgreSQL tip. It is the first declared-vs-derived lesson: the pipeline's exit code is derived from whichever process happened to survive.&lt;/p&gt;

&lt;h2&gt;
  
  
  The median is a derived standard
&lt;/h2&gt;

&lt;p&gt;Dafili's cheapest real guard, from the same article, is comparing the dump's size against a rolling median of recent runs — it costs nothing and catches the 2.9% shape without opening the file. My question in his thread was where it belongs: in the backup script, failing the run fast, or in monitoring, where the alert carries the history?&lt;/p&gt;

&lt;p&gt;His answer rearranged the whole design: &lt;strong&gt;monitoring — and never in the script — but not for the reason I was torn about.&lt;/strong&gt; The median is &lt;em&gt;self-referential&lt;/em&gt;. It is computed from the backup's own output, so it drifts toward whatever it is fed. His arithmetic: thirty healthy runs, and one truncated night fires at 2.9%. But feed it five consecutive truncated nights first, and the median itself becomes ~81 KB — now the &lt;em&gt;healthy&lt;/em&gt; run alarms, at 3,399% of the median. In his words: &lt;strong&gt;the thing being watched supplies the standard it is judged against.&lt;/strong&gt; It is the fingerprint-beside-the-config mistake in a second costume.&lt;/p&gt;

&lt;p&gt;We rebuilt that arithmetic as a simulation before writing this paragraph — thirty healthy nights generated around his published full-dump size, the truncated size his rig measured, a two-sided deviation check of the kind his example implies:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;STEP 1 — healthy history, one truncated night
  window median : 2,753,525 bytes (n=30)
  tonight       : 81,068 bytes = 2.9% of median -&amp;gt; median check: ALARM

STEP 2 — 5 consecutive truncated nights, short window (last 5 runs)
  window median : 80,905 bytes (n=5)
  tonight (healthy) : 2,753,525 bytes = 3,403.4% of median -&amp;gt; median check: ALARM

STEP 3 — majority-poisoned 30-night window (14 healthy + 16 truncated)
  window median : 81,797 bytes (n=30)
  tonight (truncated) : 81,068 bytes = 99.1% of median -&amp;gt; median check: pass
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Our run lands where his arithmetic said it would (3,403.4% against his published ~3,399% — the small difference is our generated jitter, and it is ours, not his). And the simulation shows the step his example only implied: once truncated runs are the &lt;em&gt;majority&lt;/em&gt; of the window, the poisoning completes — a truncated night scores 99.1% of the median and &lt;strong&gt;passes&lt;/strong&gt;. A longer window delays the collapse; it does not change its nature. A median is a derived standard, and given a persistent failure it derives the failure.&lt;/p&gt;

&lt;p&gt;None of this makes the median useless. It makes it &lt;em&gt;placed&lt;/em&gt;. Its job is real — it is the failure the next section's floor cannot see — but it has to live where a drifting standard does the least harm and where the check's own death is visible. That place is monitoring — and the sections below show it working there.&lt;/p&gt;

&lt;h2&gt;
  
  
  The floor is a declared standard
&lt;/h2&gt;

&lt;p&gt;What goes inside the backup script, then? Dafili's split is by &lt;em&gt;kind of failure&lt;/em&gt;, not by preference:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;In the script: an absolute floor, declared by a human.&lt;/strong&gt; Not a median — a constant. "This dump is never smaller than 400 KB." It cannot be poisoned by the script's own history because the script did not write it, it fails fast with a non-zero exit on the catastrophic case, and it costs one comparison.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Two follow-ups sharpened it. The floor is set &lt;strong&gt;per database&lt;/strong&gt; — a per-box constant has to be borrowed from one of the databases on the box, after which it is loose for the smallest of them or never fires for it at all. And it is set at &lt;strong&gt;"this would be catastrophically wrong"&lt;/strong&gt;, not "a bit below today" — because the second, in his phrase, is &lt;em&gt;a health check wearing a catastrophe's name&lt;/em&gt;. A floor is a tripwire for the 2.9% shape, nothing more. Asking it to detect ordinary unhealth is what makes teams tune it into silence.&lt;/p&gt;

&lt;p&gt;Here is the rig version, exactly as tested — a table of constants, nothing computed, nothing read from history:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;#!/usr/bin/env bash&lt;/span&gt;
&lt;span class="c"&gt;# floor_check.sh — runs INSIDE the backup script, immediately after the dump.&lt;/span&gt;
&lt;span class="c"&gt;# The only expectations here are constants a HUMAN declared: one absolute&lt;/span&gt;
&lt;span class="c"&gt;# floor per database, set at "this would be catastrophically wrong".&lt;/span&gt;
&lt;span class="c"&gt;# Usage: floor_check.sh &amp;lt;db&amp;gt; &amp;lt;dump-size-bytes&amp;gt;&lt;/span&gt;
&lt;span class="nb"&gt;set&lt;/span&gt; &lt;span class="nt"&gt;-u&lt;/span&gt;

&lt;span class="nb"&gt;declare&lt;/span&gt; &lt;span class="nt"&gt;-A&lt;/span&gt; &lt;span class="nv"&gt;FLOOR&lt;/span&gt;&lt;span class="o"&gt;=(&lt;/span&gt;
  &lt;span class="o"&gt;[&lt;/span&gt;billing-db]&lt;span class="o"&gt;=&lt;/span&gt;400000     &lt;span class="c"&gt;# healthy dump ~2.75 MB; below 400 KB is not the dump&lt;/span&gt;
  &lt;span class="o"&gt;[&lt;/span&gt;analytics-db]&lt;span class="o"&gt;=&lt;/span&gt;60000    &lt;span class="c"&gt;# healthy dump ~150 KB; per-database, because one&lt;/span&gt;
                          &lt;span class="c"&gt;# per-box constant is loose for the smallest database&lt;/span&gt;
&lt;span class="o"&gt;)&lt;/span&gt;

&lt;span class="nv"&gt;db&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;1&lt;/span&gt;:?usage:&lt;span class="p"&gt; floor_check.sh &amp;lt;db&amp;gt; &amp;lt;size-bytes&amp;gt;&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;
&lt;span class="nv"&gt;size&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;2&lt;/span&gt;:?usage:&lt;span class="p"&gt; floor_check.sh &amp;lt;db&amp;gt; &amp;lt;size-bytes&amp;gt;&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;
&lt;span class="nv"&gt;floor&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;FLOOR&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nv"&gt;$db&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="k"&gt;:-}&lt;/span&gt;

&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;[&lt;/span&gt; &lt;span class="nt"&gt;-z&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$floor&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;then
  &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"FLOOR ERROR: no floor declared for '&lt;/span&gt;&lt;span class="nv"&gt;$db&lt;/span&gt;&lt;span class="s2"&gt;' — declare one by hand"&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&amp;amp;2
  &lt;span class="nb"&gt;exit &lt;/span&gt;2
&lt;span class="k"&gt;fi

if&lt;/span&gt; &lt;span class="o"&gt;[&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$size&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="nt"&gt;-lt&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$floor&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;then
  &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"FLOOR FAIL: &lt;/span&gt;&lt;span class="nv"&gt;$db&lt;/span&gt;&lt;span class="s2"&gt; dump is &lt;/span&gt;&lt;span class="nv"&gt;$size&lt;/span&gt;&lt;span class="s2"&gt; bytes, below the declared floor of &lt;/span&gt;&lt;span class="nv"&gt;$floor&lt;/span&gt;&lt;span class="s2"&gt; bytes — failing the backup run"&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&amp;amp;2
  &lt;span class="nb"&gt;exit &lt;/span&gt;1
&lt;span class="k"&gt;fi

&lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"floor ok: &lt;/span&gt;&lt;span class="nv"&gt;$db&lt;/span&gt;&lt;span class="s2"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$size&lt;/span&gt;&lt;span class="s2"&gt; bytes &amp;gt;= declared floor &lt;/span&gt;&lt;span class="nv"&gt;$floor&lt;/span&gt;&lt;span class="s2"&gt; bytes"&lt;/span&gt;
&lt;span class="nb"&gt;exit &lt;/span&gt;0
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;And its tested behaviour, across the median-poisoning story from the previous section:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;healthy night                       2,753,525 bytes -&amp;gt; floor check: pass
truncated night 1                      81,068 bytes -&amp;gt; floor check: ALARM (fail fast)
truncated night 30                     81,068 bytes -&amp;gt; floor check: ALARM (fail fast)
healthy night after poisoning       2,753,525 bytes -&amp;gt; floor check: pass
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Night 1 and night 30 get the same verdict, because the standard did not move. That is the entire virtue of a declared constant: it is boring, it is dumb, and no run of bad nights can talk it out of itself.&lt;/p&gt;

&lt;h2&gt;
  
  
  Monitoring holds the median — and can notice its own death
&lt;/h2&gt;

&lt;p&gt;The median's job is the failure the floor is deliberately blind to. In dafili's examples: someone drops a table, or a partition stops being included, and the dump steps down — 2.75 MB to 2.2 MB — never coming near a catastrophe floor, visible only as drift against history. His summary of the split deserves quoting whole: &lt;em&gt;"They are not two implementations of one check. They catch different things, and one of them needs history the other must not have."&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The rig's monitoring check, as tested:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="c1"&gt;#!/usr/bin/env python3
&lt;/span&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;&lt;span class="s"&gt;median_check.py — runs in MONITORING, never inside the backup script.
Compares tonight&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;s dump size against the rolling median of recent
history and flags drift. Every run writes a heartbeat metric file;
if this check itself breaks, the heartbeat goes stale — and a stale
metric is itself alertable.&lt;/span&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;argparse&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;csv&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;statistics&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;main&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;
    &lt;span class="n"&gt;ap&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;argparse&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;ArgumentParser&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="n"&gt;ap&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;add_argument&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;--history&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;required&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;ap&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;add_argument&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;--db&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;required&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;ap&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;add_argument&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;--tonight&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;required&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;type&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nb"&gt;int&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;ap&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;add_argument&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;--window&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;type&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nb"&gt;int&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;default&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;ap&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;add_argument&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;--low&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;type&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nb"&gt;float&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;default&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;0.80&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;ap&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;add_argument&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;--heartbeat&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;default&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;monitor_heartbeat.txt&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;args&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;ap&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;parse_args&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

    &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="nf"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;history&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;newline&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;""&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;fh&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;rows&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;csv&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;DictReader&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;fh&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;db&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;db&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
        &lt;span class="n"&gt;sizes&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nf"&gt;int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;size&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;window&lt;/span&gt;&lt;span class="p"&gt;:]&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;sizes&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;ValueError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;no history rows for db &lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;db&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;'"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="nf"&gt;except &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;OSError&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;ValueError&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;KeyError&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;exc&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="c1"&gt;# The check is broken. Exit WITHOUT writing a heartbeat: the
&lt;/span&gt;        &lt;span class="c1"&gt;# missing metric is how monitoring notices its own death —
&lt;/span&gt;        &lt;span class="c1"&gt;# exactly what a backup script cannot report about itself.
&lt;/span&gt;        &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;MEDIAN CHECK ERROR: &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;exc&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; — no metric emitted&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;file&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;stderr&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;

    &lt;span class="n"&gt;med&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;statistics&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;median&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sizes&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;ratio&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;tonight&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;med&lt;/span&gt;
    &lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;code&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;DRIFT&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;ratio&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;low&lt;/span&gt; &lt;span class="nf"&gt;else &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;ok&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="nf"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;heartbeat&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;w&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;fh&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;   &lt;span class="c1"&gt;# emit the metric first
&lt;/span&gt;        &lt;span class="n"&gt;fh&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;write&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;median_check_last_run &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="nf"&gt;int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;time&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;fh&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;write&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;dump_size_ratio{{db=&lt;/span&gt;&lt;span class="se"&gt;\"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;db&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="se"&gt;\"&lt;/span&gt;&lt;span class="s"&gt;}} &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;ratio&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;db&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;: tonight &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;tonight&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; bytes vs median of last &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
          &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sizes&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; nights &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;med&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="p"&gt;,.&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; bytes = &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;ratio&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;% &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
          &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;(drift threshold &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;low&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;%) -&amp;gt; &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;code&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The tested split, on the rig's second database — a step-change shrinkage of the dropped-partition kind, 150,000 bytes down to 118,500:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ ./floor_check.sh analytics-db 118500
floor ok: analytics-db 118500 bytes &amp;gt;= declared floor 60000 bytes        (exit 0 — by design)

$ python3 median_check.py --history sizes_history.csv --db analytics-db --tonight 118500
analytics-db: tonight 118,500 bytes vs median of last 30 nights 150,000 bytes = 79.0% (drift threshold 80%) -&amp;gt; DRIFT   (exit 1)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The floor passes it, correctly — 118,500 bytes is not a catastrophe. The median flags it, correctly — a fifth of the database went missing overnight. Neither check is wrong; they are answering different questions, which is the point of placing them apart.&lt;/p&gt;

&lt;p&gt;One honest limit, because this series has learned to print those: a rolling median compares tonight against &lt;em&gt;recent&lt;/em&gt; history, so a decline spread evenly across the whole window is gradually absorbed as the window adapts. The median earns its place on step changes and drift &lt;em&gt;relative to the window&lt;/em&gt; — and the floor and the fingerprint, which never adapt, are the layers behind it.&lt;/p&gt;

&lt;p&gt;The placement has a second payoff, and it might be the deeper one. &lt;strong&gt;A broken check inside the backup script fails silently, by returning success&lt;/strong&gt; — nothing observes the checker, so its death looks like health. &lt;strong&gt;A broken check in monitoring fails as a missing metric&lt;/strong&gt; — and a missing metric is itself alertable, with &lt;code&gt;absent()&lt;/code&gt; in Prometheus or, at the rig's shell scale, a freshness check on the heartbeat file:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight console"&gt;&lt;code&gt;&lt;span class="gp"&gt;$&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;./check_monitor_freshness.sh monitor_heartbeat.txt 90000    &lt;span class="c"&gt;# heartbeat just written&lt;/span&gt;
&lt;span class="go"&gt;monitor fresh: median check heartbeat is 0s old (limit 90000s)                      (exit 0)

&lt;/span&gt;&lt;span class="gp"&gt;$&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;touch&lt;/span&gt; &lt;span class="nt"&gt;-d&lt;/span&gt; &lt;span class="s2"&gt;"2 days ago"&lt;/span&gt; monitor_heartbeat.txt                 &lt;span class="c"&gt;# the checker died two days ago&lt;/span&gt;
&lt;span class="go"&gt;MISSING METRIC: median check heartbeat is 172800s old (limit 90000s) — the checker itself is down   (exit 1)

&lt;/span&gt;&lt;span class="gp"&gt;$&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;python3 median_check.py &lt;span class="nt"&gt;--history&lt;/span&gt; MISSING.csv &lt;span class="nt"&gt;--db&lt;/span&gt; billing-db &lt;span class="nt"&gt;--tonight&lt;/span&gt; 2753525
&lt;span class="go"&gt;MEDIAN CHECK ERROR: [Errno 2] No such file or directory: 'MISSING.csv' — no metric emitted   (exit 2)
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;You can build a monitoring check that notices its own death. You cannot build that for a check inside the script it watches. In dafili's closing line on the placement question: &lt;em&gt;the script asserts what a human wrote down and cannot be talked out of; monitoring asserts against history and is expected to notice when it stops reporting. Anything that needs to be right for years goes in the second bucket.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Where the fingerprint lives
&lt;/h2&gt;

&lt;p&gt;Which brings back the restore-drills post's open question, now with its answer. My worry, asking it, was drift: keep the fingerprint next to the backup config and it might drift along with the thing it is meant to catch. Dafili's answer: not next to the config — he put it there first too — and the failure is worse than drift. It is &lt;strong&gt;quiet version skew&lt;/strong&gt;. If the fingerprint lives in the same file, the same repo, the same commit as the backup definition, the two are edited together, by the same person, in the same frame of mind — so they drift &lt;em&gt;together and consistently&lt;/em&gt;, and the nightly comparison keeps passing while both slowly stop describing what you actually care about. His sentence for it: &lt;strong&gt;"A fingerprint that changes when the thing it guards changes is not a fingerprint, it is a mirror."&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;His three rules, in the order they bite:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Keep it out of the tree it describes.&lt;/strong&gt; Not the backup config, not the repo, not the directory beside the exporter. The comparison must be a diff between two things that fail &lt;em&gt;independently&lt;/em&gt; — otherwise one bad &lt;code&gt;rsync --delete&lt;/code&gt; or one careless checkout takes out both the data and the record of what the data should have been. His placement: on the same box as the alerting, in a directory the backup job has no write access to. &lt;em&gt;If the backup can write its own expectations, it can satisfy them.&lt;/em&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The human writes it; the script only reads.&lt;/strong&gt; This has to be structural, not a convention: no code path may be able to regenerate the fingerprint. The moment a helper exists that recomputes it "to fix" a drift alarm, the alarm is gone — someone runs the helper, the night turns green, and nobody looks at why it drifted.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Every entry carries a reason, and the reason is dated.&lt;/strong&gt; When the file list changes because a table was legitimately added, the diff should be small and the note should say why. When it changes because something shrank, the note is the thing that makes you stop. A bare blob of values gives "we added a feature" and "we lost half the rows" the same visual weight.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;And two additions to the fingerprint from the restore-drills post. Include &lt;strong&gt;one value that is not derived from the backup at all&lt;/strong&gt; — asserted from the outside world. His example: a table count taken from the &lt;em&gt;live&lt;/em&gt; server, compared against what the dump reports. That catches the case where the dump is internally consistent, passes every checksum, and was taken from the wrong instance entirely — or from a replica that had quietly stopped applying changes. And keep &lt;strong&gt;a checksum of the fingerprint file itself&lt;/strong&gt;, written when the fingerprint is written: it does not stop anyone editing the file, but it converts a silent edit into a loud one.&lt;/p&gt;

&lt;p&gt;The checksum raises the obvious next question. It is the one I asked him, and it is the best answer in either thread.&lt;/p&gt;

&lt;h2&gt;
  
  
  What ends the checksum recursion: history
&lt;/h2&gt;

&lt;p&gt;If the fingerprint needs a checksum, where does the checksum live — and what protects &lt;em&gt;it&lt;/em&gt;? Isn't it the same problem one level up, forever?&lt;/p&gt;

&lt;p&gt;His answer, in full: yes, it is the same problem one level up, and &lt;strong&gt;it doesn't terminate — it just stops being worth solving.&lt;/strong&gt; The checksum only buys you one thing: &lt;em&gt;an edit by anything other than the writer is loud.&lt;/em&gt; What ends the recursion is not another checksum. It is &lt;strong&gt;history: fingerprints append-only, alert on the diff.&lt;/strong&gt; Keep every declaration as an appended, dated record, and the script can rewrite its own file if it wants — &lt;em&gt;you still see it move.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;We built exactly that into the test kit, and then we tried to sneak past it. First the kit itself:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;vault/fingerprint.txt&lt;/code&gt; — the declared entries (file list, one sha256, one row-count minimum, one config value, one outside-world assertion), each with its date and reason, in a directory outside both the backup's tree and the restore's tree.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;vault/fingerprint.sha256&lt;/code&gt; — the checksum, refreshed only by the declaration tool.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;vault/history.log&lt;/code&gt; — append-only. Every declaration appends one line: timestamp, the fingerprint's new checksum, the reason.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;declare_fingerprint.sh&lt;/code&gt; — the human's tool, and the only writer. It does not edit the fingerprint's contents; it cannot regenerate a single declared value. It refreshes the checksum and appends the history line.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The compare script checks the fingerprint's integrity &lt;em&gt;before&lt;/em&gt; it trusts any comparison: checksum must match, &lt;strong&gt;and the current checksum must be the last entry in the history.&lt;/strong&gt; Then the demos, all run before this post was written:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Scenario&lt;/th&gt;
&lt;th&gt;Result&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Clean state&lt;/td&gt;
&lt;td&gt;PASS — every comparison green&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;A non-writer quietly edits the fingerprint (row-count minimum 400,000 → 400)&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;ALERT&lt;/strong&gt; — checksum mismatch; comparison not trusted&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;The restore drifts (orders 412,308 → 312,308, below the declared minimum)&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;ALERT&lt;/strong&gt; — row-count drift&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;The dump came from the wrong instance (24 tables vs the live server's 26)&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;ALERT&lt;/strong&gt; — outside-world assertion&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Fingerprint edited &lt;em&gt;and its checksum regenerated to cover it&lt;/em&gt;
&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;ALERT&lt;/strong&gt; — checksum passes; history catches it: no declaration was appended&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;A legitimate re-declaration (schema really rolled; human updates the entry by hand and declares it, with a dated reason)&lt;/td&gt;
&lt;td&gt;PASS — with the change visible as a dated history line&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The fifth row is dafili's recursion answer, demonstrated: the checksum alone does &lt;em&gt;not&lt;/em&gt; end the recursion — an edit that covers its own checksum sails through the checksum check. The append-only history ends it, because the edit has no declaration. And the sixth row is the path that makes the system livable: legitimate change is not blocked, it is &lt;em&gt;declared&lt;/em&gt; — appended, dated, reasoned, visible in the diff forever.&lt;/p&gt;

&lt;p&gt;Honesty requires the footnote, and dafili's framing already contains it: an operator with write access to the vault can append a &lt;em&gt;fake&lt;/em&gt; declaration too. Append-only history does not make edits impossible. It makes them &lt;strong&gt;loud and attributable&lt;/strong&gt; — a silent edit becomes a visible, dated, reasoned act that a second person can review. That is the most you can ask of any of this, and it is enough, because the failures this series is about were never attackers. They were quiet edits at 2 a.m. that nobody ever saw.&lt;/p&gt;

&lt;p&gt;One more thing the kit did during testing, unscripted: it caught &lt;em&gt;us&lt;/em&gt;. Our first re-declaration demo updated the schema value in the fingerprint and declared it — and the compare refused to pass, because the fingerprint's sha256 entry still pointed at the old config file. We had updated the declaration's words and forgotten its hash, and the kit treated our half-finished declaration exactly like drift. It was right to. A declaration is a human act with several parts; the tooling cannot do any of them for you, and the comparison is what makes an incomplete one visible instead of green. We completed the declaration by hand, re-declared, and it passed — with both the mistake and the completion sitting in the history, dated, for anyone to read.&lt;/p&gt;

&lt;p&gt;For reference, the compare script's integrity step (lightly cleaned from the tested rig version; the full rig, with every demo output, is reproducible and was run before publication):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="c1"&gt;#!/usr/bin/env python3
&lt;/span&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;&lt;span class="s"&gt;fingerprint_compare.py — the comparison half of the kit.
Automation runs this; it only READS the fingerprint. Step 0 checks the
fingerprint itself before any comparison is trusted: the checksum must
match, and the current checksum must be the LAST line of the append-only
history. Step 1 then compares the restore against each declared entry:
expected files, the declared sha256, the row-count minimum, the config
value, and the outside-world assertion (the dump&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;s table count must
equal the live server&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;s — a value not derived from the backup at all).
Exit 0 = pass; 1 = any alert.&lt;/span&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;

&lt;span class="n"&gt;actual_sum&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;sha256_of&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;vault&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;fingerprint.txt&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;recorded&lt;/span&gt;   &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;vault&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;fingerprint.sha256&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;read_text&lt;/span&gt;&lt;span class="p"&gt;().&lt;/span&gt;&lt;span class="nf"&gt;split&lt;/span&gt;&lt;span class="p"&gt;()[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="nf"&gt;report&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;actual_sum&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="n"&gt;recorded&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;fingerprint checksum&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;history&lt;/span&gt;  &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;ln&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;ln&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;vault&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;history.log&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;read_text&lt;/span&gt;&lt;span class="p"&gt;().&lt;/span&gt;&lt;span class="nf"&gt;splitlines&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;ln&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;()]&lt;/span&gt;
&lt;span class="n"&gt;declared&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;sha256:&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;actual_sum&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;history&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="nf"&gt;report&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;declared&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;append-only history&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;   &lt;span class="c1"&gt;# an edit that regenerated its
&lt;/span&gt;                                          &lt;span class="c1"&gt;# own checksum fails HERE
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  The gauge watches the drill
&lt;/h2&gt;

&lt;p&gt;One piece of the restore-drills post survives unchanged, and dafili's storage answer promoted it: asked where the fingerprint lives, he closed by saying the &lt;strong&gt;gauge is the part he would do first&lt;/strong&gt;. &lt;code&gt;days_since_last_successful_manual_restore&lt;/code&gt; — one exporter line, written only when a human-run restore &lt;em&gt;passes&lt;/em&gt; — turns an invisible process into one that can go stale in public. His reasoning closes the loop with the very first article in this series: the fix is not making the human more reliable; it is making &lt;strong&gt;the absence of the human's action visible&lt;/strong&gt; — the same move as monitoring for the absence of a fresh backup artifact instead of the presence of an error.&lt;/p&gt;

&lt;p&gt;The rig's shell-scale version, with its three tested states:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;#!/usr/bin/env bash&lt;/span&gt;
&lt;span class="c"&gt;# restore_gauge.sh — the state file is written ONLY by the final step of&lt;/span&gt;
&lt;span class="c"&gt;# a human-run restore drill that passed its fingerprint comparison.&lt;/span&gt;
&lt;span class="nv"&gt;last&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;cat&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$state&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="si"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="nv"&gt;now&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;date&lt;/span&gt; +%s&lt;span class="si"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="nv"&gt;days&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$((&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;now &lt;span class="o"&gt;-&lt;/span&gt; last&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="m"&gt;86400&lt;/span&gt; &lt;span class="k"&gt;))&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$days&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="nt"&gt;-gt&lt;/span&gt; 120 &lt;span class="o"&gt;]&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt; &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"GAUGE PAGE: days_since_last_successful_manual_restore = &lt;/span&gt;&lt;span class="nv"&gt;$days&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&amp;amp;2&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="nb"&gt;exit &lt;/span&gt;1&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"gauge ok: days_since_last_successful_manual_restore = &lt;/span&gt;&lt;span class="nv"&gt;$days&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;drill passed 2026-10-20  -&amp;gt;  gauge ok: days_since_last_successful_manual_restore = 14    (exit 0)
last drill 2026-06-01    -&amp;gt;  GAUGE PAGE: days_since_last_successful_manual_restore = 155  (exit 1)
no state file at all     -&amp;gt;  GAUGE ALERT: no manual restore has ever been recorded        (exit 2)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  The whole approach on one page
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Layer&lt;/th&gt;
&lt;th&gt;Lives in&lt;/th&gt;
&lt;th&gt;Its standard&lt;/th&gt;
&lt;th&gt;Catches&lt;/th&gt;
&lt;th&gt;Fails as&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Absolute floor, per database&lt;/td&gt;
&lt;td&gt;The backup script&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;Declared&lt;/strong&gt; — a human-set constant&lt;/td&gt;
&lt;td&gt;Catastrophic truncation (the 2.9% shape), tonight, before rotation&lt;/td&gt;
&lt;td&gt;Loudly, in the run — and it cannot be poisoned, because it reads no history&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Rolling median&lt;/td&gt;
&lt;td&gt;Monitoring&lt;/td&gt;
&lt;td&gt;Derived — recent history&lt;/td&gt;
&lt;td&gt;Step-change shrinkage the floor cannot see (dropped table, excluded partition)&lt;/td&gt;
&lt;td&gt;A &lt;strong&gt;missing metric&lt;/strong&gt; — alertable via &lt;code&gt;absent()&lt;/code&gt; / heartbeat freshness&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Restore fingerprint&lt;/td&gt;
&lt;td&gt;A vault outside the tree it describes&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;Declared&lt;/strong&gt; — hand-written, dated reasons, checksum, append-only history&lt;/td&gt;
&lt;td&gt;Wrong-but-intact restores; wrong instance; quiet edits to the expectations themselves&lt;/td&gt;
&lt;td&gt;A drift alert, or an integrity alert before any comparison is trusted&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Restore-recency gauge&lt;/td&gt;
&lt;td&gt;Monitoring&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;Declared&lt;/strong&gt; — the 120-day threshold&lt;/td&gt;
&lt;td&gt;The drill itself quietly stopping&lt;/td&gt;
&lt;td&gt;A page on absence, not on error&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Read the "standard" column top to bottom and the whole doctrine is there: wherever a standard &lt;em&gt;must not move&lt;/em&gt;, it is declared; the one derived standard in the stack is placed where its movement is watched and its silence is an alarm.&lt;/p&gt;

&lt;h2&gt;
  
  
  Declared-vs-derived checklist
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;No backup pipeline reports success from a surviving second process — the producer's own exit status (or its internal compression) is what the run reports&lt;/li&gt;
&lt;li&gt;Every backup script contains exactly one size expectation: a human-declared absolute floor, &lt;strong&gt;per database&lt;/strong&gt;, set at "catastrophically wrong"&lt;/li&gt;
&lt;li&gt;No median, average, or history-derived threshold exists anywhere inside a backup script&lt;/li&gt;
&lt;li&gt;The rolling-median drift check lives in monitoring, emits a metric every run, and has an alert on the metric's &lt;em&gt;absence&lt;/em&gt;
&lt;/li&gt;
&lt;li&gt;The restore fingerprint lives outside the tree it describes, in a directory the backup job cannot write&lt;/li&gt;
&lt;li&gt;The fingerprint is written only by a human (via a declaration tool that cannot regenerate values); the comparison only reads it&lt;/li&gt;
&lt;li&gt;Every fingerprint entry carries a dated reason; one entry is asserted from the outside world, not derived from the backup&lt;/li&gt;
&lt;li&gt;The fingerprint has a checksum, and an append-only history whose last entry must match the current checksum before any comparison is trusted&lt;/li&gt;
&lt;li&gt;Re-declarations are appended, dated, and reasoned — never silent edits&lt;/li&gt;
&lt;li&gt;A &lt;code&gt;days_since_last_successful_manual_restore&lt;/code&gt; gauge (or equivalent) pages at about 120 days&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  FAQ
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Isn't a rolling median the standard way to sanity-check backup sizes?&lt;/strong&gt;&lt;br&gt;
Yes — and nothing here retires it. The argument is about &lt;em&gt;placement&lt;/em&gt;. In monitoring, the median does the job it is good at (drift against history) and its failure mode is a missing metric. Inside the backup script, the same median is a derived standard guarding its own source, and its failure mode is returning success. Same check, different blast radius.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Why per database rather than one floor per box?&lt;/strong&gt;&lt;br&gt;
Because a per-box constant has to be borrowed from one of the databases on the box — dafili's answer, and it is arithmetic, not taste. Set it for the largest database and it is loose for the smallest; set it for the smallest and it never fires for the largest. Catastrophe is per-database, so the floor is too.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Who checks the checksum of the history? And the history of the history?&lt;/strong&gt;&lt;br&gt;
Nobody — that is the honest answer, and it is dafili's. The recursion does not terminate; it stops being worth solving. The checksum makes non-writer edits loud; the append-only history makes &lt;em&gt;all&lt;/em&gt; edits visible as movement. Past that point you are buying attribution, not impossibility, and attribution is what the real failures — quiet 2 a.m. edits, not attackers — actually require.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Your median and poisoning numbers come from generated data. Why should I trust them?&lt;/strong&gt;&lt;br&gt;
Trust them for exactly what they are: logic tests. The healthy and truncated sizes are the magnitudes from dafili's published rig (2,753,525 and 81,068 bytes), and the PostgreSQL measurements — the 2.9% file, the TOC listing on a truncated dump, the exit-137 with &lt;code&gt;-Z6&lt;/code&gt; — are his, credited, and linked. What our rig establishes is data-independent: a median of a poisoned window collapses at majority-bad regardless of whose bytes they are; a constant does not move regardless of the night; a checksum-plus-history kit behaves as designed across six scenarios. Where a claim needed PostgreSQL and we could not run PostgreSQL, the post says so rather than borrowing his numbers silently.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Does this replace the ladder and the drill from the earlier posts?&lt;/strong&gt;&lt;br&gt;
No — it guards them. The ladder (exists, intact, readable, restored) still defines the checking; the fingerprint still defines what "correct" means for the drill. This post is about the layer underneath both: where the standards those checks compare against come from, and who — or what — is allowed to move them.&lt;/p&gt;

&lt;h2&gt;
  
  
  Summary
&lt;/h2&gt;

&lt;p&gt;Three posts ago, this series started with a backup that exited 0 for three weeks and wrote nothing. The ladder taught it to check. The fingerprint taught the checks what "correct" means. Dafili's answers in the threads since then taught the whole stack where its standards are allowed to live: &lt;strong&gt;declared where they must not move&lt;/strong&gt; (the floor in the script, the fingerprint in its vault, the gauge's threshold), &lt;strong&gt;derived only where derivation is watched&lt;/strong&gt; (the median in monitoring, failing as a missing metric when it dies). A check that grades its own homework will eventually pass a failing student and call it consistency. Declare the standard by hand, keep it where the system can't rewrite it, append the changes where everyone can see them — and let the system argue with the declaration, out loud, every night.&lt;/p&gt;

&lt;p&gt;Credit where it is due, and it is considerable: the placement doctrine, the per-database floor, the mirror line, the version-skew analysis, and the append-only answer to the checksum recursion all come from &lt;strong&gt;dafili&lt;/strong&gt;, across the comment threads under &lt;a href="https://dev.to/dafili/my-backup-exited-0-for-three-weeks-and-nothing-paged-me-5ce6"&gt;"My Backup Exited 0 for Three Weeks and Nothing Paged Me"&lt;/a&gt; and &lt;a href="https://dev.to/dafili/three-checks-that-look-like-they-passed-but-validated-nothing-ejd"&gt;"Three checks that look like they passed but validated nothing"&lt;/a&gt; — the second of which also holds his original PostgreSQL measurements. The restore-drills post ended by promising that its open question would get an update when he answered; this post is that update, and the answer was better than the question. The rig described here was built to his specifications and tested before a word was published — which is, he would presumably note, the least a series about verification can do. The threads remain open. They have improved this series three times now.&lt;/p&gt;

</description>
      <category>devops</category>
      <category>backup</category>
      <category>monitoring</category>
      <category>tutorial</category>
    </item>
    <item>
      <title>Restore Drills in 2026: You Can't Automate the Declaration, Only the Comparison</title>
      <dc:creator>Rabin Adhikari</dc:creator>
      <pubDate>Mon, 02 Nov 2026 06:10:10 +0000</pubDate>
      <link>https://dev.to/softverseadmin/restore-drills-in-2026-you-cant-automate-the-declaration-only-the-comparison-1nio</link>
      <guid>https://dev.to/softverseadmin/restore-drills-in-2026-you-cant-automate-the-declaration-only-the-comparison-1nio</guid>
      <description>&lt;h1&gt;
  
  
  Restore Drills in 2026: You Can't Automate the Declaration, Only the Comparison
&lt;/h1&gt;

&lt;p&gt;The previous post in this little series — &lt;a href="https://www.devopsenginer.com/blog/backup-verification-in-2026-exit-code-0-is-not-a-backup" rel="noopener noreferrer"&gt;Backup Verification in 2026: Exit Code 0 Is Not a Backup&lt;/a&gt; — grew out of a comment thread under a dev.to article by &lt;strong&gt;dafili&lt;/strong&gt;, &lt;a href="https://dev.to/dafili/my-backup-exited-0-for-three-weeks-and-nothing-paged-me-5ce6"&gt;"My Backup Exited 0 for Three Weeks and Nothing Paged Me"&lt;/a&gt;. The conversation did not stop at publication. Dafili corrected one claim in that post (the correction is in, and credited), and then pushed the whole idea one level deeper — to a place the four-level ladder does not reach on its own.&lt;/p&gt;

&lt;p&gt;His point, in his words: &lt;strong&gt;"You cannot automate the declaration, only the comparison."&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;This post is about what that sentence means, because once you hear it, a hole opens up in the middle of most restore drills — including the tidy ones from the ladder post. The ladder tells you &lt;em&gt;how&lt;/em&gt; to check a backup. It never tells you what "correct" means. And if the definition of correct comes from the same system you are checking, your drill can pass beautifully while verifying nothing at all.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;TL;DR:&lt;/strong&gt; A restore drill is only as honest as the expectations it compares against. If those expectations are derived from the backup or the system being verified — row counts read from the dump itself, file lists generated by the backup tool — the check is circular: it catches corruption, never wrongness. The fix is a &lt;strong&gt;fingerprint&lt;/strong&gt;: a short, hand-written declaration of what a correct restore must contain (an expected file list, one file's sha256, a row count, one config value). A human re-declares it quarterly; automation compares against it nightly and shouts on drift. And the drill itself gets monitored with a &lt;code&gt;days_since_last_successful_manual_restore&lt;/code&gt; gauge that pages at about 120 days — because an unmonitored drill quietly stops happening.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;&lt;em&gt;Originally published on &lt;a href="https://www.devopsenginer.com/blog/restore-drills-in-2026-you-cant-automate-the-declaration" rel="noopener noreferrer"&gt;DevOps Enginer&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The ladder checks the climb, not the destination
&lt;/h2&gt;

&lt;p&gt;The four levels from the previous post — the artifact exists, it is intact, it can be read, it has actually been restored — still stand. Level 4 is where trust lives: restore into a scratch environment and verify the result like a user would.&lt;/p&gt;

&lt;p&gt;But look closely at how a typical level-4 drill "verifies." It restores the dump, then runs queries against the restored database: count the rows in &lt;code&gt;users&lt;/code&gt;, check that &lt;code&gt;orders&lt;/code&gt; has recent entries, confirm the application boots. Where do the expected values for those checks come from? Usually from the restored data itself, or from the same monitoring that watched the original system. The drill asks the backup: &lt;em&gt;do you agree with yourself?&lt;/em&gt; The backup always says yes.&lt;/p&gt;

&lt;p&gt;That is how a drill passes on data that is perfectly intact and completely wrong. Dafili made this distinction in the same thread: an integrity check catches media or repository corruption, but only a restore catches data that is wrong — and even then, only if the restore is compared against something that did not come from the restore. An empty mount backed up faithfully, checksummed faithfully, restored faithfully. A dump of the wrong database, in perfect condition. Every level green. Nothing recovered that you actually wanted.&lt;/p&gt;

&lt;p&gt;He explored the same trap from another angle in a follow-up article, &lt;a href="https://dev.to/dafili/three-checks-that-look-like-they-passed-but-validated-nothing-ejd"&gt;"Three checks that look like they passed but validated nothing"&lt;/a&gt;: checks that execute correctly, return success, and prove nothing, because they validate a different property than the one you care about. A restore drill with self-derived expectations is exactly that shape of check — bigger, more expensive, and just as capable of validating nothing.&lt;/p&gt;

&lt;h2&gt;
  
  
  The fingerprint: four things a human writes down
&lt;/h2&gt;

&lt;p&gt;Dafili's answer is a &lt;strong&gt;fingerprint&lt;/strong&gt; — a small file a human writes by hand, declaring what a correct restore must contain. Not generated. Not exported from the database. Declared, the way you would declare what matters before a trip: passport, keys, the one document you cannot replace. His fingerprint has four components:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;An expected file list.&lt;/strong&gt; The files (or tables, or object prefixes) that must be present in the restore. Not all of them — the ones whose absence would mean the backup missed its point.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;One file's sha256.&lt;/strong&gt; Pick one file whose content you can predict — a config file, a seed data file, a known document — and record its hash. The restore must reproduce it bit for bit.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A row count.&lt;/strong&gt; One table that matters, with a count (or a sane range) a human states from knowing the business, not from querying the backup. Orders should be in the hundreds of thousands by now; if the restore says 41, something is deeply wrong even if the restore "worked."&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;One config value.&lt;/strong&gt; A single setting whose correctness you can declare: the application's environment name, a feature flag that must be on, the schema version the code expects. Config drift is how "successful" restores fail at boot time.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Four declarations. Small enough to write in fifteen minutes, small enough to read at a glance, and — this is the whole design — &lt;strong&gt;none of them can be produced by the system they verify&lt;/strong&gt;. The fingerprint stands outside the backup pipeline and judges it.&lt;/p&gt;

&lt;p&gt;An example, as a plain file kept with the runbook:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="c1"&gt;# restore-fingerprint.yaml — declared by hand, re-declared quarterly
&lt;/span&gt;&lt;span class="na"&gt;system&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;billing-db&lt;/span&gt;
&lt;span class="na"&gt;expected_files&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="s"&gt;/srv/app/config/app.env&lt;/span&gt;
  &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="s"&gt;/srv/app/data/seed-countries.csv&lt;/span&gt;
&lt;span class="na"&gt;sha256&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;/srv/app/config/app.env&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"9f2c…e41a"&lt;/span&gt;   &lt;span class="c1"&gt;# recorded at declaration time
&lt;/span&gt;&lt;span class="na"&gt;row_counts&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;orders&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&amp;gt;= 400000"&lt;/span&gt;                      &lt;span class="c1"&gt;# stated from the business, not the dump
&lt;/span&gt;&lt;span class="na"&gt;config_values&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;schema_version&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"2026.09"&lt;/span&gt;                &lt;span class="c1"&gt;# what the current code expects
&lt;/span&gt;&lt;span class="na"&gt;declared&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;2026-11-02&lt;/span&gt;
&lt;span class="na"&gt;next_review&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;2027-02-02&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The exact format does not matter. What matters is the direction of the arrow: expectations flow from a human's knowledge &lt;em&gt;into&lt;/em&gt; the check, never from the checked system &lt;em&gt;outward&lt;/em&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  You cannot automate the declaration
&lt;/h2&gt;

&lt;p&gt;Why insist a human writes it? Because the declaration is not data entry — it is a review disguised as data entry. To write the fingerprint, someone has to sit down and answer questions that no monitoring dashboard asks: &lt;em&gt;What in this system would we actually miss? Which table is the business? Which config value, if wrong, turns a good restore into a bad morning?&lt;/em&gt; Those answers live in people's heads and in the incident history, not in the database.&lt;/p&gt;

&lt;p&gt;In dafili's practice, the human &lt;strong&gt;re-declares the fingerprint quarterly&lt;/strong&gt; — deliberately on the same cadence as the full restore drill. Systems drift legitimately over a quarter: new tables appear, row counts grow, config values change on purpose. The quarterly re-declaration is the moment a human looks at that drift on purpose, in calm conditions, and says: &lt;em&gt;yes, this is what correct looks like now.&lt;/em&gt; Skip it, and the fingerprint slowly becomes a declaration about a system that no longer exists — which is its own quiet failure mode, and exactly why the declaration cannot be a one-time setup task.&lt;/p&gt;

&lt;p&gt;This is also why automating the declaration defeats the purpose. A generator that rebuilds the fingerprint from production every night produces a fingerprint that is always exactly right and therefore says nothing: it re-declares whatever the system currently is, including its mistakes. The value of the fingerprint is precisely that it can &lt;em&gt;disagree&lt;/em&gt; with the system — and that a human chose the terms of the disagreement in advance.&lt;/p&gt;

&lt;h2&gt;
  
  
  The comparison is the automation's job
&lt;/h2&gt;

&lt;p&gt;Everything after the declaration should be boring, mechanical, and nightly. Restore the latest backup into a scratch environment, extract the actuals, and diff them against the fingerprint:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;#!/usr/bin/env bash&lt;/span&gt;
&lt;span class="c"&gt;# compare-restore.sh — runs after the nightly restore-to-scratch&lt;/span&gt;
&lt;span class="nb"&gt;set&lt;/span&gt; -euo pipefail

&lt;span class="nv"&gt;FAIL&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;0

&lt;span class="c"&gt;# 1. Expected files present?&lt;/span&gt;
&lt;span class="k"&gt;while&lt;/span&gt; &lt;span class="nb"&gt;read&lt;/span&gt; -r f&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt;
  &lt;span class="o"&gt;[&lt;/span&gt; -f &lt;span class="s2"&gt;"/tmp/restore-test$f"&lt;/span&gt; &lt;span class="o"&gt;]&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt; &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"MISSING: $f"&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="nv"&gt;FAIL&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;1&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="k"&gt;done&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;yq -r &lt;span class="s1"&gt;'.expected_files[]'&lt;/span&gt; restore-fingerprint.yaml&lt;span class="o"&gt;)&lt;/span&gt;

&lt;span class="c"&gt;# 2. The declared hash matches?&lt;/span&gt;
sha256sum -c &lt;span class="o"&gt;&amp;lt;&amp;lt;&amp;lt;&lt;/span&gt;&lt;span class="s2"&gt;"$(yq -r '.sha256."&lt;/span&gt;/srv/app/config/app.env&lt;span class="s2"&gt;"' restore-fingerprint.yaml)  /tmp/restore-test/srv/app/config/app.env"&lt;/span&gt; \
  &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt; &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"HASH DRIFT: app.env"&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="nv"&gt;FAIL&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;1&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="o"&gt;}&lt;/span&gt;

&lt;span class="c"&gt;# 3. The declared row count is in range?&lt;/span&gt;
&lt;span class="nv"&gt;ORDERS&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$(&lt;/span&gt;psql restore_test -tAc &lt;span class="s2"&gt;"SELECT count(*) FROM orders;"&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt; &lt;span class="s2"&gt;"$ORDERS"&lt;/span&gt; -ge 400000 &lt;span class="o"&gt;]&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt; &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"ROW COUNT DRIFT: orders=$ORDERS"&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="nv"&gt;FAIL&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;1&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="o"&gt;}&lt;/span&gt;

&lt;span class="c"&gt;# 4. The declared config value survived?&lt;/span&gt;
psql restore_test -tAc &lt;span class="s2"&gt;"SELECT value FROM settings WHERE key='schema_version';"&lt;/span&gt; \
  &lt;span class="o"&gt;|&lt;/span&gt; grep -qx &lt;span class="s2"&gt;"2026.09"&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt; &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"CONFIG DRIFT: schema_version"&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="nv"&gt;FAIL&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;1&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="o"&gt;}&lt;/span&gt;

&lt;span class="nb"&gt;exit&lt;/span&gt; &lt;span class="nv"&gt;$FAIL&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The details will differ with your stack — the pattern is the point. Automation compares; it does not get an opinion. When the comparison fails, it &lt;strong&gt;shouts&lt;/strong&gt;: a failed fingerprint comparison is a page, not a log line, because it means one of two things, both important — the backup is wrong, or the world changed and nobody re-declared it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Drift is a conversation, not just an alarm
&lt;/h2&gt;

&lt;p&gt;That second case deserves respect rather than dread. Most fingerprint failures, in a healthy team, will be legitimate change: a migration added a table, the row count outgrew the declared range, a config value rolled forward. The correct response is not to silence the check but to &lt;strong&gt;re-declare deliberately&lt;/strong&gt; — a human reviews the drift, confirms it is intended, and updates the fingerprint as its own small, visible act.&lt;/p&gt;

&lt;p&gt;Contrast that with the alternative everyone actually lives with: expectations embedded in the check script, edited quietly at 2 a.m. to make the red go away, until the check asserts nothing. The fingerprint does not prevent someone from weakening a check. It makes the weakening a declared event, in a file with a date on it, that a second person can review. That is the difference between a system that learns and a system that merely quiets down.&lt;/p&gt;

&lt;h2&gt;
  
  
  Monitor the drill itself
&lt;/h2&gt;

&lt;p&gt;There is one more layer, and it might be the most practical idea in the whole thread. Automated nightly comparisons are excellent — right up until the human parts of the practice quietly die. The quarterly manual restore is the first thing skipped in a busy quarter. Then it is skipped twice. Nothing alarms, because the automation is still green; the drill's &lt;em&gt;judgment&lt;/em&gt; — a person restoring by hand, noticing the weird thing no fingerprint anticipated — has been absent for a year and nobody knows.&lt;/p&gt;

&lt;p&gt;Dafili's countermeasure is to monitor the drill itself with a gauge: &lt;strong&gt;&lt;code&gt;days_since_last_successful_manual_restore&lt;/code&gt;&lt;/strong&gt;, and to page when it crosses about &lt;strong&gt;120 days&lt;/strong&gt;. Not a ticket. A page — the same severity as a failed backup, because that is what it is: past roughly one quarter, you no longer have evidence that a human can restore this system, only evidence that a script once could.&lt;/p&gt;

&lt;p&gt;In Prometheus terms it is a timestamp gauge the drill updates when — and only when — a manual restore passes its fingerprint comparison:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# Written by the drill checklist's final step, after a human-run&lt;/span&gt;
&lt;span class="c"&gt;# restore passes the fingerprint comparison:&lt;/span&gt;
&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"last_successful_manual_restore_timestamp $(date +%s)"&lt;/span&gt; \
  &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; /var/lib/node_exporter/textfile/restore_drill.prom
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;# Page when no successful manual restore in ~120 days
(time() - last_successful_manual_restore_timestamp) / 86400 &amp;gt; 120
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Readers of the ladder post will recognise the shape: it is the dead-man's-switch from that article, pointed at the humans. There, the monitored absence was &lt;em&gt;verification success&lt;/em&gt; — page when the checks stop succeeding. Here, the monitored absence is the &lt;em&gt;drill itself&lt;/em&gt; — page when the people stop practising. Automation watches the backups; the gauge watches the watchers' habits. A restore you have not performed in four months is a hypothesis, and the gauge exists to say so out loud while it is still cheap to fix.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where does the fingerprint live?
&lt;/h2&gt;

&lt;p&gt;One honest open question, because it came up in the thread and I do not have dafili's answer yet: &lt;strong&gt;where should the fingerprint be stored?&lt;/strong&gt; I asked him directly in the conversation; while his reply is pending, the trade-offs are worth stating plainly, because the obvious answer has a thorn in it.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;With the backup configuration&lt;/strong&gt; is convenient — the drill finds it without ceremony. But it risks the fingerprint drifting &lt;em&gt;with&lt;/em&gt; the thing it verifies: the same change that breaks the backup can quietly rewrite the expectations, the way a scale that recalibrates itself to whatever it weighs can never report a wrong weight.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;In a separate repository, owned like a runbook,&lt;/strong&gt; keeps the declaration independent and reviewable — changes arrive as commits a second person can see. The cost is one more place to keep current, and a drill that must fetch its expectations from somewhere it trusts.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;On paper, or in the incident binder,&lt;/strong&gt; sounds like a joke until you remember what the fingerprint is for: it is the declaration that survives the system. Four values fit on an index card.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Wherever it lives, the property to protect is independence: the fingerprint must not be writable by the pipeline it judges. When dafili answers, this section gets an update — his reasoning has improved this series twice already.&lt;/p&gt;

&lt;h2&gt;
  
  
  Restore-drill fingerprint checklist
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;A fingerprint file exists for each system that matters, written by hand — not generated from the system it verifies&lt;/li&gt;
&lt;li&gt;It declares all four: an expected file list, one file's sha256, one row count (or range), one config value&lt;/li&gt;
&lt;li&gt;A human re-declares it quarterly, on the same cadence as the full restore drill&lt;/li&gt;
&lt;li&gt;The nightly drill restores to scratch and compares actuals against the fingerprint automatically&lt;/li&gt;
&lt;li&gt;A failed comparison pages — and legitimate drift is resolved by deliberate re-declaration, never by editing the check to be quiet&lt;/li&gt;
&lt;li&gt;A &lt;code&gt;days_since_last_successful_manual_restore&lt;/code&gt; gauge (or equivalent) exists, with a page at about 120 days&lt;/li&gt;
&lt;li&gt;The fingerprint is stored somewhere the backup pipeline itself cannot rewrite&lt;/li&gt;
&lt;li&gt;Expectations in older checks that were derived from the checked system have been replaced with declared ones&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  FAQ
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Isn't this just the level-4 verification from the previous post?&lt;/strong&gt;&lt;br&gt;
It is level 4 done honestly. The ladder says &lt;em&gt;restore and verify&lt;/em&gt;; the fingerprint answers the question that post left open — verify &lt;em&gt;against what?&lt;/em&gt; If the answer is "against values taken from the restore itself," the verification is circular. Declared expectations are what break the circle.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Four checks seem too few. Shouldn't the fingerprint cover everything?&lt;/strong&gt;&lt;br&gt;
Coverage is the automation's job — checksums, listings, and sampled reads already sweep the whole artifact nightly. The fingerprint has a different job: to be an independent witness. A witness who testifies about everything testifies about nothing; four declarations a human can hold in their head, and re-declare with full attention each quarter, beat four hundred generated ones nobody reads.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What if the business can't state a row count in advance?&lt;/strong&gt;&lt;br&gt;
Declare a range or a floor, stated from business knowledge ("orders grow by roughly ten thousand a month; a restore below last quarter's count is wrong"). Precision is not the point — independence is. An expectation that is roughly right and genuinely external catches the catastrophic wrongness the drill exists to catch.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Does the gauge replace scheduling the quarterly drill?&lt;/strong&gt;&lt;br&gt;
No — it catches the schedule failing. Calendars get ignored under load; that is precisely the failure the gauge exists to surface. Schedule the drill &lt;em&gt;and&lt;/em&gt; monitor its successful completion, the same way you schedule backups &lt;em&gt;and&lt;/em&gt; monitor verification success.&lt;/p&gt;

&lt;h2&gt;
  
  
  Summary
&lt;/h2&gt;

&lt;p&gt;The ladder from the previous post — exists, intact, readable, restored — tells you how to check a backup. Dafili's fingerprint tells you what makes the checking mean something: expectations a human declared, standing outside the system, that the system cannot rewrite to flatter itself. &lt;em&gt;You cannot automate the declaration, only the comparison.&lt;/em&gt; So declare by hand, re-declare quarterly, compare nightly, shout on drift — and keep a gauge on the drill itself, paging at about 120 days, so the practice cannot die in silence.&lt;/p&gt;

&lt;p&gt;Credit where it is due: the fingerprint doctrine, the declaration/comparison line, and the drill gauge all come from &lt;strong&gt;dafili&lt;/strong&gt;, in the comment thread under &lt;a href="https://dev.to/dafili/my-backup-exited-0-for-three-weeks-and-nothing-paged-me-5ce6"&gt;"My Backup Exited 0 for Three Weeks and Nothing Paged Me"&lt;/a&gt; — the same thread that produced the ladder post and corrected it. His follow-up, &lt;a href="https://dev.to/dafili/three-checks-that-look-like-they-passed-but-validated-nothing-ejd"&gt;"Three checks that look like they passed but validated nothing"&lt;/a&gt;, is the natural next read. This series keeps getting better because a reader keeps arguing with it in public. Long may that continue.&lt;/p&gt;

</description>
      <category>devops</category>
      <category>backup</category>
      <category>monitoring</category>
      <category>tutorial</category>
    </item>
    <item>
      <title>Backup Verification in 2026: Exit Code 0 Is Not a Backup</title>
      <dc:creator>Rabin Adhikari</dc:creator>
      <pubDate>Sun, 01 Nov 2026 05:19:56 +0000</pubDate>
      <link>https://dev.to/softverseadmin/backup-verification-in-2026-exit-code-0-is-not-a-backup-2fd7</link>
      <guid>https://dev.to/softverseadmin/backup-verification-in-2026-exit-code-0-is-not-a-backup-2fd7</guid>
      <description>&lt;h1&gt;
  
  
  Backup Verification in 2026: Exit Code 0 Is Not a Backup
&lt;/h1&gt;

&lt;p&gt;A story made the rounds in a DevOps community recently, and it should make anyone who owns a cron job uncomfortable. An engineer checked on a backup script that had been running perfectly by every visible measure — green in the scheduler, exit code 0, night after night — and found it had been "succeeding" at backing up nothing at all for three weeks, ever since something upstream had quietly changed. No error. No page. Just three weeks of confidence with nothing underneath it.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;TL;DR:&lt;/strong&gt; A backup job's exit code tells you the script ran, not that you have a backup. A backup is only trustworthy at four levels: (1) the artifact exists, (2) it is structurally intact, (3) it can be opened and read, and (4) it has actually been restored and the data verified. Most teams verify level 1 and stop. This guide walks a practical verification ladder — checksums, &lt;code&gt;restic check&lt;/code&gt;, &lt;code&gt;pg_restore --list&lt;/code&gt;, scheduled restore drills — plus the monitoring change that matters most: page on &lt;strong&gt;verification&lt;/strong&gt; failure, not job failure.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;&lt;em&gt;Originally published on &lt;a href="https://www.devopsenginer.com/blog/backup-verification-in-2026-exit-code-0-is-not-a-backup" rel="noopener noreferrer"&gt;DevOps Enginer&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;This is deliberately not a Kubernetes backup guide — cluster tooling like Velero and etcd snapshots is covered in the Kubernetes Backup and Disaster Recovery post. This is about the layer underneath every stack: database dumps, file archives, and object-storage backups, and how to prove they work before the day you need them.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why exit code 0 lies
&lt;/h2&gt;

&lt;p&gt;Exit code 0 means one narrow thing: the last command in your script did not report an error. It says nothing about what the script actually captured. Backup scripts fail silently in a handful of classic ways:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Wrong or empty source paths.&lt;/strong&gt; A directory gets moved, a mount fails to come up, or a path is renamed upstream. The archiver dutifully compresses an empty directory and exits 0. Your backup is a perfectly valid archive of nothing.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Expired or rotated credentials.&lt;/strong&gt; The dump tool fails to authenticate and writes a 0-byte file — or the wrapper script swallows the error, logs a warning nobody reads, and still exits successfully because the &lt;em&gt;upload&lt;/em&gt; of that empty file worked fine.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Silent partial captures.&lt;/strong&gt; A database dump taken without the right flags can complete while missing schemas, extensions, or large objects. The file is large, recent, and wrong.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The pattern underneath all of these: the job was monitored, but the &lt;em&gt;backup&lt;/em&gt; was never verified. Those are different jobs, and they need different checks.&lt;/p&gt;

&lt;h2&gt;
  
  
  The four levels of backup confidence
&lt;/h2&gt;

&lt;p&gt;A useful way to think about backup verification is as a ladder. Each level proves something the level below cannot:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Level 1 — The artifact exists.&lt;/strong&gt; A file or snapshot with the expected name appeared in the expected place, recently, with a plausible size. This is what most monitoring checks. It proves a process ran.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Level 2 — The artifact is structurally intact.&lt;/strong&gt; The archive is not truncated or bit-rotted. Checksums match; the backup tool's own integrity check confirms its internal structures — indexes, metadata, pack files — are consistent. This proves the bits survived storage.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Level 3 — The artifact can be opened and read.&lt;/strong&gt; The restore tool can actually parse the backup: list the table of contents of a dump, enumerate the files in an archive, read back a sample of the stored data. This proves the format is valid and the content is what you think it is — not an empty tree, not the wrong source.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Level 4 — The artifact has been restored and the data verified.&lt;/strong&gt; You restored it into a real (throwaway) environment and checked the result: the database answers queries, row counts and key tables look right, application files open. This is the only level that proves you can recover. Everything below it is inference.&lt;/p&gt;

&lt;p&gt;Notice what the ladder implies: levels 1–3 are cheap and automatable on every backup. Level 4 is more expensive, so you schedule it — sampled nightly or weekly, full restores on a rotation. What you cannot do is pretend level 1 is level 4. That gap is exactly where the three-weeks-of-nothing story lives.&lt;/p&gt;

&lt;h2&gt;
  
  
  The verification ladder in practice
&lt;/h2&gt;

&lt;p&gt;Here is how to test yours, level by level. The examples use common tools — &lt;code&gt;restic&lt;/code&gt;, PostgreSQL dumps, plain tarballs in object storage — but the pattern transfers to whatever you run.&lt;/p&gt;

&lt;h3&gt;
  
  
  Level 1: existence, freshness, and size sanity
&lt;/h3&gt;

&lt;p&gt;Check that the newest artifact exists, is younger than your backup interval (plus slack), and is within a sane size band of its predecessors. A backup that suddenly shrinks by 90% is a failed backup wearing a success costume.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# Fail loudly if no backup newer than 26 hours exists in the bucket&lt;/span&gt;
aws s3 ls s3://my-backups/postgres/ --recursive \
  &lt;span class="o"&gt;|&lt;/span&gt; sort &lt;span class="o"&gt;|&lt;/span&gt; tail -n 1

&lt;span class="c"&gt;# Size sanity: compare today's artifact against the 7-day median&lt;/span&gt;
&lt;span class="c"&gt;# and alert if it deviates by more than ~50%&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Tools like restic make this easier because the repository keeps snapshot metadata:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;restic snapshots --last 1 --json   &lt;span class="c"&gt;# newest snapshot time, paths, tags&lt;/span&gt;
restic stats                       &lt;span class="c"&gt;# repository size trend&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Encode the rule: &lt;strong&gt;a missing, stale, or suddenly-tiny backup is a verification failure&lt;/strong&gt;, even if the job that produced it exited 0.&lt;/p&gt;

&lt;h3&gt;
  
  
  Level 2: structural integrity with checksums and tool-native checks
&lt;/h3&gt;

&lt;p&gt;If you produce plain archives, generate a checksum at backup time and verify it at check time:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;sha256sum -c postgres-2026-11-01.dump.sha256
gzip -t nightly-files.tar.gz   &lt;span class="c"&gt;# quick structural test for gzip streams&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;restic has integrity checking built in:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;restic check
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;restic check&lt;/code&gt; verifies the repository structure — all packs, indexes, and metadata present and consistent. It is fast and belongs in the nightly job. What it does &lt;em&gt;not&lt;/em&gt; do by default is re-read your backup data — that is level 3.&lt;/p&gt;

&lt;h3&gt;
  
  
  Level 3: prove it can be read back
&lt;/h3&gt;

&lt;p&gt;For archives, listing the contents is the minimum viable read test — it forces the tool to walk the whole structure:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;tar -tzf nightly-files.tar.gz &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; /dev/null   &lt;span class="c"&gt;# fails on truncation/corruption&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For PostgreSQL custom-format dumps, &lt;code&gt;pg_restore --list&lt;/code&gt; parses the dump's table of contents without restoring anything:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;pg_restore --list postgres-2026-11-01.dump &lt;span class="o"&gt;|&lt;/span&gt; head -n 50
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Scan that listing for what &lt;em&gt;should&lt;/em&gt; be there: your critical tables, extensions, and the expected object count. An empty dump has a valid header and almost nothing under it — this is the check that catches the "backed up nothing" failure mode.&lt;/p&gt;

&lt;p&gt;For restic, the read-back check is:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;restic check --read-data-subset=5%
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Understand the caveat before you rely on it: &lt;code&gt;--read-data-subset=5%&lt;/code&gt; reads and verifies &lt;strong&gt;5% of the data packs&lt;/strong&gt;, not 5% of each file. It is sampling. A corrupt pack outside the sample will not be found tonight. Two practices make sampling honest:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Rotate with the n/t form, not the percentage.&lt;/strong&gt; &lt;code&gt;--read-data-subset=5%&lt;/code&gt; picks a &lt;em&gt;random&lt;/em&gt; subset on every run, so runs can overlap and some packs may never get sampled at all. The &lt;code&gt;n/t&lt;/code&gt; form is the deterministic one: restic splits the packs into t groups and checks group n. Pin it to the calendar — &lt;code&gt;--read-data-subset=$(date +%u)/7&lt;/code&gt; checks a different seventh each night and guarantees every pack is read within a week, for the same total I/O as one full check spread out.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Run a full read periodically.&lt;/strong&gt; &lt;code&gt;restic check --read-data&lt;/code&gt; (100%) on a monthly cadence closes any remaining gaps.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Also add a content spot-check at this level: restore a handful of known files to a temp directory and diff them, or query the snapshot for a file you know changed yesterday. Existence of &lt;em&gt;expected content&lt;/em&gt; is a much stronger signal than existence of &lt;em&gt;an artifact&lt;/em&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  Level 4: restore drills — the only level that counts
&lt;/h3&gt;

&lt;p&gt;Restore into a throwaway environment and verify the result like a user would. For PostgreSQL:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# Restore last night's dump into a scratch database&lt;/span&gt;
createdb -T template0 restore_test
pg_restore --no-owner --dbname=restore_test postgres-2026-11-01.dump

&lt;span class="c"&gt;# Then verify like the application would&lt;/span&gt;
psql restore_test -c &lt;span class="s2"&gt;"SELECT count(*) FROM users;"&lt;/span&gt;
psql restore_test -c &lt;span class="s2"&gt;"SELECT max(created_at) FROM orders;"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Row counts in a plausible range and a recent &lt;code&gt;max(created_at)&lt;/code&gt; on your busiest table prove the restore contains live, current data — not a skeleton schema and not last month's state.&lt;/p&gt;

&lt;p&gt;For file backups, restore the latest snapshot to a scratch path and verify file counts and a sample of checksums against expectations:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;restic restore latest --target /tmp/restore-test
find /tmp/restore-test -type f &lt;span class="o"&gt;|&lt;/span&gt; wc -l
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The drill is not done when the restore command exits 0. The drill is done when a check &lt;em&gt;of the restored data&lt;/em&gt; passes. And time the drill: the duration you measure is your real recovery time, not the one in the runbook.&lt;/p&gt;

&lt;h2&gt;
  
  
  Make verification a separate job that can fail
&lt;/h2&gt;

&lt;p&gt;The single most important structural change: &lt;strong&gt;verification must be its own job, with its own failure state, that runs after the backup job.&lt;/strong&gt; Folded into the backup script, a verification failure disappears into the same green checkmark. Give it its own exit code and its own alert path.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;#!/usr/bin/env bash&lt;/span&gt;
&lt;span class="c"&gt;# verify-backups.sh — runs after the backup job, fails independently&lt;/span&gt;
&lt;span class="nb"&gt;set&lt;/span&gt; -euo pipefail

&lt;span class="nv"&gt;LATEST&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$(&lt;/span&gt;restic snapshots --last 1 --json &lt;span class="o"&gt;|&lt;/span&gt; jq -r &lt;span class="s1"&gt;'.[0].id'&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;

restic check                                   &lt;span class="c"&gt;# level 2&lt;/span&gt;
restic check --read-data-subset=5%             &lt;span class="c"&gt;# level 3 (sampled)&lt;/span&gt;
restic restore &lt;span class="s2"&gt;"$LATEST"&lt;/span&gt; --target /tmp/restore-test --include /srv/app/config
&lt;span class="nb"&gt;test&lt;/span&gt; -s /tmp/restore-test/srv/app/config/app.env   &lt;span class="c"&gt;# expected content present&lt;/span&gt;

&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"backup_verification_success $(date +%s)"&lt;/span&gt; \
  &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; /var/lib/node_exporter/textfile/backup_verification.prom
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The script writes a &lt;strong&gt;success timestamp&lt;/strong&gt; only if every level passed — and that timestamp is what you monitor.&lt;/p&gt;

&lt;h2&gt;
  
  
  Monitor the verification, not the job
&lt;/h2&gt;

&lt;p&gt;Alerting on "backup job failed" catches crashes. It does not catch the failure mode this article is about, because in that failure mode the job &lt;em&gt;succeeds&lt;/em&gt;. Change what you page on:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Page on verification failure.&lt;/strong&gt; Any level failing — stale artifact, size anomaly, checksum mismatch, unreadable dump, failed drill — is a page-worthy event, because it means you currently do not have a proven backup.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Page on missing success.&lt;/strong&gt; Track "time since last successful &lt;em&gt;verification&lt;/em&gt;" and alert when it exceeds your threshold (for a nightly cycle, ~30 hours). This dead-man's-switch pattern catches the silent cases: the verification job itself dying, the scheduler stopping, credentials expiring before the check even runs. Absence of failure is not presence of success — so monitor the presence of success.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Track your verified restore point.&lt;/strong&gt; The metric that matters in an incident is "how old is the newest backup that has passed level 4?" If that number quietly grows to three weeks, you want to know before the incident, not during it.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If you use Prometheus, the textfile pattern above plus one alert — &lt;code&gt;time() - backup_verification_success &amp;gt; 30 * 3600&lt;/code&gt; — covers most of this. The tooling is not the hard part; deciding that verification failure pages like a production outage is.&lt;/p&gt;

&lt;h2&gt;
  
  
  A restore-drill cadence you can actually keep
&lt;/h2&gt;

&lt;p&gt;Verification fails as a practice when it depends on someone remembering. Put it on a calendar:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Cadence&lt;/th&gt;
&lt;th&gt;Scope&lt;/th&gt;
&lt;th&gt;Levels&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Nightly (automated)&lt;/td&gt;
&lt;td&gt;Every backup, right after it lands&lt;/td&gt;
&lt;td&gt;1–3, with n/t-rotated read-data subsets&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Weekly (automated)&lt;/td&gt;
&lt;td&gt;Partial level-4: restore one database or one snapshot subset to a scratch environment; run content checks&lt;/td&gt;
&lt;td&gt;4 (partial)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Monthly (automated + reviewed)&lt;/td&gt;
&lt;td&gt;Full restore of &lt;strong&gt;one&lt;/strong&gt; production system on rotation, timed end to end&lt;/td&gt;
&lt;td&gt;4 (full)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Quarterly&lt;/td&gt;
&lt;td&gt;Full restore drill treated like a game day: restore, verify, record duration and surprises, update the runbook&lt;/td&gt;
&lt;td&gt;4 (full, measured)&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The rotation matters: every system should get a full restore at least once a quarter, rather than an annual hero-drill on the one system everybody already understands. Record each drill's duration — that number is your measured RTO, and it is almost always worse than the assumed one. Finding that out in a drill is the entire point.&lt;/p&gt;

&lt;p&gt;Cloud snapshots deserve a specific warning here: a snapshot of a disk or database is level 1 by default. Snapshots faithfully preserve corruption, they can be silently incomplete if taken mid-write without coordination, and "we have snapshots" has ended more than one incident review badly. Snapshots need the same ladder — restore one into an isolated instance, open it, query it — before they count as backups.&lt;/p&gt;

&lt;h2&gt;
  
  
  Backup verification checklist
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Every backup has a &lt;strong&gt;separate verification job&lt;/strong&gt; with its own pass/fail, running after the backup completes&lt;/li&gt;
&lt;li&gt;Level 1: freshness and size-band checks on every artifact, every cycle&lt;/li&gt;
&lt;li&gt;Level 2: checksums verified for plain archives; &lt;code&gt;restic check&lt;/code&gt; (or your tool's equivalent) nightly&lt;/li&gt;
&lt;li&gt;Level 3: read-back tests — &lt;code&gt;pg_restore --list&lt;/code&gt;, &lt;code&gt;tar -t&lt;/code&gt;, &lt;code&gt;restic check --read-data-subset&lt;/code&gt; rotated with the &lt;strong&gt;n/t&lt;/strong&gt; form, plus expected-content spot checks&lt;/li&gt;
&lt;li&gt;Level 4: automated partial restores weekly; full restores monthly on rotation; quarterly timed drill&lt;/li&gt;
&lt;li&gt;Monitoring pages on &lt;strong&gt;verification failure&lt;/strong&gt; and on &lt;strong&gt;missing verification success&lt;/strong&gt; (~30h for nightly cycles)&lt;/li&gt;
&lt;li&gt;"Age of newest level-4-verified backup" is a visible metric, not a surprise&lt;/li&gt;
&lt;li&gt;Restore durations are recorded, so your RTO is measured rather than assumed&lt;/li&gt;
&lt;li&gt;Snapshots are restore-tested like any other backup before being trusted&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  FAQ
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Is &lt;code&gt;restic check&lt;/code&gt; on its own enough?&lt;/strong&gt;&lt;br&gt;
It covers level 2 well and, with &lt;code&gt;--read-data-subset&lt;/code&gt;, samples level 3. It never restores anything, so it cannot prove level 4. Treat it as the nightly baseline, not the whole strategy.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Doesn't all this verification double my costs?&lt;/strong&gt;&lt;br&gt;
Levels 1–3 are metadata, checksums, listings, and sampled reads — cheap. Level 4 restores cost scratch compute for a few hours a month. Compare that with the cost of discovering, mid-incident, that the backups were empty.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;My backups are managed by my cloud provider. Am I covered?&lt;/strong&gt;&lt;br&gt;
Managed backup services still produce artifacts that can be empty, misconfigured, or pointed at the wrong source. The provider runs the job; verification that the artifact restores &lt;em&gt;your&lt;/em&gt; data is still yours.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What is the one change to make first?&lt;/strong&gt;&lt;br&gt;
Add the dead-man's-switch alert on "time since last successful verification." It is a small change, and it converts the worst failure mode — silent, indefinite, total — into a page within a day.&lt;/p&gt;

&lt;h2&gt;
  
  
  Summary
&lt;/h2&gt;

&lt;p&gt;An exit code tells you a script ran. A trustworthy backup has been checked at four levels: it exists, it is intact, it can be read, and it has been restored with the data verified. Automate the first three nightly; schedule the fourth — weekly partials, monthly full restores on rotation, quarterly timed drills. And page when &lt;em&gt;verification&lt;/em&gt; fails or goes missing, not merely when a job crashes. The engineer in the opening story had monitoring. What they did not have was a single check that asked the only question that matters: &lt;em&gt;if I needed this backup tonight, could I restore it?&lt;/em&gt; Build the checks that answer that question while the answer is still cheap.&lt;/p&gt;

</description>
      <category>devops</category>
      <category>backup</category>
      <category>monitoring</category>
      <category>tutorial</category>
    </item>
    <item>
      <title>Prometheus in 2026: Metrics, PromQL and Alerting That Pages Only When It Matters</title>
      <dc:creator>Rabin Adhikari</dc:creator>
      <pubDate>Sat, 31 Oct 2026 11:58:56 +0000</pubDate>
      <link>https://dev.to/softverseadmin/prometheus-in-2026-metrics-promql-and-alerting-that-pages-only-when-it-matters-p7a</link>
      <guid>https://dev.to/softverseadmin/prometheus-in-2026-metrics-promql-and-alerting-that-pages-only-when-it-matters-p7a</guid>
      <description>&lt;h1&gt;
  
  
  Prometheus in 2026: Metrics, PromQL and Alerting That Pages Only When It Matters
&lt;/h1&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;TL;DR:&lt;/strong&gt; Prometheus fails teams in two predictable ways: cardinality explosions from unbounded labels, and alert noise that trains everyone to ignore pages. The fixes are disciplines, not features. Model metrics as counters, gauges, and histograms with small, bounded label sets; measure services with RED (rate, errors, duration) and resources with USE (utilisation, saturation, errors); compute ratios and quantiles with recording rules so dashboards and alerts stay fast; and alert on user-visible symptoms with an action attached — via Alertmanager grouping and inhibition — never on every cause you can measure. Keep Prometheus itself for what it is (superb short-to-medium-term metrics), and remote-write to long-term storage when history matters.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;&lt;em&gt;Originally published on &lt;a href="https://www.devopsenginer.com/blog/prometheus-monitoring-guide-2026" rel="noopener noreferrer"&gt;DevOps Enginer&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The model in one picture
&lt;/h2&gt;

&lt;p&gt;Prometheus &lt;strong&gt;pulls&lt;/strong&gt;: on an interval, it scrapes an HTTP endpoint (conventionally &lt;code&gt;/metrics&lt;/code&gt;) on each target, parses the exposition format, and stores samples in a time-series database. A &lt;strong&gt;time series&lt;/strong&gt; is a metric name plus a set of labels — &lt;code&gt;http_requests_total{job="api", status="200"}&lt;/code&gt; — and each unique combination is a separate series with its own storage and query cost. That one sentence explains almost every Prometheus behaviour and failure mode in this guide.&lt;/p&gt;

&lt;p&gt;Three metric types do the work:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Counters&lt;/strong&gt; only increase (requests served, errors seen). You almost never graph a counter raw; you graph its &lt;code&gt;rate()&lt;/code&gt; — which also handles counter resets when a process restarts.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Gauges&lt;/strong&gt; go up and down (queue depth, memory in use, in-flight requests). Current value, min/max/avg over a window — no rate needed.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Histograms&lt;/strong&gt; count observations into buckets (&lt;code&gt;le&lt;/code&gt; labels), which is what makes server-side quantiles possible: &lt;code&gt;histogram_quantile()&lt;/code&gt; aggregates across instances, which summary-type metrics cannot do correctly. Choose bucket boundaries around the latencies you actually care about — your SLO threshold must live inside your buckets to be measurable.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What to measure: RED and USE
&lt;/h2&gt;

&lt;p&gt;Instrumentation fails by excess more often than by absence. Two checklists keep it purposeful:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;RED for every service and endpoint that matters:&lt;/strong&gt; request &lt;strong&gt;Rate&lt;/strong&gt;, &lt;strong&gt;Errors&lt;/strong&gt; (ratio, not just count), &lt;strong&gt;Duration&lt;/strong&gt; (histogram). This is the user-visible truth of a service, and the raw material of every good alert below.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;USE for every resource:&lt;/strong&gt; &lt;strong&gt;Utilisation&lt;/strong&gt; (busy fraction), &lt;strong&gt;Saturation&lt;/strong&gt; (queued work — the earliest warning), &lt;strong&gt;Errors&lt;/strong&gt;. CPU steal aside, saturation metrics are the ones that predict incidents instead of narrating them.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Everything else — business metrics, per-feature counters, debug detail — is optional gravy. High-cardinality detail belongs in &lt;strong&gt;logs&lt;/strong&gt; (the logging guide's correlation IDs are exactly for this), not in labels.&lt;/p&gt;

&lt;h2&gt;
  
  
  Cardinality: the budget everyone blows
&lt;/h2&gt;

&lt;p&gt;Series count is the product of label-value combinations. A metric with 10 endpoints × 5 status codes × 4 instances is 200 series — fine. Add a &lt;code&gt;user_id&lt;/code&gt; label and it becomes millions — the database slows, memory climbs, queries time out, and the postmortem writes itself. The rules:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Labels identify dimensions with few, bounded values&lt;/strong&gt;: endpoint (normalised — &lt;code&gt;/users/123&lt;/code&gt; becomes &lt;code&gt;/users/:id&lt;/code&gt; before it is a label), status class or code, method, instance, zone, version.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Never label with unbounded values&lt;/strong&gt;: user IDs, request IDs, session tokens, full URLs, error message text. Each is a series factory.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Scrape-time relabeling is the safety net&lt;/strong&gt;: Prometheus can drop or rewrite label values during scraping — the place to strip an offending label from a third-party exporter you cannot change.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Watch the budget&lt;/strong&gt;: the targets and metrics with the highest series counts should be known numbers in your stack, reviewed like cloud spend (the Kubernetes cost guide makes the same argument for compute).&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  PromQL that earns its keep
&lt;/h2&gt;

&lt;p&gt;Four patterns cover most production questions:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Error ratio&lt;/strong&gt; (the alert-worthy form of errors):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;sum(rate(http_requests_total{job="api", status=~"5.."}[5m]))
/
sum(rate(http_requests_total{job="api"}[5m]))
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Latency quantile from a histogram:&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;histogram_quantile(0.95,
  sum by (le) (rate(http_request_duration_seconds_bucket{job="api"}[10m])))
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Saturation&lt;/strong&gt; — in-flight work versus capacity is usually a gauge ratio; the shape matters more than the exact metric names.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Absence detection&lt;/strong&gt; — &lt;code&gt;absent()&lt;/code&gt; catches the series that &lt;em&gt;stopped&lt;/em&gt;, which is how dead exporters and vanished jobs announce themselves; pair it with a dead man's switch on the alerting pipeline itself, so "no alerts" can be distinguished from "alerting is broken."&lt;/p&gt;

&lt;p&gt;Two habits separate useful PromQL from fragile PromQL: always &lt;code&gt;rate()&lt;/code&gt; counters (never subtract samples by hand), and aggregate with &lt;code&gt;sum by (...)&lt;/code&gt; naming exactly the labels you intend to keep — a query that silently sums across a label you cared about produces confident, wrong dashboards.&lt;/p&gt;

&lt;h2&gt;
  
  
  Recording rules: pay for queries once
&lt;/h2&gt;

&lt;p&gt;Dashboards and alerts re-run the same expressions relentlessly. &lt;strong&gt;Recording rules&lt;/strong&gt; precompute them into new series on an interval:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;groups&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;api-slo&lt;/span&gt;
    &lt;span class="na"&gt;interval&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;30s&lt;/span&gt;
    &lt;span class="na"&gt;rules&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;record&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;job:http_error_ratio:5m&lt;/span&gt;
        &lt;span class="na"&gt;expr&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;|&lt;/span&gt;
          &lt;span class="s"&gt;sum(rate(http_requests_total{job="api", status=~"5.."}[5m]))&lt;/span&gt;
          &lt;span class="s"&gt;/&lt;/span&gt;
          &lt;span class="s"&gt;sum(rate(http_requests_total{job="api"}[5m]))&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Now the alert is trivial, fast, and testable against the recorded series:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;alert&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ApiHighErrorRate&lt;/span&gt;
        &lt;span class="na"&gt;expr&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;job:http_error_ratio:5m &amp;gt; 0.05&lt;/span&gt;
        &lt;span class="na"&gt;for&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;10m&lt;/span&gt;
        &lt;span class="na"&gt;labels&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;severity&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;page&lt;/span&gt;
        &lt;span class="na"&gt;annotations&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;summary&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"API 5xx ratio above 5% for 10 minutes"&lt;/span&gt;
          &lt;span class="na"&gt;runbook_url&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"https://wiki.example.com/runbooks/api-errors"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Note the anatomy of a good alert: a symptom (error &lt;em&gt;ratio&lt;/em&gt;, sustained), a &lt;code&gt;for&lt;/code&gt; duration that ignores blips, a severity, and a &lt;strong&gt;runbook link&lt;/strong&gt;. An alert without a next action is a notification, not an alert.&lt;/p&gt;

&lt;h2&gt;
  
  
  Alerting philosophy: symptoms page, causes ticket
&lt;/h2&gt;

&lt;p&gt;Alert fatigue is an availability risk in its own right — a team paged twenty times a night stops reading page one. The discipline that fixes it:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Page on symptoms users feel&lt;/strong&gt;: error ratio, latency burn, saturation of the user path. These have a natural threshold — the SLO — and a natural owner.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Ticket (or dashboard) the causes&lt;/strong&gt;: disk filling, a replica down, a certificate ageing. Important, rarely 3 a.m.-shaped; a Deployment with one of three replicas down is healthy by design and should page nobody.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Burn-rate thinking&lt;/strong&gt;: rather than a single static threshold, alert on how fast the error budget is being consumed (fast burn pages, slow burn tickets). Recording rules make multi-window burn alerts practical; the concept matters more than any specific window pair.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Alertmanager does the traffic control&lt;/strong&gt;: &lt;strong&gt;grouping&lt;/strong&gt; collapses a hundred instance alerts from one outage into one notification; &lt;strong&gt;inhibition&lt;/strong&gt; suppresses downstream symptom alerts while a known root alert fires; &lt;strong&gt;silences&lt;/strong&gt; mute deliberately during maintenance; &lt;strong&gt;routing&lt;/strong&gt; sends each alert to the team that owns the action. Configure these on purpose — default Alertmanager behaviour is a starting point, not a design.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Architecture and storage honesty
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Local retention is short by design.&lt;/strong&gt; A Prometheus server's local TSDB commonly retains around 15 days by default; that is a feature of its role — fast, recent, operational metrics — not a limitation to apologise for.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Long-term history goes elsewhere&lt;/strong&gt;: remote-write to Thanos, Grafana Mimir, or VictoriaMetrics (the common open options) when you need months of data for capacity and trend work. Keep the operational Prometheus queryable even if the long-term tier is down.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Federation is rarely the answer&lt;/strong&gt; it first appears to be; hierarchical scraping duplicates data and queries painfully. Prefer remote-write and per-team/per-cluster servers with a global query layer.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;In Kubernetes&lt;/strong&gt;, the Prometheus Operator's &lt;code&gt;ServiceMonitor&lt;/code&gt;/&lt;code&gt;PodMonitor&lt;/code&gt; resources turn scrape config into the same GitOps-reviewed YAML as everything else in this series (the observability guide covers the wider stack); &lt;code&gt;kube-state-metrics&lt;/code&gt; supplies object state (pending pods, failed jobs, replica mismatches) and node exporter supplies host USE metrics. Watch the operator's own cardinality — Kubernetes state is prolific.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Short-lived jobs&lt;/strong&gt; can't be scraped on an interval; the Pushgateway exists for them, with real caveats: it doesn't expire series for you, and pushed metrics lose the scrape's health signal. Prefer recording outcomes from the platform side (the Jobs guide's failure conditions) or pushing sparingly and deliberately.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Common mistakes
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Unbounded labels&lt;/strong&gt;, the cardinal sin above — most often a well-meant &lt;code&gt;user_id&lt;/code&gt;, &lt;code&gt;path&lt;/code&gt;, or &lt;code&gt;error&lt;/code&gt; label that turns a metric into a million series over a weekend.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Alerting on causes.&lt;/strong&gt; CPU at 80% pages; the error budget untouched. Pages spent on non-symptoms are how real symptoms get muted.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Counters graphed raw&lt;/strong&gt; — ever-climbing lines that say nothing; or worse, alerts on them. &lt;code&gt;rate()&lt;/code&gt; exists; use it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Averaging quantiles.&lt;/strong&gt; Averaging per-instance p95s produces a number that is no instance's p95 and nobody's truth. Histograms plus &lt;code&gt;histogram_quantile&lt;/code&gt; over the aggregate is the correct shape.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;No recording rules on heavy dashboards&lt;/strong&gt; — the same 30-day, high-cardinality query from every open browser tab, grinding the server that is supposed to diagnose the incident.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Treating Prometheus as the archive.&lt;/strong&gt; Local retention quietly rotates; the quarter's capacity review finds 15 days of data. Decide the retention tiers like log retention (logging guide): hot and short in Prometheus, long in the remote tier.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Silent alerting pipeline.&lt;/strong&gt; Alertmanager down, webhook expired, routing typo — discovered during the incident it existed to announce. A dead man's switch (an always-firing alert whose &lt;em&gt;absence&lt;/em&gt; pages) is the standard guard.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Where this fits
&lt;/h2&gt;

&lt;p&gt;Prometheus is the metrics pillar of the observability guide's triad: metrics say &lt;em&gt;what and how much&lt;/em&gt;, logs say &lt;em&gt;what exactly&lt;/em&gt;, traces say &lt;em&gt;where in the path&lt;/em&gt;. The discipline rhymes across all three — bounded cardinality here, volume control in logging, sampling in tracing — because all three are the same budget wearing different clothes: keep the signal, shed the noise, and make sure the alerting end of the pipeline deserves the trust the team puts in it.&lt;/p&gt;

&lt;h2&gt;
  
  
  FAQ
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Prometheus or Grafana — do I need both?&lt;/strong&gt;&lt;br&gt;
They are different layers: Prometheus collects and stores metrics and evaluates alerts; Grafana visualises (Prometheus has a basic built-in UI for queries, not dashboards). Most stacks run Prometheus (+ Alertmanager) for data and alerting, Grafana for dashboards — Grafana also fronts logs and traces, which is where the triad becomes one investigation.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Pull or push — which is right?&lt;/strong&gt;&lt;br&gt;
Pull, by default: it gives Prometheus target health for free (&lt;code&gt;up&lt;/code&gt; series), centralises interval control, and survives target restarts gracefully. Push (Pushgateway) is the exception for genuinely short-lived batch work, used with its caveats in mind.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;How long should Prometheus retain data?&lt;/strong&gt;&lt;br&gt;
Operationally, two to four weeks local is common and usually plenty for incident work. Anything longer belongs in a remote-write long-term tier, where storage is cheaper and the queries are trend-shaped rather than incident-shaped.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Prometheus or OpenTelemetry?&lt;/strong&gt;&lt;br&gt;
Complementary, not rivals. OpenTelemetry is instrumentation and pipelines (APIs, SDKs, the Collector); Prometheus is a metrics backend with its own exposition format, which OTel can produce and feed (directly or via remote-write-compatible paths). Many stacks instrument with OTel and still query with PromQL.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;When have I outgrown a single Prometheus?&lt;/strong&gt;&lt;br&gt;
When scrape volume, cardinality, or query load makes the single server the bottleneck — symptoms are scrape duration creeping toward the interval, slow range queries, and memory pressure dominated by series churn. The step is sharding by team/cluster plus a remote-write tier, not a bigger instance forever.&lt;/p&gt;

</description>
      <category>prometheus</category>
      <category>monitoring</category>
      <category>observability</category>
      <category>devops</category>
    </item>
    <item>
      <title>Kubernetes Jobs and CronJobs in 2026: Batch Work That Finishes (or Fails) Cleanly</title>
      <dc:creator>Rabin Adhikari</dc:creator>
      <pubDate>Sat, 31 Oct 2026 11:48:58 +0000</pubDate>
      <link>https://dev.to/softverseadmin/kubernetes-jobs-and-cronjobs-in-2026-batch-work-that-finishes-or-fails-cleanly-4ad2</link>
      <guid>https://dev.to/softverseadmin/kubernetes-jobs-and-cronjobs-in-2026-batch-work-that-finishes-or-fails-cleanly-4ad2</guid>
      <description>&lt;h1&gt;
  
  
  Kubernetes Jobs and CronJobs in 2026: Batch Work That Finishes (or Fails) Cleanly
&lt;/h1&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;TL;DR:&lt;/strong&gt; Deployments run forever; Jobs run to completion — and that one difference changes how you configure everything. A production Job needs an explicit contract: &lt;code&gt;restartPolicy: Never&lt;/code&gt; (or OnFailure, knowingly), a &lt;code&gt;backoffLimit&lt;/code&gt; you chose, an &lt;code&gt;activeDeadlineSeconds&lt;/code&gt; so nothing runs forever, and &lt;code&gt;ttlSecondsAfterFinished&lt;/code&gt; so finished work cleans itself up. CronJobs add three decisions on top: a &lt;code&gt;timeZone&lt;/code&gt;, a &lt;code&gt;concurrencyPolicy&lt;/code&gt; that matches whether runs may overlap, and a &lt;code&gt;startingDeadlineSeconds&lt;/code&gt; so a controller outage doesn't fire a pile of missed runs at once. Batch failures are normal — design for retries that are idempotent, and alert on failure rather than discovering it in the data.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;&lt;em&gt;Originally published on &lt;a href="https://www.devopsenginer.com/blog/kubernetes-jobs-cronjobs-guide-2026" rel="noopener noreferrer"&gt;DevOps Enginer&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The workload Deployments can't express
&lt;/h2&gt;

&lt;p&gt;Everything earlier in this series assumes services: replicas that should always exist, replaced when they die. Batch work inverts the assumption — a migration, a report, a backup, an import has a &lt;em&gt;done&lt;/em&gt;, and success means the pods stop existing. Running batch work in a Deployment produces the classic pathologies: a completed container restarted forever, a failing task CrashLooping with no concept of "give up," no record of whether the run succeeded.&lt;/p&gt;

&lt;p&gt;The Job controller exists to own that lifecycle: create pods, watch them finish, count successes against &lt;code&gt;completions&lt;/code&gt;, count failures against &lt;code&gt;backoffLimit&lt;/code&gt;, and stop. CronJob is a Job factory on a schedule — nothing more mystical than that, which is the right way to reason about both.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Job contract, field by field
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;batch/v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Job&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;nightly-export&lt;/span&gt;
&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;completions&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
  &lt;span class="na"&gt;parallelism&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
  &lt;span class="na"&gt;backoffLimit&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt;
  &lt;span class="na"&gt;activeDeadlineSeconds&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;3600&lt;/span&gt;
  &lt;span class="na"&gt;ttlSecondsAfterFinished&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;86400&lt;/span&gt;
  &lt;span class="na"&gt;template&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;restartPolicy&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Never&lt;/span&gt;
      &lt;span class="na"&gt;containers&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;export&lt;/span&gt;
          &lt;span class="na"&gt;image&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;registry.example.com/export:1.4.2&lt;/span&gt;
          &lt;span class="na"&gt;args&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"--date"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"yesterday"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The fields that decide behaviour:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;restartPolicy: Never&lt;/code&gt; (or &lt;code&gt;OnFailure&lt;/code&gt;)&lt;/strong&gt; — Jobs require one of these two; &lt;code&gt;Always&lt;/code&gt; is rejected. The choice matters: with &lt;strong&gt;Never&lt;/strong&gt;, a failed container means the pod fails and the controller creates a &lt;em&gt;replacement pod&lt;/em&gt; (visible, countable, each with its own logs). With &lt;strong&gt;OnFailure&lt;/strong&gt;, the kubelet restarts the container &lt;em&gt;inside the same pod&lt;/em&gt; — fewer pod objects, murkier failure history. Default to Never unless you have a reason; it makes failures legible.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;backoffLimit&lt;/code&gt; (default 6)&lt;/strong&gt; — how many pod failures before the Job itself is marked Failed. Each retry waits with exponential backoff (10s, 20s, 40s… capped at 6 minutes). Six silent retries of a broken image is how a bad deploy burns an hour unnoticed; for most batch work 2–4 is a saner, louder choice. Finer control exists via &lt;strong&gt;pod failure policies&lt;/strong&gt; — rules that fail the Job immediately on specific exit codes (a config error will never fix itself) while ignoring disruptions like node drains.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;activeDeadlineSeconds&lt;/code&gt;&lt;/strong&gt; — the Job's total time budget; exceed it and the Job fails and its pods are terminated. This is the field that prevents the "export job from March is still running" class of surprise.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;ttlSecondsAfterFinished&lt;/code&gt;&lt;/strong&gt; — after the Job completes or fails, wait this long, then delete the Job and its pods. Without it, finished Jobs and their pod corpses accumulate forever (and with them, cluttered &lt;code&gt;kubectl get pods&lt;/code&gt; output and etcd weight). A day is a reasonable default window for debugging; the logging pipeline from the logging guide is what preserves evidence beyond it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;completions&lt;/code&gt; / &lt;code&gt;parallelism&lt;/code&gt;&lt;/strong&gt; — for work queues: &lt;code&gt;completions&lt;/code&gt; is the total successes required, &lt;code&gt;parallelism&lt;/code&gt; the pod ceiling. Leave both unset for the simple one-shot case (one success, one pod). &lt;strong&gt;Indexed Jobs&lt;/strong&gt; (&lt;code&gt;completionMode: Indexed&lt;/code&gt;) give each pod a stable index (&lt;code&gt;JOB_COMPLETION_INDEX&lt;/code&gt;) for sharded work — partition the input by index instead of running a separate queue service.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Idempotency: the requirement retries impose
&lt;/h2&gt;

&lt;p&gt;A retried Job runs your code again against a world the first attempt may have half-changed. Batch code must therefore be &lt;strong&gt;idempotent or transactional&lt;/strong&gt;: write to a staging key and publish atomically, upsert rather than insert, name outputs deterministically (by date/partition), and check-before-create where the sink allows. The failure this prevents is specific and common: attempt one writes 90% of the export and dies; attempt two appends a second 100%; downstream reads 190% and nobody knows which records are real. Retries are the controller doing its job — making them safe is yours.&lt;/p&gt;

&lt;h2&gt;
  
  
  CronJobs: a schedule plus three decisions
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;batch/v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;CronJob&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;nightly-export&lt;/span&gt;
&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;schedule&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"15 2 * * *"&lt;/span&gt;
  &lt;span class="na"&gt;timeZone&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Asia/Tokyo"&lt;/span&gt;
  &lt;span class="na"&gt;concurrencyPolicy&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Forbid&lt;/span&gt;
  &lt;span class="na"&gt;startingDeadlineSeconds&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;900&lt;/span&gt;
  &lt;span class="na"&gt;successfulJobsHistoryLimit&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt;
  &lt;span class="na"&gt;failedJobsHistoryLimit&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
  &lt;span class="na"&gt;jobTemplate&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;backoffLimit&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt;
      &lt;span class="na"&gt;activeDeadlineSeconds&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;3600&lt;/span&gt;
      &lt;span class="na"&gt;ttlSecondsAfterFinished&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;86400&lt;/span&gt;
      &lt;span class="na"&gt;template&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;restartPolicy&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Never&lt;/span&gt;
          &lt;span class="na"&gt;containers&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;export&lt;/span&gt;
              &lt;span class="na"&gt;image&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;registry.example.com/export:1.4.2&lt;/span&gt;
              &lt;span class="na"&gt;args&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"--date"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"yesterday"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;timeZone&lt;/code&gt;&lt;/strong&gt; — set it, always. Without it the schedule runs in the controller-manager's timezone (effectively UTC on most clusters), and "2:15 a.m." silently means something else after a cluster migration. The field takes an IANA name and handles daylight-saving transitions the way the name implies.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;concurrencyPolicy&lt;/code&gt;&lt;/strong&gt; — the overlap decision, made explicitly:

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;Allow&lt;/code&gt; (the default): runs may stack up. Right for independent, cheap, idempotent work; dangerous when a slow run plus a punctual schedule compounds load.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;Forbid&lt;/code&gt;: skip a scheduled run while the previous is still going. The right default for most sequential batch work — exports, backups, syncs.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;Replace&lt;/code&gt;: kill the running Job and start fresh. For "latest state wins" work where a stale run is worse than none.&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;startingDeadlineSeconds&lt;/code&gt;&lt;/strong&gt; — if the controller misses a start (downtime, upgrades), how late a run may still fire. Without it, a controller recovering after an outage can launch a burst of stale runs; with it, missed runs beyond the window are skipped. Roughly 100 missed starts is the controller's own safety cap, but your deadline should be far tighter and deliberate.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;History limits&lt;/strong&gt; — &lt;code&gt;successfulJobsHistoryLimit&lt;/code&gt; (default 3) and &lt;code&gt;failedJobsHistoryLimit&lt;/code&gt; (default 1) bound how many finished Job objects linger for inspection. Failed history is debugging gold and set lowest by default — raise it if you investigate on Mondays what failed on Fridays.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;suspend: true&lt;/code&gt;&lt;/strong&gt; — the pause button. CronJobs honour it immediately; it is the clean way to stop a schedule during an incident without deleting the object.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Failure is a normal state — observe it like one
&lt;/h2&gt;

&lt;p&gt;A service failing pages someone; a Job failing at 2 a.m. writes a status field nobody reads. Close that gap deliberately:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Alert on Job failure and on CronJob staleness.&lt;/strong&gt; The signals exist in the metrics stack from the observability guide: job-failed conditions, and time since a CronJob's last successful run versus its schedule. "The nightly export hasn't succeeded in 26 hours" is an alert; discovering it from a stale dashboard is an incident review.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Logs outlive pods only if you shipped them.&lt;/strong&gt; Failed pods are deleted by TTL and history limits by design — the pipeline from the logging guide is the memory. At minimum, make the failure visible where the team already looks (the alert carries the job name; the runbook says where its logs live).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Distinguish "failed" from "never ran."&lt;/strong&gt; Scheduler outages, suspended CronJobs, image-pull failures before any pod starts — the Job object tells different stories in each case (conditions, events, pod statuses), and the fixes differ. The troubleshooting guide's habit applies: read the object's status before touching anything.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Resource and disruption behaviour for batch
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Requests still matter, limits protect neighbours.&lt;/strong&gt; Batch jobs are the classic noisy neighbour — bursty CPU during everyone else's peak. Set requests honestly (requests-and-limits guide) and consider scheduling heavy batch into off-peak windows or lower-priority classes so services win contention; a report can wait 20 minutes, a checkout cannot.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Disruption budgets apply differently.&lt;/strong&gt; A Job's pods are not protected by PDBs the way replicas are (the PDB guide covers voluntary disruptions) — node drains during a long Job mean pod termination and a controller retry. Long jobs should checkpoint progress or be restartable from partial state; the idempotency work above is what makes that survivable rather than catastrophic.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;One-off manual runs:&lt;/strong&gt; &lt;code&gt;kubectl create job --from=cronjob/nightly-export export-manual-1&lt;/code&gt; clones the template for an immediate run — the clean pattern for backfills and tests, far better than hand-writing a twin Job manifest that drifts from the scheduled one.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  When a Job is the wrong tool
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Long-running or never-finishing work&lt;/strong&gt; is a service — use a Deployment.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Per-node work&lt;/strong&gt; (agents, collectors) is a DaemonSet's shape, not a Job's.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Complex DAG workflows&lt;/strong&gt; — fan-out/fan-in pipelines, artefact passing, per-step retries with dependencies — outgrow the Job controller quickly. That is the territory of workflow engines (Argo Workflows and kin); the Job primitives remain the right answer for the large simple majority: one task, a schedule or a trigger, a definition of done.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Common mistakes
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;restartPolicy: Always&lt;/code&gt; copied from a Deployment.&lt;/strong&gt; The API rejects it for Jobs — usually discovered at apply time in a hurry. The template is not interchangeable.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;No deadline.&lt;/strong&gt; A hung Job holds resources and blocks &lt;code&gt;Forbid&lt;/code&gt;-policy successors indefinitely; the schedule looks fine and nothing runs. &lt;code&gt;activeDeadlineSeconds&lt;/code&gt; is the difference between a failure and a mystery.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Retries without idempotency.&lt;/strong&gt; The double-write pattern above; the controller retries by design, so the hazard is guaranteed to fire eventually.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Default &lt;code&gt;backoffLimit&lt;/code&gt; and no alerting.&lt;/strong&gt; Six quiet retries, a Failed status, and silence. Batch observability is an alerting decision, not a dashboard.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;CronJobs in the wrong timezone.&lt;/strong&gt; The 2 a.m. job running at 11 a.m. after a migration is a rite of passage; the &lt;code&gt;timeZone&lt;/code&gt; field is how you skip it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;Allow&lt;/code&gt; concurrency on heavy work.&lt;/strong&gt; One slow run plus an hourly schedule equals a self-inflicted load test by morning.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Secrets via environment for one-shot jobs.&lt;/strong&gt; Jobs end; their specs (and any env values in them) linger in history objects and etcd. The ConfigMaps and Secrets guide's file-based patterns apply with extra force to objects designed to be kept around as records.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Where this fits in the series
&lt;/h2&gt;

&lt;p&gt;Batch is the third workload shape in this series, beside services (Deployments) and per-node agents (DaemonSets): it leans on the requests-and-limits guide for fair scheduling, the storage guide when jobs move real data, the backup guide — whose restores and snapshots are themselves usually Jobs — and the autoscaling guide's cluster-autoscaler behaviour, which treats pending batch pods as the elastic demand they are. Model the done-state honestly and the controller does the rest.&lt;/p&gt;

&lt;h2&gt;
  
  
  FAQ
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Job or CronJob for a task I trigger from CI?&lt;/strong&gt;&lt;br&gt;
A Job, created by the pipeline when the event happens (a migration after deploy, an export on demand). CronJob is for wall-clock schedules; forcing event-driven work into a polling CronJob adds latency and pointless runs.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Why did my CronJob not run at the expected time?&lt;/strong&gt;&lt;br&gt;
In order of likelihood: timezone (no &lt;code&gt;timeZone&lt;/code&gt; set, cluster in UTC), the previous run still going under &lt;code&gt;Forbid&lt;/code&gt;, &lt;code&gt;suspend: true&lt;/code&gt; left on, or a missed start beyond &lt;code&gt;startingDeadlineSeconds&lt;/code&gt; after controller downtime. The CronJob's status and events record which.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;How many times will a failing Job retry?&lt;/strong&gt;&lt;br&gt;
&lt;code&gt;backoffLimit&lt;/code&gt; pod failures (default 6), with exponential backoff between attempts — then the Job is marked Failed and stops. With pod failure policies, specific exit codes can fail it sooner, on the first occurrence.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Can a Job run on a schedule &lt;em&gt;and&lt;/em&gt; be triggered manually?&lt;/strong&gt;&lt;br&gt;
Yes — define the CronJob, and create one-off Jobs from its template with &lt;code&gt;kubectl create job --from=cronjob/&amp;lt;name&amp;gt;&lt;/code&gt;. The manual run uses the same spec; it just isn't part of the schedule's history.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Should batch jobs share the cluster with services?&lt;/strong&gt;&lt;br&gt;
Usually, yes — with honest requests, sane priorities, and schedules that respect peak hours. Dedicated node pools earn their cost when batch is heavy, spiky, or needs special hardware; reach for isolation when contention is measured, not imagined.&lt;/p&gt;

</description>
      <category>kubernetes</category>
      <category>batch</category>
      <category>cronjobs</category>
      <category>tutorial</category>
    </item>
    <item>
      <title>Kubernetes RBAC in 2026: Roles, Bindings and Least Privilege That Actually Holds</title>
      <dc:creator>Rabin Adhikari</dc:creator>
      <pubDate>Sat, 31 Oct 2026 11:40:23 +0000</pubDate>
      <link>https://dev.to/softverseadmin/kubernetes-rbac-in-2026-roles-bindings-and-least-privilege-that-actually-holds-4080</link>
      <guid>https://dev.to/softverseadmin/kubernetes-rbac-in-2026-roles-bindings-and-least-privilege-that-actually-holds-4080</guid>
      <description>&lt;h1&gt;
  
  
  Kubernetes RBAC in 2026: Roles, Bindings and Least Privilege That Actually Holds
&lt;/h1&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;TL;DR:&lt;/strong&gt; Kubernetes RBAC fails in practice not because the model is weak but because grants accrete: a wildcard here, a cluster-admin binding there, a "temporary" service account that becomes permanent. The model itself is simple — subjects get verbs on resources in a scope, grants are additive only, and nothing is denied, only not granted. Least privilege that holds means: one service account per workload with token automount off by default, namespace-scoped Roles as the default unit, groups (not individuals) as human subjects, secrets treated as the crown jewels they are, and a regular audit of who can do what — verified with &lt;code&gt;kubectl auth can-i&lt;/code&gt;, not assumed.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;&lt;em&gt;Originally published on &lt;a href="https://www.devopsenginer.com/blog/kubernetes-rbac-guide-2026" rel="noopener noreferrer"&gt;DevOps Enginer&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Why RBAC is where cluster security is won or lost
&lt;/h2&gt;

&lt;p&gt;The security-basics guide in this series covered the checklist; this guide goes deep on the one control that decides what a compromised credential can actually do. Most cluster incidents follow the same shape: an attacker (or a fat-fingered script) obtains a token — from a pod, a CI job, a leaked kubeconfig — and the blast radius is whatever that token may do. Network policies limit where it can talk; RBAC limits what it can &lt;em&gt;be&lt;/em&gt;. A read-only token in one namespace is an embarrassment. A cluster-admin token is a breach report.&lt;/p&gt;

&lt;p&gt;And RBAC sprawl is the default outcome, because every grant is a small act of unblocking someone: the deploy pipeline "needs" broader rights, the dashboard "needs" cluster-wide secrets, the new tool "needs" admin "for setup." Each is reasonable on the day. Together they are the incident, pre-positioned.&lt;/p&gt;

&lt;h2&gt;
  
  
  The model, precisely
&lt;/h2&gt;

&lt;p&gt;Four ideas carry the whole system:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Subjects&lt;/strong&gt; — who: a &lt;strong&gt;User&lt;/strong&gt; (external identity, typically from a certificate or OIDC token), a &lt;strong&gt;Group&lt;/strong&gt;, or a &lt;strong&gt;ServiceAccount&lt;/strong&gt; (in-cluster identity).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Rules&lt;/strong&gt; — what: &lt;strong&gt;verbs&lt;/strong&gt; (&lt;code&gt;get&lt;/code&gt;, &lt;code&gt;list&lt;/code&gt;, &lt;code&gt;watch&lt;/code&gt;, &lt;code&gt;create&lt;/code&gt;, &lt;code&gt;update&lt;/code&gt;, &lt;code&gt;patch&lt;/code&gt;, &lt;code&gt;delete&lt;/code&gt;, and a few specials) on &lt;strong&gt;resources&lt;/strong&gt; (&lt;code&gt;pods&lt;/code&gt;, &lt;code&gt;deployments&lt;/code&gt;, &lt;code&gt;secrets&lt;/code&gt;…) within &lt;strong&gt;API groups&lt;/strong&gt; (&lt;code&gt;""&lt;/code&gt; for core, &lt;code&gt;apps&lt;/code&gt;, &lt;code&gt;batch&lt;/code&gt;, &lt;code&gt;rbac.authorization.k8s.io&lt;/code&gt;…).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Scope&lt;/strong&gt; — where: a &lt;strong&gt;Role&lt;/strong&gt; grants within one namespace; a &lt;strong&gt;ClusterRole&lt;/strong&gt; grants cluster-wide or for cluster-scoped resources (nodes, persistent volumes, namespaces themselves).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Bindings&lt;/strong&gt; — the connection: a &lt;strong&gt;RoleBinding&lt;/strong&gt; attaches a Role &lt;em&gt;or&lt;/em&gt; ClusterRole to subjects &lt;strong&gt;inside one namespace&lt;/strong&gt; (a ClusterRole bound by a RoleBinding is effectively trimmed to that namespace); a &lt;strong&gt;ClusterRoleBinding&lt;/strong&gt; attaches a ClusterRole everywhere.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Two properties shape everything else:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Additive only.&lt;/strong&gt; There are no deny rules. A subject's rights are the union of all its bindings. "Remove the bad grant" always means deleting or narrowing a binding — and "who granted what" questions are answered by enumerating bindings, never by finding an exception.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Default deny.&lt;/strong&gt; With RBAC as the authorisation mode, a subject with no grants can do nothing. Every capability is an explicit decision — which is why the accretion pattern above is a process failure, not a model failure.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  A worked example: read-only log access for support
&lt;/h2&gt;

&lt;p&gt;The support team needs to read pods and logs in one production namespace — nothing else, nowhere else:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;rbac.authorization.k8s.io/v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Role&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;namespace&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;shop-prod&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;log-reader&lt;/span&gt;
&lt;span class="na"&gt;rules&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;apiGroups&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;""&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="na"&gt;resources&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"pods"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"pods/log"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="na"&gt;verbs&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"get"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"list"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"watch"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="nn"&gt;---
&lt;/span&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;rbac.authorization.k8s.io/v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;RoleBinding&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;namespace&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;shop-prod&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;log-reader-support&lt;/span&gt;
&lt;span class="na"&gt;subjects&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Group&lt;/span&gt;
    &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;support-engineers&lt;/span&gt;
    &lt;span class="na"&gt;apiGroup&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;rbac.authorization.k8s.io&lt;/span&gt;
&lt;span class="na"&gt;roleRef&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Role&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;log-reader&lt;/span&gt;
  &lt;span class="na"&gt;apiGroup&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;rbac.authorization.k8s.io&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Details doing quiet work here: the subject is a &lt;strong&gt;Group&lt;/strong&gt; (people join and leave groups; bindings should not name individuals), &lt;code&gt;pods/log&lt;/code&gt; is a subresource that must be granted explicitly, and there is no &lt;code&gt;secrets&lt;/code&gt;, no &lt;code&gt;create&lt;/code&gt;, no second namespace. Verify it the way an auditor would:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl auth can-i get pods --namespace shop-prod --as jane --as-group support-engineers   &lt;span class="c"&gt;# yes&lt;/span&gt;
kubectl auth can-i get secrets --namespace shop-prod --as jane --as-group support-engineers &lt;span class="c"&gt;# no&lt;/span&gt;
kubectl auth can-i list pods --all-namespaces --as jane --as-group support-engineers         &lt;span class="c"&gt;# no&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;kubectl auth can-i&lt;/code&gt; with &lt;code&gt;--as&lt;/code&gt;/&lt;code&gt;--as-group&lt;/code&gt; impersonation is the ground-truth test for any identity — build the habit of checking grants with it instead of reading YAML and reasoning hopefully.&lt;/p&gt;

&lt;h2&gt;
  
  
  Service accounts: the identities that actually get stolen
&lt;/h2&gt;

&lt;p&gt;Humans are rarely the compromised identity; workloads are. Service-account discipline is therefore the highest-value RBAC work:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;One service account per workload.&lt;/strong&gt; The &lt;code&gt;default&lt;/code&gt; service account in each namespace should have &lt;em&gt;no&lt;/em&gt; grants. A dedicated account per deployment makes grants reviewable and revocation surgical.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Turn off token automount unless needed.&lt;/strong&gt; Most applications never call the Kubernetes API. For them the token is pure attack surface:
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ServiceAccount&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;shop-api&lt;/span&gt;
  &lt;span class="na"&gt;namespace&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;shop-prod&lt;/span&gt;
&lt;span class="na"&gt;automountServiceAccountToken&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;(The same field exists per-pod in the pod spec, which wins when both are set.) Where a token &lt;em&gt;is&lt;/em&gt; needed, modern clusters issue short-lived projected (bound) tokens rather than long-lived secrets — prefer them, and treat any long-lived service-account token secret as a credential to inventory.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Deployer accounts are prime targets.&lt;/strong&gt; A CI service account that can create deployments in production can, by design, run arbitrary images with that namespace's service accounts — near-admin inside its scope. Narrow its verbs to exactly the resources the pipeline applies, keep it namespace-scoped, and protect its credential like a production key (the OIDC guide's keyless pattern applies to how the pipeline authenticates to the cloud; the in-cluster account deserves the same suspicion).&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  ClusterRoles: handle with both hands
&lt;/h2&gt;

&lt;p&gt;Cluster-wide grants are sometimes right — an observability agent reading node and pod metadata everywhere, an operator managing its resource type across namespaces. The disciplines:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Prefer a trimmed custom role over the built-ins' extremes.&lt;/strong&gt; The default &lt;code&gt;cluster-admin&lt;/code&gt; is total; &lt;code&gt;admin&lt;/code&gt;, &lt;code&gt;edit&lt;/code&gt;, and &lt;code&gt;view&lt;/code&gt; are namespace-oriented defaults with specific shapes (notably, &lt;code&gt;view&lt;/code&gt; excludes secrets — a deliberate, instructive choice). For cluster-wide read, write the ClusterRole that lists exactly the resources the tool reads; "cluster-reader" patterns from the community are a starting point, not a destination.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Aggregation composes, so audit the aggregate.&lt;/strong&gt; A ClusterRole with an &lt;code&gt;aggregationRule&lt;/code&gt; collects rules from labelled ClusterRoles (this is how the built-in admin/edit/view roles grow with CRDs). Powerful for platforms, dangerous when a stray label quietly widens a widely bound role — include aggregated roles in audits.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;ClusterRoleBindings deserve a list, kept short.&lt;/strong&gt; Every entry is a cluster-wide statement. Review the full list on a schedule; in a healthy cluster it fits on one screen and every line has an owner who can explain it.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The dangerous verbs and resources
&lt;/h2&gt;

&lt;p&gt;Audits should hunt specific shapes, because damage concentrates:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Wildcards&lt;/strong&gt; — &lt;code&gt;verbs: ["*"]&lt;/code&gt; or &lt;code&gt;resources: ["*"]&lt;/code&gt; in anything but a deliberately tiny set of admin roles. A wildcard silently includes tomorrow's new subresources.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;get&lt;/code&gt;/&lt;code&gt;list&lt;/code&gt; on &lt;code&gt;secrets&lt;/code&gt;&lt;/strong&gt; — especially cluster-wide. Reading secrets is effectively holding every credential in scope (the ConfigMaps and Secrets guide covers why). Treat any new secrets grant as a security review, not a config change.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;create&lt;/code&gt; on &lt;code&gt;pods&lt;/code&gt;&lt;/strong&gt; — pod creation with a chosen service account and mounts is a well-trodden path to privilege escalation; so are &lt;code&gt;create&lt;/code&gt; on rolebindings/clusterrolebindings and the special &lt;code&gt;bind&lt;/code&gt; and &lt;code&gt;escalate&lt;/code&gt; verbs, which exist precisely to gate grant-making itself. &lt;strong&gt;&lt;code&gt;impersonate&lt;/code&gt;&lt;/strong&gt; is equally sensitive: it &lt;em&gt;is&lt;/em&gt; the other identity, for whoever holds it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;deletecollection&lt;/code&gt;&lt;/strong&gt; and broad &lt;code&gt;delete&lt;/code&gt; — the availability side of the same coin.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Human access: groups, short-lived credentials, no shared kubeconfigs
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Map your identity provider's &lt;strong&gt;groups&lt;/strong&gt; to Group subjects once, and manage membership in the IdP — not by editing bindings per person. Joiner/leaver processes then work through the system people already trust for access changes.&lt;/li&gt;
&lt;li&gt;Issue &lt;strong&gt;short-lived&lt;/strong&gt; credentials: exec-plugin/OIDC flows from your cloud or IdP, so a copied kubeconfig expires instead of opening a permanent door. A long-lived client certificate cannot be revoked (Kubernetes has no certificate revocation for this) — another reason to prefer token-based, expiring auth.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Break-glass, deliberately.&lt;/strong&gt; Keep an emergency admin path, alarm on its use, and test it — an incident is the wrong time to discover nobody can grant the rights to fix it. Emergency access is a designed feature; routine access through it is the failure.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Auditing: make "who can do what" a query, not a debate
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;kubectl auth can-i --list --as &amp;lt;identity&amp;gt;&lt;/code&gt; prints an identity's effective rights in a namespace — the fastest spot check.&lt;/li&gt;
&lt;li&gt;Enumerate bindings as code review: RoleBindings and ClusterRoleBindings are YAML in your GitOps repo (the Argo CD guide's pattern) — changes to them should be pull requests with security-aware reviewers, not &lt;code&gt;kubectl&lt;/code&gt; improvisation.&lt;/li&gt;
&lt;li&gt;Tooling in the &lt;code&gt;kubectl-who-can&lt;/code&gt; / RBAC-lookup family answers the reverse question ("who can get secrets in prod?") across the cluster; run it in CI or on a schedule and diff the output — new entries are events worth a human glance.&lt;/li&gt;
&lt;li&gt;API-server &lt;strong&gt;audit logs&lt;/strong&gt; record who did what (the logging guide covers collecting them); RBAC denials in those logs are a signal of both attacks and of teams fighting their own permissions — both worth knowing.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Common mistakes
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Binding cluster-admin "temporarily."&lt;/strong&gt; Temporary grants are permanent until the audit that never happens. Time-box with a ticket, an owner, and a removal date — or better, grant the specific role the task needs.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;One service account for everything in the namespace.&lt;/strong&gt; Every pod inherits the union of rights; one compromised container spends them all.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Secrets access for convenience.&lt;/strong&gt; Dashboards and debug tools asking for cluster-wide secrets are asking for the keys to everything. Scope to the namespace, or redesign the tool's access.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Confusing RoleBinding-to-ClusterRole scope.&lt;/strong&gt; Binding a ClusterRole via a RoleBinding grants it &lt;strong&gt;only in that namespace&lt;/strong&gt; — teams regularly believe they granted (or limited) something else. Read the binding kind, not just the role kind.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Stale bindings.&lt;/strong&gt; People change teams; projects end; vendor tools are uninstalled and their service accounts and bindings remain, valid and forgotten. The audit diff is how they surface.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Testing with your own admin account.&lt;/strong&gt; It works for you, so permissions "work." Test as the workload's identity (&lt;code&gt;--as=system:serviceaccount:ns:name&lt;/code&gt; or by running the pod) before the 3 a.m. discovery.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Where this fits in the series
&lt;/h2&gt;

&lt;p&gt;RBAC is the identity layer the rest of the series assumes: the security-basics guide sets the posture, the network policies guide constrains where identities can reach, the ConfigMaps and Secrets guide protects what grants can expose, and the observability and logging guides watch it all — including the audit stream that records every grant being used. Get RBAC boring, and the other controls get to do their jobs.&lt;/p&gt;

&lt;h2&gt;
  
  
  FAQ
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Can RBAC deny a specific action?&lt;/strong&gt;&lt;br&gt;
No. RBAC is additive — a subject's permissions are the union of its grants, and there is no deny rule to carve exceptions. If a grant is too broad, narrow the role or split it; for true deny semantics (blocking an action even for the granted), admission-time policy tools are the layer that does it.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Role or ClusterRole for a cross-namespace tool?&lt;/strong&gt;&lt;br&gt;
If it genuinely acts in many namespaces, a ClusterRole — bound cluster-wide only if it must act everywhere, or via RoleBindings in selected namespaces when its reach should be enumerated namespace by namespace. The second pattern keeps the grant list explicit and reviewable.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Should each pod have its own service account?&lt;/strong&gt;&lt;br&gt;
Each &lt;em&gt;workload identity&lt;/em&gt; should: one per deployment/workload is the practical unit, shared by its replicas. Per-pod accounts multiply objects without adding a trust boundary; per-namespace sharing erases the boundary that exists.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;How do I see everything a service account can do?&lt;/strong&gt;&lt;br&gt;
&lt;code&gt;kubectl auth can-i --list --as=system:serviceaccount:&amp;lt;namespace&amp;gt;:&amp;lt;name&amp;gt; -n &amp;lt;namespace&amp;gt;&lt;/code&gt; for a namespace view, repeated per namespace or answered cluster-wide with RBAC lookup tooling. Make it a habit after every binding change.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Are the built-in admin/edit/view roles safe to use?&lt;/strong&gt;&lt;br&gt;
They are reasonable, maintained defaults — with shapes you should know: &lt;code&gt;view&lt;/code&gt; excludes secrets and most write access, &lt;code&gt;edit&lt;/code&gt; adds writes but not role management, &lt;code&gt;admin&lt;/code&gt; adds role management within the namespace but not namespace deletion or quota changes. The risk is rarely the built-ins; it is the custom wildcard beside them.&lt;/p&gt;

</description>
      <category>kubernetes</category>
      <category>security</category>
      <category>rbac</category>
      <category>tutorial</category>
    </item>
    <item>
      <title>Terraform Testing in 2026: Unit Tests With terraform test and Mock Providers</title>
      <dc:creator>Rabin Adhikari</dc:creator>
      <pubDate>Sat, 31 Oct 2026 11:23:47 +0000</pubDate>
      <link>https://dev.to/softverseadmin/terraform-testing-in-2026-unit-tests-with-terraform-test-and-mock-providers-2e5k</link>
      <guid>https://dev.to/softverseadmin/terraform-testing-in-2026-unit-tests-with-terraform-test-and-mock-providers-2e5k</guid>
      <description>&lt;h1&gt;
  
  
  Terraform Testing in 2026: Unit Tests With terraform test and Mock Providers
&lt;/h1&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;TL;DR:&lt;/strong&gt; Most Terraform changes are still reviewed by reading a plan and hoping. The native test framework (&lt;code&gt;terraform test&lt;/code&gt;, stable since Terraform 1.6) changes that: test files declare inputs, run a plan or apply, and assert on the results — and with mock providers (since 1.7) the logic tests run in CI in seconds with no cloud account and no credentials. Test your modules' contracts — outputs, conditional resources, naming and tagging rules — not the cloud provider's behaviour. Keep a small number of real apply tests for critical modules in an ephemeral sandbox, and let linters and scanners do their separate jobs alongside.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;&lt;em&gt;Originally published on &lt;a href="https://www.devopsenginer.com/blog/terraform-testing-guide-2026" rel="noopener noreferrer"&gt;DevOps Enginer&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The problem: the plan is not a test
&lt;/h2&gt;

&lt;p&gt;Application code gets unit tests as a matter of course. Infrastructure code — which can delete a database just as dead as any application bug — usually gets a human skimming &lt;code&gt;terraform plan&lt;/code&gt; output in a pull request. That review catches typos. It does not catch the conditional that inverted last refactor, the variable default that quietly changed a bucket from private to public, or the count logic that creates zero replicas of the thing production needs three of.&lt;/p&gt;

&lt;p&gt;The reason was tooling: for years, testing Terraform meant heavyweight frameworks or applying to a real account and inspecting by hand. The native framework removes the excuse. Tests live next to the module, run with one command, and — the decisive part — can run against &lt;strong&gt;mocked providers&lt;/strong&gt;, so the suite executes like unit tests: fast, hermetic, credential-free.&lt;/p&gt;

&lt;h2&gt;
  
  
  What is worth testing (and what is not)
&lt;/h2&gt;

&lt;p&gt;Test &lt;strong&gt;your module's decisions&lt;/strong&gt;, not the provider's implementation:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Outputs and contracts.&lt;/strong&gt; Callers depend on output values and shapes. If &lt;code&gt;bucket_name&lt;/code&gt; changes format, downstream code breaks — a test should break first.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Conditional logic.&lt;/strong&gt; Every &lt;code&gt;count&lt;/code&gt;, &lt;code&gt;for_each&lt;/code&gt;, and ternary is a decision your module makes: versioning on/off, optional resources created or skipped, single vs multi-AZ shapes. Each branch deserves a run.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Naming, tagging, and policy invariants.&lt;/strong&gt; The rules platform teams actually care about: names compose from project/environment correctly, required tags are always present, encryption is enabled, nothing is public unless an explicit variable says so.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Variable edge cases.&lt;/strong&gt; Empty lists, zero counts, nulls — the inputs real callers will eventually pass.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Do &lt;strong&gt;not&lt;/strong&gt; test that AWS creates an S3 bucket when asked, that an attribute AWS generates looks a certain way beyond your control, or other provider internals. Those tests are slow, brittle, and verify someone else's software.&lt;/p&gt;

&lt;h2&gt;
  
  
  The framework in one module
&lt;/h2&gt;

&lt;p&gt;A small module to test:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight hcl"&gt;&lt;code&gt;&lt;span class="c1"&gt;# main.tf&lt;/span&gt;
&lt;span class="n"&gt;variable&lt;/span&gt; &lt;span class="s2"&gt;"project"&lt;/span&gt;     &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="k"&gt;type&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kt"&gt;string&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="n"&gt;variable&lt;/span&gt; &lt;span class="s2"&gt;"environment"&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="k"&gt;type&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kt"&gt;string&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="n"&gt;variable&lt;/span&gt; &lt;span class="s2"&gt;"enable_versioning"&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;type&lt;/span&gt;    &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kt"&gt;bool&lt;/span&gt;
  &lt;span class="k"&gt;default&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="n"&gt;resource&lt;/span&gt; &lt;span class="s2"&gt;"aws_s3_bucket"&lt;/span&gt; &lt;span class="s2"&gt;"this"&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;bucket&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"${var.project}-${var.environment}-assets"&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="n"&gt;resource&lt;/span&gt; &lt;span class="s2"&gt;"aws_s3_bucket_versioning"&lt;/span&gt; &lt;span class="s2"&gt;"this"&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;bucket&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;aws_s3_bucket&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;this&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;

  &lt;span class="n"&gt;versioning_configuration&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;status&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kd"&gt;var&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;enable_versioning&lt;/span&gt; &lt;span class="o"&gt;?&lt;/span&gt; &lt;span class="s2"&gt;"Enabled"&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Suspended"&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="n"&gt;output&lt;/span&gt; &lt;span class="s2"&gt;"bucket_name"&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;value&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;aws_s3_bucket&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;this&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;bucket&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;And its test file (conventionally in a &lt;code&gt;tests/&lt;/code&gt; directory inside the module):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight hcl"&gt;&lt;code&gt;&lt;span class="c1"&gt;# tests/bucket.tftest.hcl&lt;/span&gt;
&lt;span class="n"&gt;mock_provider&lt;/span&gt; &lt;span class="s2"&gt;"aws"&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="n"&gt;variables&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;project&lt;/span&gt;     &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"shop"&lt;/span&gt;
  &lt;span class="n"&gt;environment&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"prod"&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="n"&gt;run&lt;/span&gt; &lt;span class="s2"&gt;"versioning_on_by_default"&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;command&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;plan&lt;/span&gt;

  &lt;span class="n"&gt;assert&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;condition&lt;/span&gt;     &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;aws_s3_bucket_versioning&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;this&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;versioning_configuration&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;"Enabled"&lt;/span&gt;
    &lt;span class="n"&gt;error_message&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"Versioning must be enabled by default"&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;

  &lt;span class="n"&gt;assert&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;condition&lt;/span&gt;     &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;output&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;bucket_name&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;"shop-prod-assets"&lt;/span&gt;
    &lt;span class="n"&gt;error_message&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"Bucket name must compose project and environment"&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="n"&gt;run&lt;/span&gt; &lt;span class="s2"&gt;"versioning_can_be_disabled"&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;command&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;plan&lt;/span&gt;

  &lt;span class="n"&gt;variables&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;enable_versioning&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;

  &lt;span class="n"&gt;assert&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;condition&lt;/span&gt;     &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;aws_s3_bucket_versioning&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;this&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;versioning_configuration&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;"Suspended"&lt;/span&gt;
    &lt;span class="n"&gt;error_message&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"Versioning should suspend when explicitly disabled"&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Run it with &lt;code&gt;terraform test&lt;/code&gt; from the module root. The anatomy:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;mock_provider "aws" {}&lt;/code&gt;&lt;/strong&gt; replaces the real provider. Terraform generates fake computed values, so a &lt;code&gt;plan&lt;/code&gt; (or even an &lt;code&gt;apply&lt;/code&gt;) runs without credentials, network, or a cloud account — in milliseconds.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;variables {}&lt;/code&gt;&lt;/strong&gt; at file level sets shared inputs; a &lt;code&gt;run&lt;/code&gt; block can override them per case.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;run&lt;/code&gt; blocks&lt;/strong&gt; are the test cases. &lt;code&gt;command = plan&lt;/code&gt; is the default workhorse; &lt;code&gt;command = apply&lt;/code&gt; executes against the (possibly mocked) provider for shape checks that only appear after apply.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;assert&lt;/code&gt;&lt;/strong&gt; pairs a condition over any value in the configuration — resources, outputs, variables, locals — with the &lt;code&gt;error_message&lt;/code&gt; a human reads when it fails. Write the messages as explanations, not echoes.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Where a mocked computed value needs to be specific — an ARN a policy references, an ID another resource consumes — &lt;code&gt;override_resource&lt;/code&gt; and &lt;code&gt;override_data&lt;/code&gt; blocks pin those values inside a run. Use them sparingly; a test that overrides everything asserts nothing.&lt;/p&gt;

&lt;h2&gt;
  
  
  Mock providers: what they change, and their honest limits
&lt;/h2&gt;

&lt;p&gt;Before mocks, a "unit test" of a module still configured the real provider, which meant credentials in CI, provider API calls in every plan, and suites measured in minutes. Mocks invert all three: the pipeline job needs &lt;strong&gt;no cloud credentials at all&lt;/strong&gt; for the test stage (which, per the keyless-CI pattern in the GitHub Actions OIDC guide on this site, is exactly where you want the fewest credentials), tests run at unit-test speed, and nobody's sandbox account accumulates test debris.&lt;/p&gt;

&lt;p&gt;The limits are equally clear, and stating them keeps the suite trustworthy:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Mocks validate &lt;strong&gt;your configuration logic against the provider's schema&lt;/strong&gt;, not against the real API. A value the real AWS API would reject (an invalid name pattern, an unsupported combination) can sail through a mocked plan.&lt;/li&gt;
&lt;li&gt;Data sources return generated values unless overridden — tests depending on a data source's real content need overrides or a real read.&lt;/li&gt;
&lt;li&gt;Provider-side quirks and eventual-consistency behaviour are, by definition, absent.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;So the strategy is layered: mocked tests for logic, in bulk; a few real tests for reality, on purpose.&lt;/p&gt;

&lt;h2&gt;
  
  
  The top of the pyramid: real apply tests, kept few
&lt;/h2&gt;

&lt;p&gt;For a critical module — the network foundation, the shared database module — a mocked plan cannot prove the one thing that matters: it actually builds. Keep a small suite of &lt;code&gt;command = apply&lt;/code&gt; runs against the real provider in a &lt;strong&gt;sandbox account&lt;/strong&gt; with three disciplines:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Ephemeral by design.&lt;/strong&gt; Tests create, assert, and the framework destroys what it created at the end of the run (that is the framework's contract for test-created resources). Verify the destroy happened; orphaned test resources are a cost and security smell — the state-management guide's hygiene rules apply to test state too.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Separate credentials, narrow scope.&lt;/strong&gt; The sandbox role can create and destroy only the resource types under test, in the sandbox account only. The OIDC guide's per-environment role pattern fits exactly here.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Scheduled, not per-commit.&lt;/strong&gt; Real applies take minutes and cost real (if small) money. Run them nightly or on module release, while the mocked suite gates every pull request.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Where tests sit among the other tools
&lt;/h2&gt;

&lt;p&gt;Terraform already has a quality neighbourhood, and the test framework does not evict anyone:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Variable validation blocks&lt;/strong&gt; — the module rejecting bad input at plan time. First line of defence, and the test suite should cover that they fire.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;terraform fmt&lt;/code&gt; / &lt;code&gt;validate&lt;/code&gt;&lt;/strong&gt; — shape and syntax. Free; run always.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;tflint and friends&lt;/strong&gt; — lint rules, naming conventions, deprecated syntax. Style and hygiene, not behaviour.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Checkov / tfsec-style scanners&lt;/strong&gt; — policy and security posture of the configuration. They check &lt;em&gt;what the config says&lt;/em&gt;; tests check &lt;em&gt;what the module decides&lt;/em&gt; given inputs. Complementary, not overlapping.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;terraform test&lt;/code&gt;&lt;/strong&gt; — behaviour: outputs, conditionals, invariants across input variations.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A sane pipeline order: fmt check → validate → lint/scan → &lt;code&gt;terraform test&lt;/code&gt; (mocked) → plan for human review. Each stage is cheap; together they mean the human reviews intent, not correctness arithmetic.&lt;/p&gt;

&lt;h2&gt;
  
  
  Modules are where tests pay rent
&lt;/h2&gt;

&lt;p&gt;For a root configuration used once, tests are nice. For &lt;strong&gt;modules&lt;/strong&gt; — shared, versioned, consumed by teams who did not write them — they are the contract enforcement mechanism. The modules guide in this series covers structuring reusable modules; tests are what let a maintainer refactor a module's internals while proving its callers see the same outputs, the same resource decisions, the same invariants. That confidence is precisely what a module library runs on, and it is why test files should ship inside the module repository, versioned with the code they describe.&lt;/p&gt;

&lt;h2&gt;
  
  
  Common mistakes
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Testing the provider.&lt;/strong&gt; Asserting that a mocked (or real) AWS resource has AWS-chosen properties teaches the suite to fail for other people's changes. Assert your decisions.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Overriding everything.&lt;/strong&gt; Heavy &lt;code&gt;override_resource&lt;/code&gt; use pins the fake world so completely that the test passes regardless of the configuration. Overrides are scalpels for specific computed values, not a substitute world.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;One giant run block.&lt;/strong&gt; A failed assert should name the broken decision. Many small runs with descriptive names beat one mega-run with forty asserts.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Apply tests against shared accounts.&lt;/strong&gt; Test resources in an account other teams use will eventually collide, linger, or be "cleaned up" by someone mid-run. Sandbox account, always.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Mocked-only confidence for critical paths.&lt;/strong&gt; The suite is green, the module cannot actually apply. Keep the few real apply tests for the modules whose failure would hurt.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Ignoring test state.&lt;/strong&gt; &lt;code&gt;terraform test&lt;/code&gt; manages its own state for runs, but interrupted real applies can leave residue; a periodic sandbox sweep belongs in the routine.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Writing tests once, at the end.&lt;/strong&gt; The suite earns its keep when a refactor lands six months later. Add the run block when you add the conditional — that is when the intent is clearest and cheapest to encode.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  FAQ
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;terraform test or Terratest?&lt;/strong&gt;&lt;br&gt;
They answer different questions. &lt;code&gt;terraform test&lt;/code&gt; is native, HCL-based, and ideal for module logic and contracts — no Go toolchain, no extra framework. Terratest (Go) remains strong for end-to-end verification that spans tools and real cloud behaviour beyond one module. Many teams use both: native tests in every module repo, Terratest-style end-to-end checks for platform assemblies.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Does &lt;code&gt;terraform test&lt;/code&gt; cost money?&lt;/strong&gt;&lt;br&gt;
Mocked and plan-only runs create nothing and cost nothing. &lt;code&gt;command = apply&lt;/code&gt; runs against a real provider create real resources for the duration of the test — priced accordingly, then destroyed by the framework. Keep them few, small, and in a sandbox.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Can tests run in CI without cloud credentials?&lt;/strong&gt;&lt;br&gt;
Mocked-provider tests: yes, completely — that is their headline feature. Plan runs against a real provider still need whatever credentials that provider requires to configure itself, which is one more reason the mocked suite should carry the bulk of the assertions.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Where do test files live?&lt;/strong&gt;&lt;br&gt;
In the module root or a &lt;code&gt;tests/&lt;/code&gt; subdirectory (the conventional choice), as &lt;code&gt;*.tftest.hcl&lt;/code&gt;. &lt;code&gt;terraform test&lt;/code&gt; discovers them in both places; keep one convention per repository.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What about OpenTofu?&lt;/strong&gt;&lt;br&gt;
OpenTofu implements the same test framework model — test files, run blocks, and mock providers are compatible concepts, so the approach in this guide carries across. Check your version's documentation for exact feature parity before assuming a specific block behaves identically.&lt;/p&gt;

</description>
      <category>terraform</category>
      <category>testing</category>
      <category>devops</category>
      <category>tutorial</category>
    </item>
  </channel>
</rss>
