Build a Kubernetes Upgrade Readiness Agent: Deprecated APIs, Add-on Checks, and a Migration PR

💡 Originally published on devtocash.com — where this guide stays updated. I write hands-on DevOps/SRE deep-dives there weekly.

Why upgrades rot at the bottom of the backlog

A Kubernetes upgrade readiness agent answers one question before you bump the control plane: what exactly will break, and what diff fixes it? It runs deterministic scanners (kubent, pluto) against the cluster and your manifests, checks your add-on versions against their support matrices, then uses an LLM for the one part scanners can't do — mapping each finding back to the actual file in your repo and opening a migration pull request. It never runs the upgrade itself.

The economics of postponing are brutal now. Kubernetes ships three minor versions a year and upstream supports each for about 14 months. EKS charges $0.10/hour per cluster on a standard-support version and $0.60/hour on extended support — roughly $5,300/year instead of $880 per cluster, purely for being late. Most teams aren't late because upgrades are hard; they're late because nobody can confidently answer "is anything going to break?" That answer is exactly what's automatable.

The three ways an upgrade breaks you

Every failed upgrade I've seen lands in one of three buckets, and only two of them are machine-checkable:

1. Removed APIs. Your manifests or Helm charts still say flowcontrol.apiserver.k8s.io/v1beta3 (removed in 1.32) or some other version that the new API server simply no longer serves. Applies start failing after the bump.
2. Add-on incompatibility. The CNI, ingress controller, cert-manager, or CSI driver you're running doesn't support the target version. The control plane upgrades fine; networking or storage degrades afterward.
3. Behavioral changes. Feature gates flip defaults, a scheduler nuance changes, a kubelet flag disappears. No scanner catches these — this bucket is why the agent drafts a plan for a human instead of pressing the button.

The agent's job is to drain buckets one and two completely so your review time goes to bucket three.

Layer 1: deterministic scanners, JSON out

Start with kubent (kube-no-trouble), which inspects live cluster objects — including the last-applied-configuration annotation and Helm release secrets — for APIs deprecated or removed in your target version:

kubent --target-version 1.34 --output json > kubent.json

There's a trap here worth understanding: the API server rewrites stored objects to the newest version it serves, so a Deployment you applied years ago as apps/v1beta1 is stored as apps/v1 today. A purely live-cluster scan can therefore look clean while your Git repo is full of dead API versions that will fail on the next kubectl apply. That's why you also scan the source of truth with Pluto:

# Static manifests in the repo
pluto detect-files -d ./manifests \
  --target-versions k8s=v1.34 -o json > pluto-files.json

# Rendered output of your Helm charts
helm template ./charts/api | pluto detect - \
  --target-versions k8s=v1.34 -o json > pluto-helm.json

Add two cheap checks the scanners don't cover — version skew and node lag:

kubectl version -o json | jq -r '.serverVersion.gitVersion'
kubectl get nodes -o custom-columns=NAME:.metadata.name,VERSION:.status.nodeInfo.kubeletVersion

Kubelets may lag the API server by up to three minor versions, but if half your nodes are already at the skew limit, the "one quick control-plane bump" is actually a node-group rollout too, and the plan should say so.

Layer 2: the add-on compatibility table

Add-on compat is a lookup, not a judgment call, so keep it in code. Enumerate what's installed:

helm list -A -o json | jq '.[] | {name, namespace, chart, app_version}'

Then check each against a small matrix you maintain in the repo:

# upgrade-agent/addon-matrix.yaml
target: "1.34"
addons:
  - name: ingress-nginx
    min_chart: "4.12.0"     # first release supporting 1.34
  - name: cert-manager
    min_chart: "v1.17.0"
  - name: aws-ebs-csi-driver
    min_chart: "2.40.0"
  - name: cilium
    min_chart: "1.17.0"

Yes, maintaining this file is toil — but it's your toil, reviewed in Git, instead of an LLM's guess about version compatibility. Hallucinated compatibility claims are precisely the failure mode you can't afford here, the same reason a Terraform drift agent decides the obvious cases with rules and saves the model for ranking. When a chart is below minimum, that's a deterministic finding: "upgrade ingress-nginx to ≥4.12.0 before the control plane."

Layer 3: the LLM maps findings to files

Here's what scanners genuinely can't do. kubent tells you FlowSchema "canary-priority" uses flowcontrol.apiserver.k8s.io/v1beta3, removed in 1.32. It does not know that this object lives at gitops/base/priority.yaml in your monorepo, that two overlays patch it, and that the v1 schema renamed nothing so the migration is a one-line apiVersion bump. Mapping finding → file → minimal diff is a reading-comprehension task over your repo, which is exactly what an LLM with a search tool does well.

Give the agent three read-only tools:

[
  {
    "name": "search_repo",
    "description": "Search the GitOps repo for a string. Returns file paths and matching lines.",
    "input_schema": {
      "type": "object",
      "properties": {"query": {"type": "string"}},
      "required": ["query"]
    }
  },
  {
    "name": "read_file",
    "description": "Read one file from the repo at HEAD.",
    "input_schema": {
      "type": "object",
      "properties": {"path": {"type": "string"}},
      "required": ["path"]
    }
  },
  {
    "name": "propose_change",
    "description": "Stage a unified diff for one file. Goes into a draft PR, never applied directly.",
    "input_schema": {
      "type": "object",
      "properties": {
        "path": {"type": "string"},
        "diff": {"type": "string"},
        "finding_id": {"type": "string"}
      },
      "required": ["path", "diff", "finding_id"]
    }
  }
]

And a system prompt that pins it to the evidence:

You are preparing a Kubernetes 1.34 upgrade. You will receive scanner
findings (kubent, pluto) and add-on matrix violations as JSON.

For each finding: locate the source file with search_repo, read it,
and stage the smallest diff that migrates it via propose_change.
Every diff must cite its finding_id. If you cannot find the source
file for a finding, report it as UNMAPPED — do not guess a path or
invent a manifest. Do not propose changes for anything not present
in a finding.

The finding_id linkage is the anti-hallucination guardrail: CI rejects any staged diff whose ID doesn't exist in the scanner output, so the model structurally cannot "fix" things nobody detected. This is the same discipline as giving agents scoped kubectl through an MCP server instead of a shell — the boundary lives in code, not in the prompt.

The output is a PR, not an upgrade

Everything the agent produces lands in one pull request, because agents that open PRs instead of running kubectl get review, CI, and rollback for free. A useful readiness PR body has four sections, in this order:

## Upgrade readiness: 1.31 → 1.34

**Verdict: NOT READY — 2 blockers, 5 migrations staged**

### Blockers (fix before control plane bump)
- ingress-nginx 4.9.1 < required 4.12.0 (addon-matrix)
- 3 nodes on kubelet 1.31 would hit skew limit vs 1.34

### Migrations in this PR (all cite scanner finding IDs)
- gitops/base/priority.yaml: flowcontrol v1beta3 → v1 (KUBENT-004)
- charts/api/templates/pdb.yaml: policy/v1beta1 → policy/v1 (PLUTO-012)

### Unmapped findings (human required)
- KUBENT-007: HelmRelease "legacy-cron" — source repo not found

### Manual review (no scanner coverage)
- 1.33 changelog: in-place pod resize graduated; check VPA mode

The "unmapped" and "manual review" sections are the honesty budget. An agent that only ever reports what it fixed trains reviewers to rubber-stamp; one that surfaces what it couldn't verify keeps the human doing the part only a human can do. Merging the PR migrates the manifests — the actual version bump is a separate, human-initiated change behind an approval gate, because "manifests are clean" and "upgrade now" are different decisions with different blast radii.

The cluster credentials for the scan phase should be a dedicated read-only ServiceAccount:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: upgrade-readiness-scanner
rules:
  - apiGroups: ["*"]
    resources: ["*"]
    verbs: ["get", "list"]     # no watch, no write, no secrets exec

kubent needs get/list broadly (it reads Helm release secrets to inspect rendered manifests) — but nothing here can mutate the cluster, so a scanner gone wrong wastes tokens, not uptime.

Run it on a schedule, not before the deadline

The failure mode this replaces is the quarterly panic scan. Run the whole pipeline weekly in CI — scanners, matrix check, agent mapping — and have it update a single long-lived readiness issue instead of opening PR spam when nothing changed. Readiness then becomes a trend you watch, like the drift count or the CVE backlog: new deprecated usage shows up the week someone merges it, when the author still has context, not eighteen months later when the removal ships. Since scheduling failures are the most common thing a version bump surfaces on the node side, keep the Pod Pending / FailedScheduling debugging guide close for upgrade day itself.

What this doesn't catch

Be honest about the coverage boundary. Scanners see API versions; nobody's tooling sees semantics. Feature-gate default flips, kubelet flag removals, and subtle scheduler or CSI behavior changes only show up in the upstream changelog and in a staging soak — budget a real one, with production-shaped workloads, before every minor bump. And the LLM layer inherits the scanners' blind spots: if a manifest lives outside the repos you gave it, search_repo can't find it and the finding stays UNMAPPED. That's the correct outcome — an upgrade agent that guesses is worse than a checklist, but one that drains the mechanical 80% and labels the rest turns a dreaded quarter-long project into a mergeable PR and one focused review.

---

📌 Read the latest version of this guide — plus the full library of DevOps, SRE, Kubernetes, observability & cloud-cost guides — on devtocash.com.