DEV Community: Nadav Erell

Kubernetes 1.37 Breaking Changes: What to Check Before You Upgrade

Nadav Erell — Fri, 28 Aug 2026 13:57:52 +0000

# kubelet configuration - added for sidecar support, harmless ever since
featureGates:
  SidecarContainers: true

Harmless until Kubernetes 1.37. The feature went GA in 1.33 and the gate has been a no-op even longer - and 1.37 removes it. A kubelet that finds a removed gate in its config doesn't warn, it refuses to start. On upgrade day that looks like new nodes going NotReady over a line nobody has touched since it was added. The same applies to a control-plane component carrying it in --feature-gates.

SidecarContainers is one of 25 gates removed in 1.37. Another 13 get locked to fixed values, and an explicit setting that contradicts a locked default fails the same way.

Kubernetes 1.37 was released on August 26, 2026. The gate removals are the sharpest edge, but not the only one: the release also drops 18 kubelet flags, kubeadm's v1beta3 config format, and an alpha scheduling API, removes a batch of metrics, changes what eventRecordQPS: 0 means, and starts the kube-proxy move from iptables to nftables. The feature side of the release - KYAML output for kubectl, pod-level resources, and DRA device taints and tolerations going stable, among the 16 graduations - is covered in the official v1.37 announcement; this post is about the upgrade side. All of it is in the changelog. The work is figuring out which of it exists in your cluster.

This post walks through each change and how to check for it by hand. If you'd rather not do that, Radar has a dedicated view for exactly this - Upgrade impact - which runs every one of these checks against your live cluster and reports what it found per change (how to run it). Here it is on a GKE cluster:

What breaks in Kubernetes 1.37

The short version:

Change in 1.37	What happens on upgrade
25 feature gates removed, 13 locked to defaults	Component rejects the setting and fails to start
`--concurrent-service-syncs`, `PodGroupWorkloadExists` admission plugin removed	Component fails to start
18 kubelet cAdvisor flags removed	Kubelet fails to start
Static Pods referencing Secrets or ConfigMaps	The static Pod cannot start
`scheduling.k8s.io/v1alpha2` Workload and PodGroup dropped	Stored objects must be deleted before the upgrade
kubeadm v1beta3 configuration API removed	kubeadm rejects the stored config
`eventRecordQPS: 0` semantics change	Event rate limit silently lifts from 5/s to unlimited
SELinux mount-time labeling expands	Conflicting labels on shared volumes can leave Pods in ContainerCreating
API-server, DRA, and cAdvisor metric changes	Alerts and dashboards silently stop matching data
kube-proxy implicit mode	Warnings start in 1.37; the Linux default flips to nftables in 1.40

The first six rows are the ones to clear before upgrading: they can keep a component from starting or stop the upgrade itself. The rest change runtime behavior or monitoring, or set up work for 1.40.

Feature gates, component flags, and kubeadm config removed in 1.37

Start with the gates, because they're the likeliest thing to actually be in your cluster. 1.37 removes 25 feature gates and locks 13 more to their defaults; the exact diff is in the official feature lifecycle lists. A component started with a removed gate, or a locked gate set to a non-default value, refuses to start.

The removed set includes gates people actually set by hand: SidecarContainers, JobSuccessPolicy, PodLifecycleSleepAction, SchedulerQueueingHints. These sit in --feature-gates arguments and kubelet featureGates blocks for years because nothing ever complains about them. Until the release that removes them.

To check a node by hand, read its effective kubelet config through the API:

kubectl get --raw "/api/v1/nodes/$NODE/proxy/configz" \
  | jq '.kubeletconfig | {featureGates, eventRecordQPS}'

Two more removals fail the same way: kube-controller-manager's --concurrent-service-syncs flag, and the PodGroupWorkloadExists admission plugin. The plugin is rejected whether it appears in --enable-admission-plugins or --disable-admission-plugins - yes, disabling a plugin that no longer exists also keeps the API server from starting.

One removed gate carries a behavior change with it: PreventStaticPodAPIReferences is gone, and with it the opt-out that let static Pods reference Secrets and ConfigMaps. A static Pod that still does gets its node configuration rejected on 1.37 and cannot start.

In Radar's Upgrade impact report, the gate check reads every node's configz and every control-plane Pod's arguments (including split command/args forms) and names the exact gate on the exact node.

18 kubelet cAdvisor flags are removed

The kubelet loses 18 cAdvisor flags on top of that (--enable-load-reader, --containerd, the whole --storage-driver-* family; --housekeeping-interval survives). These are harder to audit than the gates. They live in systemd units and bootstrap scripts on the node itself, where no Kubernetes API can see them, so checking by hand means a pass over every node's kubelet startup arguments.

Radar can't see your systemd units either - nothing with only API access can - so instead of pretending the check passed, it hands you the list of 18 flags to grep for on your nodes.

kubeadm v1beta3 is gone

On kubeadm clusters, the v1beta3 config format is gone too. If kubeadm-config in kube-system still holds v1beta3 documents, run kubeadm config migrate with a pre-1.37 binary before upgrading; kubeadm 1.37 won't accept the old format even to migrate it. The PublicKeysECDSA and NodeLocalCRISocket kubeadm gates go with it.

scheduling.k8s.io/v1alpha2: delete stored objects before you upgrade

If you never enabled the alpha scheduling gates, skip this section. It doesn't apply to you.

For everyone else: 1.37 stops serving scheduling.k8s.io/v1alpha2, the alpha gang-scheduling API, as Workload and PodGroup promote to v1beta1. The trap is the order of operations. Delete the stored v1alpha2 objects before the upgrade, then recreate them as v1beta1 after the control plane reaches 1.37 - and expect to update the manifests to the v1beta1 schema, not just the apiVersion line. Upgrade first and the stored objects block you. Recreate too early and v1beta1 isn't served yet.

Worth saying clearly: this is the only served API that 1.37 removes. Deployments, StatefulSets, Services - manifests that were valid on 1.36 remain valid on 1.37. (kubeadm's v1beta3 above is a config-file format, not an API the cluster serves.)

eventRecordQPS: 0 now means unlimited

This one changes silently, which is why I'd check it even if you're confident about everything else.

# kubelet configuration, valid on 1.36 and 1.37 - with different meanings
eventRecordQPS: 0

On 1.36, that zero doesn't mean what it looks like: it reaches client-go, which quietly substitutes its own fallback of 5 events per second. On 1.37, zero means zero throttling. The kubelet starts fine. Nothing warns. Every node carrying the setting just starts recording events as fast as it produces them. This was the one that surprised us most going through the changelog - nothing in the kubelet's own configuration surface hints that the zero was being overridden downstream.

The upstream recommendation is an explicit 50, the normal kubelet default - or 5, if you were deliberately relying on the old fallback. The configz command above shows the live value per node, and Radar flags every node whose effective config carries the setting.

Removed metrics: cAdvisor series, DRA renames, API-server cache stats

None of the metric changes will block your upgrade. What they'll do is quietly break the dashboards and alerts you'd use to notice everything else on this page.

1.37 replaces the API-server cache-list metrics with apiserver_storage_list_*{storage="watchcache"} equivalents, renames DRA metrics, and drops several cAdvisor series: container_cpu_load_average_10s, container_cpu_load_d_average_10s, container_tasks_state, the custom container_application_* metrics, and userDefinedMetrics in the kubelet's /stats/summary.

The failure mode is silence. A PrometheusRule referencing a removed metric doesn't error; the expression just stops matching series, and the alert can never fire again. Grep your rules for the removed cAdvisor and cache-list names; the DRA renames are in the changelog:

kubectl get prometheusrules -A -o yaml \
  | grep -nE 'container_tasks_state|container_cpu_load(_d)?_average_10s|container_application_|apiserver_cache_list_'

Radar runs the same check against every PrometheusRule expression in the cluster and points at the exact expression and metric token. Where upstream removed a metric without a replacement, the finding says so rather than inventing one.

kube-proxy mode: warnings in 1.37, nftables default in 1.40

Nothing about how kube-proxy moves traffic changes in 1.37. What starts is the countdown: the implicit Linux default flips from iptables to nftables in 1.40 (KEP-5343), and IPVS mode begins a staged deprecation - expected disabled by default in 1.40 and removed in 1.43 (KEP-5495). So 1.37 starts warning wherever the proxy mode is implicit.

That's the finding expanded in the screenshot at the top - "Linux proxy mode is not explicit", on the kube-system/kube-proxy DaemonSet, with both KEPs linked. The fix is one line: set the mode explicitly, iptables or nftables, after validating whichever backend you pick. Set it now and there's nothing left to do when 1.40 arrives.

SELinux mount-time labeling applies to more volumes

This one only matters on SELinux-enforcing nodes. 1.37 makes more volume shapes eligible for mount-time labeling instead of recursive relabeling. Mounts get faster. But a volume shared by Pods with conflicting SELinux labels stops being slow-but-tolerated: the kubelet can mount it with only one context, and a Pod requesting a different label can hang in ContainerCreating.

The knobs are explicit. Align the SELinux label across every Pod sharing a volume, or set seLinuxChangePolicy: Recursive in the Pod's securityContext to keep the old behavior for that Pod. You can also defer the whole transition by setting the SELinuxMount feature gate to false on every kubelet, but that opt-out works for 1.37 only; the gate is expected to lock enabled in 1.38.

Radar warns on conflicts it can actually observe (kubelet mismatch metrics, SELinuxLabelConflict Events) and flags shared-volume setups that could conflict for review.

Upgrade impact: check your cluster for 1.37 in one command

Everything above is public knowledge - most of this list came out of the changelog pass we did while updating Radar's checks for 1.37. What no changelog can tell you is whether your cluster has any of it: a locked gate in one node pool's kubelet config, a v1alpha2 object left over from an experiment, or a container_tasks_state alert nobody has looked at in two years.

That's what the Upgrade impact view is for. It ships in open-source Radar, with the check catalog updated for 1.37:

curl -fsSL https://get.radarhq.io | sh && radar

Then open Checks in the sidebar, switch to the Upgrade impact tab, pick 1.37 as the target, and find out whether SidecarContainers: true is still set somewhere in your fleet.

A few things about how the report behaves, because they decide whether you can act on it without re-verifying everything yourself.

Radar checks the upgrade path, not just the target

Each check is bound to the release that makes it relevant, and only the ones your upgrade path crosses run. A cluster on 1.35 targeting 1.37 gets the 1.36 boundary checked too; a cluster already on 1.37 isn't nagged about gates it can no longer have. The release-specific checks land on top of the standing ones in the screenshot: admission webhook readiness, CRD conversion webhook readiness, node drain feasibility, removed API usage.

Every finding includes its evidence

Not "you may be affected by the cAdvisor removals" but the specific field on the specific object: a configz path, a container argument, a stored object's apiVersion, an expression inside a PrometheusRule. A blocker means Radar actually observed configuration or stored API that will reject startup or the upgrade. Warnings cover observed behavioral and monitoring impact. Review is everything that needs a human: the kubelet flags Radar can't read, the transitions you have to schedule.

Unknown is not green

"Couldn't check" is never reported as a pass. When Radar can't inspect a surface, the result is Unknown, not green. On the GKE cluster in the screenshot, control-plane checks that need mirror Pods come back N/A, provider-managed: Google owns that surface, and you couldn't fix a finding there anyway. The same absence on a self-managed cluster is Unknown, because there the surface is yours. It's the same reasoning that shaped Reachability's verdict semantics, and it's why the kubelet-flags check above gives you a grep list instead of a checkmark.

One more note for GKE, EKS, and AKS: on release channels, the control plane moves to 1.37 on the provider's schedule over the coming months, not yours. The provider-managed rows are their problem. The kubelet config, the workloads, the PrometheusRules, and the kube-proxy mode are still yours.

Scripting it: the headless scan and MCP

If you'd rather stay in the terminal, the same scan is an HTTP endpoint on the same binary. This installs Radar, starts it headless against your current kubectl context, and prints every non-passing check for the 1.37 path:

curl -fsSL https://get.radarhq.io | sh   # skip if already installed
radar --no-browser &
curl -sf --retry 40 --retry-delay 3 --retry-all-errors \
  "http://localhost:9280/api/upgrade-readiness?target=1.37" -o scan.json
jq -r '"\(.currentVersion) -> \(.targetVersion)  verdict: \(.verdict)  (\(.summary.blocked) blocked, \(.summary.warnings) warnings, \(.summary.reviews) review)",
  (.checks[] | select(.status | IN("passed","not_applicable") | not)
    | "[\(.status)] \(.title) - \(.summary)")' scan.json

The retries cover startup and the first cache sync. Drop the target parameter to scan against whatever your cluster's next minor is. The full report - evidence paths, remediation, upstream references - lands in scan.json, and the process you just started is also the full UI at localhost:9280.

The same scan is exposed over MCP, so an AI agent can run the readiness pass and read the same evidence-backed findings (MCP for Kubernetes covers the pattern). The upgrade impact docs have the full check catalog, coverage semantics, and the read-only RBAC Radar needs.

However you run the checks, run them before the control plane moves. Many of these issues are quick fixes once you know where they are - a kubelet setting, a deleted object, an updated expression. The expensive version is discovering the same list on upgrade day, one NotReady node at a time.

[Boost]

Nadav Erell — Mon, 17 Aug 2026 08:23:23 +0000

Daria Dovzhikova

Aug 13

The 3am Test: Autonomy Levels for AI Agents in Kubernetes Operations

#kubernetes #ai #devops #sre

1 min read

Are AI agents going to kill dashboards? Let me know what you think.

Nadav Erell — Tue, 28 Jul 2026 18:48:15 +0000

Daria Dovzhikova

Jul 28

We Build a Kubernetes Dashboard. AI Agents Might Make It Obsolete.

#kubernetes #devops #ai #opensource

8 min read

DevOps engineers and anyone using Kubernetes - how are you connecting your agents to your production clusters?

Nadav Erell — Tue, 21 Jul 2026 10:19:16 +0000

Daria Dovzhikova

Jul 21

We benchmarked an AI agent on 52 broken clusters: kubectl vs a Kubernetes MCP server

#kubernetes #ai #mcp #devops

4 min read

Why we stopped using Lens and built a new OSS K8s IDE/Dashboard

Nadav Erell — Thu, 05 Feb 2026 19:22:40 +0000

And why you should take 15 seconds to try it yourself :)

The "Good Old Days" of K8s Debugging

If you've been working with Kubernetes for a few years, you probably remember when Lens was the undisputed king of local development. It was fast, it was free, and it just worked. It was just so nice to not have to always resort to plain kubectl.

Of course, things haven't stayed that way...

Lens got acquired and is no longer open source. It also got bloated and slow, and requires you to log in.

Alternatives and forks like Freelens and Headlamp exist, but they're just... ok. Not very feature rich, somewhat plain UI, awkward setup.

K9s is great sometimes, for some people... but the terminal experience is not for everyone, and it's still limiting for lots of things.

Kubernetes Dashboard was another option that got some traction as an official tool, but it was never exceptional, and now that's shut down too.

So our team at Skyhook got frustrated and one day asked: what if we just built the tool we ourselves would want to use? We've already been doing related pieces of this as part of the main Skyhook platform.

So without further ado:

Meet Radar

Radar is our attempt to fix the local K8s development experience.

We had a few design rules:

Local-First: No cloud logins. No data leaving the machine.
Zero Config: It uses your existing ~/.kube/config. It should just work.
Fast: We hate waiting on slow tools.
Real-Time: No "Refresh" buttons. If a pod dies, I want to see it turn red instantly.

How we built it (The Tech Stack)

Instead of wrapping a heavy web app in Electron, we built Radar as a single Go binary with a React frontend embedded inside (using go:embed).

To get that "instant" feel, we avoided the standard API polling. Instead, we use Kubernetes SharedInformers—the same machinery that K8s controllers use internally.

We open a persistent connection to the API server.
We watch for deltas (add/update/delete events).
We push those changes to the browser frontend via Server-Sent Events (SSE).

Now everything is super fast, updates in real time. Just works.

Cool things you can do

Beyond the table-stakes resource browsing, we added features specifically for the "what is going on?" debugging moments:

Topology Graph: We use a structured DAG layout (not a bouncy physics graph) so you can actually see ownership chains and ingress paths.
GitOps Native: If you use Flux or ArgoCD, Radar shows their sync status and common operations right alongside the workloads they manage. No plugins required.
Helm Dashboard: View, manage, search and install new charts. See their health, upgrade versions.
Network Visualization: See real data flows across your microservices and in/out of your cluster, with support for Hubble (for Cilium) and Caretta.
Timeline: Because sometimes you really need to track what happened, not just the current state. See events and resource changes over time, grouped by connected resources.
Image Filesystem Viewer: A nice hidden gem - browse the files inside a container image directly from the UI without exec-ing in.

Try it out

It’s open source (Apache 2.0), no surprises.

Quick install (15 seconds to install and see live dashboard - yes we timed it):

curl -fsSL skyhook.io/get-radar.sh | bash && kubectl radar

Or install with brew, krew, scoop, or direct download - check out the repo: github.com/skyhook-io/radar

We built this because we needed it, but we’re hoping it helps other devs who are tired of fighting their tools. Let me know what you think!