DEV Community

Florient DOGBE
Florient DOGBE

Posted on

Kubernetes 1.33: What Really Breaks During an Upgrade and How to Anticipate It

Every cluster operator remembers Kubernetes 1.16 and 1.22: whole API groups disappeared overnight, Helm charts exploded, and kubectl apply started returning no matches for kind. Since then, teams built reflexes around deprecated API scanning, and that is good. The problem with Kubernetes 1.33 is that those reflexes are no longer sufficient.

1.33 removes no major GA API. If your upgrade plan is "run pluto, see zero removed APIs, click upgrade", you will pass the audit and still break the cluster — because in 1.33 the breakage moved to three other places: feature gates that no longer exist, kubelet default behaviour changes, and aggregated APIService endpoints that silently rot. This article builds a practical upgrade plan covering all four surfaces, with pluto and kubent for the API audit and a reproducible pre-production validation loop.

Why the failure modes shifted

Kubernetes has a predictable lifecycle policy: a feature gate that reaches GA stays recognized for two or three more minor releases, then the flag is removed from the binary. A removed gate is not ignored — the component refuses to start:

E0415 10:12:04.118312 1 run.go:74] "command failed" err="unrecognized feature gate: MinDomainsInPodTopologySpread"
Enter fullscreen mode Exit fullscreen mode

That is a crash loop on kube-apiserver or kubelet, on every node where the stale flag is baked into /var/lib/kubelet/config.yaml, a kubeadm ClusterConfiguration, or a Terraform/Ansible variable nobody has touched since 1.28. kubeadm upgrade plan does not audit your feature gates, and neither does pluto or kubent. This is the single most common way a 1.32 → 1.33 upgrade turns into an incident.

The second shift: the kubelet changed defaults. gitRepo volumes are disabled by default in 1.33 (behind the GitRepoVolumeDriver gate, off), kube-proxy nftables mode is GA, user namespaces (hostUsers: false) is beta and on by default, in-place pod resize is beta, and sidecar containers (initContainers with restartPolicy: Always) are GA. None of these are "removals" — they are behaviour changes that your workloads or your node images may depend on.

The third shift: v1.Endpoints is now formally deprecated in favour of EndpointSlice. Nothing breaks today, but any controller or service mesh still writing Endpoints objects is on borrowed time, and 1.33 is the right release to inventory them.

Step 1 — Audit the API surface with pluto and kubent

Both tools answer "which manifests reference an API version that is deprecated or removed in my target version", but they look in different places, so run both.

pluto: manifests, Helm releases, live objects

# install (Linux amd64)
curl -sSL https://github.com/FairwindsOps/pluto/releases/download/v5.21.6/pluto_5.21.6_linux_amd64.tar.gz \
  | tar -xz -C /usr/local/bin pluto

# 1. static manifests in Git (the source of truth you can actually fix)
pluto detect-files -d ./deploy --target-versions k8s=v1.33.0 -o wide

# 2. Helm 3 releases stored as Secrets in the cluster
pluto detect-helm --target-versions k8s=v1.33.0 -o markdown

# 3. everything the API server still serves, including CRDs
pluto detect-all-in-cluster --target-versions k8s=v1.33.0 -o wide
Enter fullscreen mode Exit fullscreen mode

pluto exits non-zero when it finds something (2 for deprecated, 3 for removed), which makes it a one-line CI gate:

pluto detect-files -d ./deploy --target-versions k8s=v1.33.0 --ignore-deprecations
# fail the pipeline only on APIs already removed in the target version
Enter fullscreen mode Exit fullscreen mode

kubent: live cluster plus Helm, with a target version

sh -c "$(curl -sSL https://git.io/install-kubent)"

kubent --target-version 1.33.0 --exit-error --output json | tee kubent-1.33.json

# scan rendered manifests before they ever reach the cluster
helm template my-app ./chart | kubent --cluster=false --filename - --target-version 1.33.0
Enter fullscreen mode Exit fullscreen mode

A key detail both tools share: they read the kubectl.kubernetes.io/last-applied-configuration annotation to recover the API version a client used, not the one the API server stores. Objects created by controllers or by kubectl create without that annotation will be invisible. That is why the next step matters more than the scanners.

Step 2 — Audit real traffic, not just stored objects

The API server exposes exactly which deprecated APIs are being requested, and in which release they disappear:

kubectl get --raw /metrics | grep apiserver_requested_deprecated_apis
# apiserver_requested_deprecated_apis{group="",removed_release="1.40",resource="endpoints",subresource="",version="v1"} 1
Enter fullscreen mode Exit fullscreen mode

Joined with request counts in Prometheus, you get the offending clients:

sum by (group, version, resource, removed_release, user_agent) (
    increase(apiserver_request_total[7d])
  * on (group, version, resource, subresource) group_left(removed_release)
    apiserver_requested_deprecated_apis
)
Enter fullscreen mode Exit fullscreen mode

If you do not have Prometheus on the control plane, add a narrow audit policy for the duration of the upgrade window:

apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
  - RequestReceived
rules:
  - level: Metadata
    resources:
      - group: ""
        resources: ["endpoints"]
      - group: "flowcontrol.apiserver.k8s.io"
      - group: "resource.k8s.io"
  - level: None
Enter fullscreen mode Exit fullscreen mode

On OpenShift, you get this for free and historically — use it:

oc get apirequestcounts -o custom-columns=\
NAME:.metadata.name,REMOVED:.status.removedInRelease,LASTHOUR:.status.requestCount \
  | awk '$2 != "<none>"'

oc get apirequestcount endpoints.v1 \
  -o jsonpath='{range .status.currentHour.byNode[*]}{.byUser[*].username}{"\n"}{end}' | sort -u
Enter fullscreen mode Exit fullscreen mode

Step 3 — Audit feature gates, flags and APIServices

This is the part no scanner does for you.

Compare your configured gates against the 1.33 binary

# list every gate the 1.33 API server still knows about
docker run --rm registry.k8s.io/kube-apiserver:v1.33.1 \
  kube-apiserver --help 2>&1 | sed -n '/--feature-gates/,/^ *--/p' > gates-1.33.txt

# what your control plane is actually configured with
kubectl -n kube-system get cm kubeadm-config -o yaml | grep -A20 featureGates
grep -o 'feature-gates=[^ ]*' /etc/kubernetes/manifests/kube-apiserver.yaml

# and on every node
grep -i featureGates -A10 /var/lib/kubelet/config.yaml
Enter fullscreen mode Exit fullscreen mode

Any gate present in your config and absent from gates-1.33.txt is a guaranteed crash loop. In practice, look hard at gates that reached GA around 1.30 (MinDomainsInPodTopologySpread, NodeOutOfServiceVolumeDetach, NewVolumeManagerReconstruction, CloudDualStackNodeIPs, StableLoadBalancerNodeSet, LegacyServiceAccountTokenCleanUp, KubeProxyDrainingTerminatingNodes…) — the GA + 3 releases rule lands precisely on 1.33.

You can also ask a running component what it has enabled:

kubectl get --raw /metrics | grep '^kubernetes_feature_enabled' | grep 'stage="ALPHA"'
Enter fullscreen mode Exit fullscreen mode

Clean up the kubeadm configuration first

Fix the configuration before running the upgrade, using the v1beta4 API (v1beta3 is deprecated):

apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
kubernetesVersion: v1.33.1
featureGates:
  ControlPlaneKubeletLocalMode: true
apiServer:
  extraArgs:
    - name: audit-log-path
      value: /var/log/kubernetes/audit.log
    - name: audit-policy-file
      value: /etc/kubernetes/audit-policy.yaml
  extraVolumes:
    - name: audit-policy
      hostPath: /etc/kubernetes/audit-policy.yaml
      mountPath: /etc/kubernetes/audit-policy.yaml
      readOnly: true
      pathType: File
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd
# no stale featureGates block here
Enter fullscreen mode Exit fullscreen mode

Note ControlPlaneKubeletLocalMode: in 1.33 it is beta and on by default, making a joining control-plane node's kubelet talk to its local API server instead of the load balancer. If your LB health checks or firewall rules assumed otherwise, test it.

Do not ignore aggregated APIServices

An APIService whose backing Deployment is gone or whose CA bundle expired makes discovery fail. kubectl get all starts printing errors, controllers that enumerate resources stall, and kubeadm upgrade preflight can hang. Check before and after:

kubectl get apiservices -o custom-columns=\
NAME:.metadata.name,AVAILABLE:'.status.conditions[?(@.type=="Available")].status',\
REASON:'.status.conditions[?(@.type=="Available")].reason' | grep -v True
Enter fullscreen mode Exit fullscreen mode

A healthy aggregated API server declaration looks like this — explicit CA bundle, TLS verification on:

apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  name: v1beta1.metrics.k8s.io
spec:
  group: metrics.k8s.io
  version: v1beta1
  groupPriorityMinimum: 100
  versionPriority: 100
  insecureSkipTLSVerify: false
  caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0t...   # base64 PEM, rotate with your CA
  service:
    name: metrics-server
    namespace: kube-system
    port: 443
Enter fullscreen mode Exit fullscreen mode

Metrics-server, Prometheus adapter, cert-manager's cainjector outputs, Kyverno's reporting APIs and KEDA all register APIService objects. Bump them to versions that declare support for 1.33 before touching the control plane.

Step 4 — Validate in pre-production

Respect the skew rules

kubeadm upgrades one minor version at a time: 1.31 → 1.32 → 1.33. kubelet may be up to three minor versions older than the API server, kubectl one in either direction, and kube-proxy must never be newer than the API server. Upgrade control plane first, then nodes.

Rehearse on a disposable cluster

# snapshot etcd before anything
kubectl -n kube-system exec etcd-cp-1 -- etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /var/lib/etcd/pre-1.33.db

kubeadm certs check-expiration
kubeadm config images list --kubernetes-version v1.33.1

kubeadm upgrade plan v1.33.1
kubeadm upgrade apply v1.33.1 --dry-run
kubeadm upgrade apply v1.33.1
Enter fullscreen mode Exit fullscreen mode

Then the nodes, one at a time:

kubectl drain node-01 --ignore-daemonsets --delete-emptydir-data --timeout=300s
# upgrade kubeadm/kubelet/kubectl packages, then:
kubeadm upgrade node
systemctl daemon-reload && systemctl restart kubelet
kubectl uncordon node-01
kubectl get nodes -o wide
Enter fullscreen mode Exit fullscreen mode

Post-upgrade smoke tests that catch 1.33-specific regressions:

# kubeProxyVersion is no longer populated in 1.33 — fix any tooling that parses it
kubectl get nodes -o jsonpath='{.items[*].status.nodeInfo.kubeProxyVersion}'

# gitRepo volumes are off by default
kubectl get pods -A -o json | jq -r '.items[] | select(.spec.volumes[]?.gitRepo) | "\(.metadata.namespace)/\(.metadata.name)"'

# cgroup v1 nodes are in maintenance mode: plan the migration
kubectl get nodes -o json | jq -r '.items[].status.nodeInfo.osImage' | sort -u
stat -fc %T /sys/fs/cgroup/   # cgroup2fs expected

# in-place resize is beta in 1.33
kubectl patch pod demo --subresource=resize \
  --patch '{"spec":{"containers":[{"name":"app","resources":{"requests":{"cpu":"200m"}}}]}}'
Enter fullscreen mode Exit fullscreen mode

Make the audit continuous rather than a one-off ritual:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: api-deprecation-scan
  namespace: platform-tools
spec:
  schedule: "0 6 * * 1"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      backoffLimit: 1
      template:
        spec:
          serviceAccountName: deprecation-scanner
          restartPolicy: Never
          securityContext:
            runAsNonRoot: true
            runAsUser: 65532
            seccompProfile:
              type: RuntimeDefault
          containers:
            - name: pluto
              image: us-docker.pkg.dev/fairwinds-ops/oss/pluto:v5.21.6
              args: ["detect-all-in-cluster", "--target-versions", "k8s=v1.33.0", "-o", "json"]
              securityContext:
                allowPrivilegeEscalation: false
                readOnlyRootFilesystem: true
                capabilities:
                  drop: ["ALL"]
Enter fullscreen mode Exit fullscreen mode

Security best practices

Give the scanners read-only, scoped access. pluto detect-helm and kubent read Helm release data stored in Secrets. A ClusterRole with get/list on secrets cluster-wide is a serious privilege. Scope it where you can, and never pipe scanner output to an external SaaS without reviewing it:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: deprecation-scanner
rules:
  - apiGroups: ["*"]
    resources: ["*"]
    verbs: ["get", "list"]
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: []          # grant per-namespace only if you need detect-helm
Enter fullscreen mode Exit fullscreen mode

Pin control plane images by digest and verify signatures with cosign verify --certificate-identity-regexp '.*' registry.k8s.io/kube-apiserver:v1.33.1 before promoting them into your internal registry. An upgrade window is a perfect time for a supply-chain mistake.

Keep audit logging on during the whole upgrade. The deprecation audit policy above is also your forensic trail if something goes wrong.

Rotate and verify certificates. kubeadm certs check-expiration before every upgrade; kubeadm upgrade apply renews control plane certs by default, which is a feature, not a reason to skip the check.

Use admission policy to stop the drift from coming back. Once you have eliminated Endpoints writes, enforce it:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: no-manual-endpoints
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["endpoints"]
  validations:
    - expression: "request.userInfo.username.startsWith('system:')"
      message: "Endpoints is deprecated since 1.33; use EndpointSlice."
Enter fullscreen mode Exit fullscreen mode

Treat 1.33 defaults as hardening opportunities. User namespaces are beta and enabled by default: adding hostUsers: false to a pod spec maps container UID 0 to an unprivileged host UID, which neutralises a whole class of container escapes on kernels ≥ 6.3. Test it on non-critical workloads during the same window.

Key takeaways

  • 1.33 removes no major GA API, so a clean pluto/kubent report proves almost nothing on its own. Run them anyway — on Git manifests, Helm releases and the live cluster — and wire the exit codes into CI.
  • The real 1.33 killers are removed feature gates in kube-apiserver and kubelet configuration. Diff your configured gates against the 1.33 binary's --help output before you touch anything.
  • Use apiserver_requested_deprecated_apis (or apirequestcounts on OpenShift) to find clients, not just stored objects. Deprecated Endpoints usage is your 1.33 inventory task.
  • Check APIService availability before and after; a broken aggregation layer breaks discovery for every controller in the cluster.
  • Behaviour changes to validate explicitly: gitRepo off by default, kubeProxyVersion no longer populated, nftables proxy GA, sidecar containers GA, in-place resize and user namespaces beta, cgroup v1 in maintenance mode.
  • Rehearse the full kubeadm upgrade plan → --dry-run → apply → upgrade node sequence in pre-production, with an etcd snapshot and a documented rollback, one minor version at a time.

An upgrade plan built on these four audit surfaces — APIs, gates, kubelet behaviour, aggregation layer — turns a 1.33 migration from a gamble into a maintenance task.

Top comments (0)