DEV Community

Cover image for Kubernetes Was Not Broken. My VM Was Too Small.
Henry Osei
Henry Osei

Posted on

Kubernetes Was Not Broken. My VM Was Too Small.

#devops #kubernetes #homelab

I recently hit one of those Kubernetes issues that looks like a networking problem at first, then a CNI problem, then an RKE2 problem, and finally turns out to be something much simpler: the control-plane VM did not have enough resources. It cost me most of an evening, so I'm writing down the trail while it's fresh.

The setup was a fresh RKE2 cluster on a Proxmox VM. After installing RKE2, I checked the node status:

kubectl get nodes
Enter fullscreen mode Exit fullscreen mode 
NAME      STATUS     ROLES                AGE     VERSION
control   NotReady   control-plane,etcd   4m15s   v1.35.6+rke2r1
Enter fullscreen mode Exit fullscreen mode

Nothing alarming yet. A fresh node often sits in NotReady while the CNI installs, so I checked the system pods:

kubectl get pods -n kube-system
Enter fullscreen mode Exit fullscreen mode

The core control-plane pods were running:

etcd-control                    1/1   Running
kube-apiserver-control          1/1   Running
kube-controller-manager-control 1/1   Running
kube-proxy-control              1/1   Running
Enter fullscreen mode Exit fullscreen mode

But all the Helm installer jobs were stuck in Pending:

helm-install-rke2-canal
helm-install-rke2-coredns
helm-install-rke2-ingress-nginx
helm-install-rke2-metrics-server
Enter fullscreen mode Exit fullscreen mode

My first thought, like probably every Kubernetes person reading this, was CNI. In RKE2, Canal is the default, and I wanted Cilium anyway, so I updated the config:

write-kubeconfig-mode: "0644"
cni: cilium
node-ip: <VM_LAN_IP>
Enter fullscreen mode Exit fullscreen mode

A side lesson worth keeping even though it turned out not to be the bug: when your nodes sit on an overlay like Tailscale, set node-ip explicitly. Kubernetes will happily pick the Tailscale 100.x.x.x address when you meant the VM LAN IP, and you will discover this at the worst possible time.

After restarting RKE2, the cluster switched from Canal to Cilium. helm-install-rke2-cilium appeared, so the config change took. And then the new install job sat in Pending, exactly like everything before it. I had changed the CNI and changed nothing.

That was the turning point. If the CNI install job is pending, the CNI has not even started, so blaming Cilium makes no sense. Kubernetes had not reached the stage where Cilium could fail. Something was preventing the installer pod from being scheduled at all.

Then I noticed a line I had scrolled past at least twice:

kube-scheduler-control   0/1   Completed
Enter fullscreen mode Exit fullscreen mode

The scheduler should never be Completed. It should be running continuously:

kube-scheduler-control   1/1   Running
Enter fullscreen mode Exit fullscreen mode

Without a scheduler, Kubernetes cannot place pods, which explained every Pending job in one shot. So I described the scheduler pod:

kubectl -n kube-system describe pod kube-scheduler-control
Enter fullscreen mode Exit fullscreen mode

And there was the real clue:

UnexpectedAdmissionError
preemption: error finding a set of pods to preempt:
no set of running pods found to reclaim resources:
[(res: cpu, q: 100)]
Enter fullscreen mode Exit fullscreen mode

That tiny q: 100 was the whole story. The kubelet was trying to admit the scheduler pod and could not find 100 millicores of CPU, and there were no lower-priority pods it could evict to free them up. The machine was so starved that the control plane itself could not stabilize. All my CNI theorizing had been happening two layers above the actual problem.

So it was never Cilium, or Canal, or CoreDNS, or kube-proxy, or Tailscale, or RKE2. The VM was undersized.

After increasing the Proxmox VM resources, the cluster recovered without any further intervention from me. The scheduler stayed running, the Cilium install job completed, Cilium came up, and the node went Ready:

kube-scheduler-control          1/1   Running
helm-install-rke2-cilium        Completed
rke2-cilium                     Running
control                         Ready
Enter fullscreen mode Exit fullscreen mode

The lesson I took away: when Kubernetes says NotReady, don't jump straight to the CNI. It often is the CNI, which is exactly why the reflex exists, but first check whether the cluster can schedule anything at all. My debugging path now looks like this:

kubectl get nodes
kubectl get pods -n kube-system
kubectl describe pod -n kube-system <pending-pod>
kubectl describe node <node-name>
kubectl get events -A --sort-by='.lastTimestamp'
Enter fullscreen mode Exit fullscreen mode

If a control-plane component like kube-scheduler is not running, fix that before touching anything else.

The related lesson is that Pending is not always a pod problem. Sometimes the scheduler itself is unhealthy. My installer pods were pending because the scheduler was not running, and the scheduler was not running because the node lacked CPU. The symptom and the cause were two levels of indirection apart, which is why the first two theories felt so plausible.

For a lab cluster it is tempting to give the VM the smallest resources you can get away with. I know because that is exactly what I did. But RKE2 is a full Kubernetes distribution: etcd, API server, controller manager, scheduler, kubelet, kube-proxy, CNI, CoreDNS, metrics server, ingress controller, all running before your first workload exists. Even a single node needs headroom. My baseline now:

Minimum:
2 vCPU
4 GB RAM
30-50 GB disk

Better:
4 vCPU
8 GB RAM
50 GB+ disk
Enter fullscreen mode Exit fullscreen mode

The visible error was Node NotReady. My first assumption was a missing CNI. The actual problem was a scheduler that could not stay running on a starved node. Kubernetes was fine, Cilium was fine, RKE2 was fine, and the fix was a slider in the Proxmox UI. Somehow that makes it worse.

Top comments (0)