The Kubernetes Weekend That Humbled Me — Part 1: Just Getting a VM to Boot

I had a simple plan. Spin up a couple of VMs, run kubeadm init, join a worker, deploy nginx, post about it. That plan lasted about twenty minutes before Windows and VirtualBox decided I hadn't suffered enough that week.

This is the honest account of everything that went wrong before I even got to write a single Kubernetes command that mattered — the ghost VMs, the corrupted caches, the recovery-mode hacking, all of it. If you're trying this on a Windows laptop yourself, I want you to actually understand why each thing broke, not just copy my fix and move on. Because trust me, you'll hit your own version of these and you'll need to reason your way out.

Round 1: Multipass, or How to Meet a Ghost

Multipass looked like the easy way in. One command and you're supposed to have a VM:

multipass launch --name k8s-master --cpus 2 --memory 4G --disk 20G

It hung, then eventually gave me timed out waiting for response. Fine, I thought, maybe it's slow. I checked what Multipass thought existed versus what was actually sitting in VirtualBox:

multipass list
# Name        State    IPv4    Image
# k8s-master  Running  N/A     Ubuntu 22.04 LTS

& "C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" list vms
# "Ubuntu 2" {b03575b5-db86-4c38-9195-d689257821d4}

See the problem? Multipass is convinced it created something called k8s-master. VirtualBox, which is the thing actually doing the work underneath, has never heard that name in its life — it just has some auto-generated "Ubuntu 2". Multipass is basically a client talking to a background service that drives VirtualBox for you, and the two sides keep their own separate records of what exists. When the launch times out mid-handshake, those two records can fall out of sync, and you end up with a VM that's "running" on paper but doesn't actually correspond to anything reachable. No wonder SSH never connected — Multipass was trying to reach a machine that, as far as VirtualBox knows, doesn't exist under that name.

The only clean way out was to delete both records separately, since neither tool knows how to clean up after the other:

multipass delete k8s-master --purge

& "C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" unregistervm "b03575b5-db86-4c38-9195-d689257821d4" --delete

multipass list
& "C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" list vms

Lesson for anyone doing this: whenever Multipass acts up, don't just trust multipass list. Cross-check it against VirtualBox directly, because those two can disagree and Multipass won't tell you.

Then it got worse. On the next launch attempt:

multipass launch 22.04 --name k8s-master --cpus 2 --memory 4G --disk 20G
# launch failed: Hash of ... does not match
# launch failed: Remote "" is unknown or unreachable.

The cached base image itself was corrupted, so I wiped it:

Remove-Item -Recurse -Force "C:\ProgramData\Multipass\cache\*"
Restart-Service -Name "Multipass" -Force

And that's when I fell into the real trap — an authentication loop. Multipass suddenly wanted a passphrase it had never asked me to set:

multipass authenticate
# Please enter passphrase:
# authenticate failed: Passphrase is not set.

multipass set local.passphrase
# set failed: The client is not authenticated

You can see the chicken-and-egg there: it wants me authenticated to set the passphrase, but it wants the passphrase to authenticate me. I tried nuking every certificate I could find on both the client and daemon side:

Stop-Service -Name "Multipass" -Force
Remove-Item -Recurse -Force "C:\ProgramData\Multipass\data"
Remove-Item -Recurse -Force "$env:LOCALAPPDATA\Multipass"
Start-Service -Name "Multipass"

Still looped. At that point I stopped treating it as something I could fix and accepted it as a known Windows bug in how Multipass handles cert corruption. I abandoned Multipass entirely and went straight to driving VirtualBox myself — more manual work, but at least I could see exactly what was happening at every step instead of trusting a layer that had just proven it could quietly lie to me.

Round 2: Cloud Images and the ISO That Wouldn't Build

Without Multipass holding my hand, I needed to do what it was doing under the hood: get an Ubuntu cloud image into VirtualBox and feed it configuration on first boot.

cd C:\Users\dell\Downloads
Invoke-WebRequest -Uri "https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.vmdk" -OutFile "ubuntu-22.04-cloud.vmdk" -UseBasicParsing

Here's the thing about cloud images that nobody tells you upfront: they're not a normal installable Ubuntu. They ship stripped down and expect a cloud provider (or you, faking one) to hand them a small config package on first boot — a username, an SSH key, network settings. That package needs to arrive as a specific kind of ISO, called a cidata ISO, containing a user-data and meta-data file that cloud-init reads on startup. Without it, the image boots into basically nothing: no user account, no password, nothing to log in with.

So I went looking for mkisofs to build that ISO:

Invoke-WebRequest -Uri "https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/cdrtools/win32.zip" -OutFile "cdrtools.zip" -UseBasicParsing
# ERROR: AccessDenied

Dead link. Tried SourceForge instead:

Invoke-WebRequest -Uri "https://sourceforge.net/projects/cdrtools/files/3.01/cdrtools-3.01-win32-bin.zip/download" -OutFile "cdrtools.zip" -UseBasicParsing
Expand-Archive -Path "cdrtools.zip" -DestinationPath "C:\cdrtools"
# ERROR: End of Central Directory record could not be found

That error means the file I downloaded wasn't actually a zip — SourceForge's download link redirects through an intermediate page, and Invoke-WebRequest saved that redirect page instead of following it to the real file. Rather than keep fighting dead mirrors for a tool that just builds an ISO 9660 image, I wrote a small Python script to build the cidata.iso myself directly. Sometimes the fastest fix isn't finding the "proper" tool, it's just building the thirty lines of code that does the one thing you actually need.

Round 3: I Nearly Downloaded the Full ISO

At one point I considered giving up on cloud images completely and grabbing the full installable Ubuntu Server ISO instead:

curl.exe -L -o "ubuntu-22.04-server.iso" "https://releases.ubuntu.com/jammy/ubuntu-22.04.5-live-server-amd64.iso"

2.1 GB, and my connection at the time made that a non-starter. Stuck with the cloud image and pushed forward another way.

Round 4: Breaking In Through Recovery Mode

Even with the cidata.iso attached, VirtualBox wasn't picking it up properly, so cloud-init never ran, which meant — same problem as before — no user account existed to log in with. Since I couldn't get in through the front door, I went in through the recovery console instead.

In the VirtualBox window: start the VM, hold Shift immediately as it boots (this is what gets you into the GRUB menu instead of straight into Ubuntu), then pick "Advanced options for Ubuntu" → "Recovery mode" → "root". That drops you into a root shell with the filesystem mounted read-only, so the first thing to do is make it writable:

mount -o remount,rw /

adduser k8suser
# Password: k8s123

usermod -aG sudo k8suser

sed -i 's/PasswordAuthentication no/PasswordAuthentication yes/' /etc/ssh/sshd_config.d/60-cloudimg-settings.conf

ssh-keygen -A

service ssh restart

That ssh-keygen -A line matters more than it looks — cloud images ship without SSH host keys generated, because normally cloud-init generates them itself on first boot as part of its regular setup routine. Since cloud-init never got to run properly on this VM, those keys were just missing, and SSH can't serve connections without them.

Even after all that, SSH from Windows still failed:

ssh -p 2222 k8suser@localhost
# kex_exchange_identification: read: Connection reset

Checked the network interface and found the real problem:

ip addr show
# enp0s3: <BROADCAST,MULTICAST> mtu 1500 ... state DOWN

The interface was just sitting there down. Again, this is cloud-init's job normally — bring up networking, request a DHCP lease, the works. With it never having run, nothing had touched the NIC. Brought it up by hand:

sudo ip link set enp0s3 up
sudo dhclient enp0s3

SSH connected right after. The takeaway here: if you're working with a cloud image outside of an actual cloud, don't assume anything cloud-init normally does for you has happened. Check the interface state before you waste time debugging SSH itself.

Building the Master Node

With a working, reachable VM, I could finally get to the actual Kubernetes setup. Container runtime first:

sudo apt-get update && sudo apt-get install -y containerd
sudo mkdir -p /etc/containerd
sudo containerd config default | sudo tee /etc/containerd/config.toml
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
sudo systemctl restart containerd
sudo systemctl enable containerd

That SystemdCgroup = true line isn't cosmetic — kubelet and containerd both need to agree on which cgroup driver manages resource limits. If they don't match, you get subtle scheduling and resource-accounting problems down the line, so it's worth setting correctly now rather than debugging it later.

Swap has to go — kubelet flatly refuses to run properly with swap enabled, because its whole resource-accounting model assumes memory limits mean something real:

sudo swapoff -a
sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

Then the kernel modules and sysctl settings pods actually depend on:

sudo modprobe overlay
sudo modprobe br_netfilter

sudo tee /etc/sysctl.d/kubernetes.conf <<EOF
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF

sudo sysctl --system

overlay supports the container image layering. br_netfilter makes sure bridged traffic between pods actually gets seen by iptables rules instead of silently bypassing them. ip_forward lets the box route traffic between interfaces at all — without it, nothing about pod networking works later, no matter how good your CNI choice is.

Installing Kubernetes itself:

sudo apt-get install -y apt-transport-https ca-certificates curl gnupg

curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.29/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.29/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list

sudo apt-get update
sudo apt-get install -y kubelet=1.29.6-1.1 kubeadm=1.29.6-1.1 kubectl=1.29.6-1.1 --allow-downgrades --allow-change-held-packages
sudo apt-mark hold kubelet kubeadm kubectl

That last apt-mark hold is deliberate — a regular apt upgrade could otherwise silently bump these to a version that doesn't match across the cluster, which is exactly the kind of thing that ruins your week later for no visible reason right now.

The Disk That Was Way Too Small

First kubeadm init attempt:

sudo kubeadm init --pod-network-cidr=192.168.0.0/16 --apiserver-advertise-address=10.0.2.15 --node-name k8s-master

It died with cannot copy extracted data... failed to write (No space left on device). Checked the disk and the root partition was sitting at a laughable 2.1 GB. This traces straight back to the cloud-init issue from earlier — cloud images ship with small root partitions on purpose, because normally cloud-init's growpart module expands the partition to match whatever disk size the cloud provider actually gave you, as part of its first-boot routine. Since cloud-init never completed properly on this VM, that resize never happened, and I was stuck with a partition sized for a tiny default disk instead of the 20 GB I'd actually allocated.

Fixing it meant leaving the guest OS entirely and working from the Windows side. VMDK format can't be resized directly, so first it had to become a VDI:

& "C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" clonehd "ubuntu-22.04-cloud.vmdk" "disk.vdi" --format VDI
& "C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" modifyhd "disk.vdi" --resize 20480

Then, back in recovery mode, grow the partition and filesystem to actually use that new space:

apt-get install -y cloud-guest-utils

growpart /dev/sda 1
resize2fs /dev/sda1

df -h
# /dev/sda1  20G  1.7G  18G  9% /

Moral of this one: check df -h before you install anything on a cloud image locally, not after it fails halfway through.

Finally, a Working Control Plane

sudo kubeadm init --pod-network-cidr=192.168.0.0/16 --apiserver-advertise-address=10.0.2.15 --node-name k8s-master

It went through this time. CoreDNS did throw a timeout warning during the addon phase, but that's expected at this stage — CoreDNS pods stay stuck pending until there's an actual pod network for them to run on, and I hadn't installed one yet. The control plane itself was healthy, which was the real milestone.

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

kubectl was finally talking to something real.

That's where I'll cut Part 1 — I had one working node and a control plane that actually responded to me, after what felt like an unreasonable amount of suffering to get there. Part 2 covers the part I actually set out to do: picking a CNI that would work, adding a second node, and getting a real multi-node cluster scheduling pods across both.