[Lab Notes] Kubernetes the Hard Way, For Real This Time (Step 03)

Continuing my Kubernetes the Hard Way homelab build on Proxmox. Prerequisites and jumpbox setup (steps 01-02) are already done, this covers step 03 only.

Steps 01 and 02 were straightforward. Well, this one was a reminder that I'm doing this manually. No scripts or AI to automate all of the setup for me. Just me and the guide. And man, even if you have a guide but if you needed to tweak something in your setup. A whole bunch of problems pop up. It goes something like:

"Guide says do it this way, but I'm using a different tool, so I do it this way instead. Now something's broken because that tool needed extra config. Now the next step in the guide needs adjusting too."

It's like a row of light switches wired together. Flip one, and a few others flip with it. Now you've got to figure out which ones to switch back. But hey, thats exactly how Ops/Infra work is done(to me at least).

Anyway, here's how it went.

Machine database

Created machines.txt on the jumpbox, listing IP, FQDN, hostname, and pod subnet for each node:

root@luger-VirtualBox:~/kubernetes-the-hard-way# cat machines.txt
XXX.XXX.XXX.XXX k8s-solo.home.lab k8s-solo
XXX.XXX.XXX.XXX k8s-solo-w1.home.lab k8s-solo-w1 XXX.XXX.XXX.XXX/24
XXX.XXX.XXX.XXX k8s-solo-w2.home.lab k8s-solo-w2 XXX.XXX.XXX.XXX/24

(IPs are masked cuz I'm this 🤏 close to wearing a tin foil hat... even if they're internal IPs)

SSH access

Enabling root login: the guide assumes Debian, where root SSH login just needs to be turned on. On Ubuntu, root login is disabled by default too, so the fix was the same. Just edit sshd_config and restart the service:

ubuntu@k8s-solo:~$ sudo sed -i 's/^#*PermitRootLogin.*/PermitRootLogin yes/' /etc/ssh/sshd_config
ubuntu@k8s-solo:~$ sudo systemctl restart sshd

Generating and distributing keys: I originally planned to use root like the guide suggests, but kept hitting a Please login as the user "ubuntu" rather than the user "root" prompt on every node. I didn't bother digging into it right away, so it wasn't until the end of this step that I actually tracked down why. Ubuntu's cloud images bake a forced command into /root/.ssh/authorized_keys that blocks direct root SSH login and prints that message instead. Rather than fight it, I just switched to the ubuntu user for all admin access. So keys were copied to ubuntu@ on each node instead of root@. Also used a dedicated keypair instead of overwriting any existing id_rsa:

root@luger-VirtualBox:~/kubernetes-the-hard-way# ssh-keygen
Enter file in which to save the key (/root/.ssh/id_rsa): /root/.ssh/id_rsa_kthw

root@luger-VirtualBox:~/kubernetes-the-hard-way# while read IP FQDN HOST SUBNET; do
  ssh-copy-id -i /root/.ssh/id_rsa_kthw.pub ubuntu@${IP}
done < machines.txt

Verified access to all three nodes:

root@luger-VirtualBox:~/kubernetes-the-hard-way# while read IP FQDN HOST SUBNET; do
  ssh -n ubuntu@${IP} hostname
done < machines.txt
k8s-solo
k8s-solo-w1
k8s-solo-w2

Hostnames

The guide sets each machine's hostname to match its role: server, node-0, node-1. I updated machines.txt accordingly and ran the rename commands against the ubuntu user instead of root:

root@luger-VirtualBox:~/kubernetes-the-hard-way# while read IP FQDN HOST SUBNET; do
    CMD="sudo sed -i 's/^127.0.1.1.*/127.0.1.1\t${FQDN} ${HOST}/' /etc/hosts"
    ssh -n ubuntu@${IP} "$CMD"
    ssh -n ubuntu@${IP} sudo hostnamectl set-hostname ${HOST}
    ssh -n ubuntu@${IP} sudo systemctl restart systemd-hostnamed
done < machines.txt

Confirmed:

root@luger-VirtualBox:~/kubernetes-the-hard-way# while read IP FQDN HOST SUBNET; do
  ssh -n ubuntu@${IP} hostname --fqdn
done < machines.txt
server.kubernetes.local
node-0.kubernetes.local
node-1.kubernetes.local

From this point on, the nodes are addressed as server, node-0, and node-1 per the guide's convention. Not the earlier k8s-solo / k8s-solo-w1 / k8s-solo-w2 names.

Host lookup table

No surprises here, followed the guide as-is. Built a hosts file on the jumpbox and appended it to the jumpbox's own /etc/hosts:

root@luger-VirtualBox:~/kubernetes-the-hard-way# echo "# Kubernetes The Hard Way" >> hosts
root@luger-VirtualBox:~/kubernetes-the-hard-way# while read IP FQDN HOST SUBNET; do
    echo "${IP} ${FQDN} ${HOST}" >> hosts
done < machines.txt
root@luger-VirtualBox:~/kubernetes-the-hard-way# cat hosts >> /etc/hosts

Verified hostname resolution works locally:

root@luger-VirtualBox:~/kubernetes-the-hard-way# for host in server node-0 node-1; do ssh ubuntu@${host} hostname; done
server
node-0
node-1

Adding /etc/hosts entries to the remote machines

Had to adjust the guide's command here too, same reason as before (ubuntu user instead of root), plus a gotcha with sudo and output redirection:

root@luger-VirtualBox:~/kubernetes-the-hard-way# while read IP FQDN HOST SUBNET; do
  scp hosts ubuntu@${HOST}:~/
  ssh -n ubuntu@"${IP}" "sudo sh -c 'cat hosts >> /etc/hosts'"
done < machines.txt

Interesting nugget of info I learned here: sudo somecommand >> file doesn't work the way you'd expect, sudo only elevates the command itself, not the >> redirection. The redirection is set up by your current shell before sudo ever runs, so it still tries to write as your regular user. Wrapping the whole thing in sudo sh -c '...' fixes it, since now the shell doing the redirecting is the one running as root.

Summary

Step 03 set up the actual infrastructure: root/admin SSH access between the jumpbox and all three nodes, and consistent hostnames (server, node-0, node-1) so every machine can address the others by name instead of IP.