[Lab Notes] Kubernetes the Hard Way, For Real This Time (Step 04)

#kubernetes #proxmox #beginners #devops

Continuing my Kubernetes the Hard Way homelab build on Proxmox. Steps 01-03 are already done, this covers step 04.

Original guide: 04-certificate-authority.md

Thoughts I had while doing this

Before the actual step though, here are some stuff I learned or was thinking about along the way:

Kubernetes is starting to feel like a miniaturized version of an entire IT infrastructure. Compute already got abstracted from physical servers to containers. It's like Kubernetes took that same idea and asked "hey we already abstracted compute, why not abstract the whole infrastructure?"
Generating and installing all these certs felt exactly like setting up TLS between a bunch of servers so they can trust each other. Every Kubernetes component (API server, kubelet, etc.) is like another service on the network, and they all need to trust each other the same way any two servers would.

Anyways, here's Step 04:

Certificate Authority

Generated the CA key and self-signed CA certificate:

root@luger-VirtualBox:~/kubernetes-the-hard-way# {
  openssl genrsa -out ca.key 4096
  openssl req -x509 -new -sha512 -noenc \
    -key ca.key -days 3653 \
    -config ca.conf \
    -out ca.crt
}

Working directory at this point:

root@luger-VirtualBox:~/kubernetes-the-hard-way# ls -ltr
total 72
drwxr-xr-x 2 root root  4096 Sep  7 03:47 units
-rw-r--r-- 1 root root  2624 Sep  7 03:47 README.md
-rw-r--r-- 1 root root 11358 Sep  7 03:47 LICENSE
-rw-r--r-- 1 root root   839 Sep  7 03:47 downloads-arm64.txt
-rw-r--r-- 1 root root   839 Sep  7 03:47 downloads-amd64.txt
drwxr-xr-x 2 root root  4096 Sep  7 03:47 docs
-rw-r--r-- 1 root root   407 Sep  7 03:47 COPYRIGHT.md
-rw-r--r-- 1 root root  1059 Sep  7 03:47 CONTRIBUTING.md
drwxr-xr-x 2 root root  4096 Sep  7 03:47 configs
-rw-r--r-- 1 root root  5863 Sep  7 03:47 ca.conf
drwxr-xr-x 6 root root  4096 Sep  7 03:59 downloads
-rw-r--r-- 1 root root   163 Sep 10 04:05 machines.txt
-rw-r--r-- 1 root root   162 Sep 10 04:16 hosts
-rw------- 1 root root  3272 Sep 11 03:39 ca.key
-rw-r--r-- 1 root root  1899 Sep 11 03:39 ca.crt

No deviations here, followed the guide exactly.

Create client and server certificates

Generated a key, CSR, and signed cert for each Kubernetes component in one loop:

root@luger-VirtualBox:~/kubernetes-the-hard-way# certs=(
  "admin" "node-0" "node-1"
  "kube-proxy" "kube-scheduler"
  "kube-controller-manager"
  "kube-api-server"
  "service-accounts"
)
root@luger-VirtualBox:~/kubernetes-the-hard-way# for i in ${certs[*]}; do
  openssl genrsa -out "${i}.key" 4096

  openssl req -new -key "${i}.key" -sha256 \
    -config "ca.conf" -section ${i} \
    -out "${i}.csr"

  openssl x509 -req -days 3653 -in "${i}.csr" \
    -copy_extensions copyall \
    -sha256 -CA "ca.crt" \
    -CAkey "ca.key" \
    -CAcreateserial \
    -out "${i}.crt"
done

Confirmed all the expected files were generated:

root@luger-VirtualBox:~/kubernetes-the-hard-way# ls -1 *.crt *.key *.csr
admin.crt
admin.csr
admin.key
ca.crt
ca.key
kube-api-server.crt
kube-api-server.csr
kube-api-server.key
kube-controller-manager.crt
kube-controller-manager.csr
kube-controller-manager.key
kube-proxy.crt
kube-proxy.csr
kube-proxy.key
kube-scheduler.crt
kube-scheduler.csr
kube-scheduler.key
node-0.crt
node-0.csr
node-0.key
node-1.crt
node-1.csr
node-1.key
service-accounts.crt
service-accounts.csr
service-accounts.key

Also no deviations here, straight from the guide.

Distribute the client and server certificates

Copied the CA cert plus each worker's own cert/key to the right nodes:

root@luger-VirtualBox:~/kubernetes-the-hard-way# for host in node-0 node-1; do
  ssh root@${host} mkdir /var/lib/kubelet/

  scp ca.crt root@${host}:/var/lib/kubelet/

  scp ${host}.crt \
    root@${host}:/var/lib/kubelet/kubelet.crt

  scp ${host}.key \
    root@${host}:/var/lib/kubelet/kubelet.key
done

Got a "directory already exists" warning on both nodes, but the file copies all succeeded, so no issue:

mkdir: cannot create directory ‘/var/lib/kubelet/’: File exists
ca.crt        100% 1899     3.1MB/s   00:00
node-0.crt    100% 2147     3.9MB/s   00:00
node-0.key    100% 3268     6.7MB/s   00:00
mkdir: cannot create directory ‘/var/lib/kubelet/’: File exists
ca.crt        100% 1899     2.8MB/s   00:00
node-1.crt    100% 2147     3.1MB/s   00:00
node-1.key    100% 3272     5.1MB/s   00:00

Then copied the CA, API server, and service-account key/cert pairs to the controller (server):

root@luger-VirtualBox:~/kubernetes-the-hard-way# scp \
  ca.key ca.crt \
  kube-api-server.key kube-api-server.crt \
  service-accounts.key service-accounts.crt \
  root@server:~/
ca.key                  100% 3272     4.5MB/s   00:00
ca.crt                  100% 1899     5.0MB/s   00:00
kube-api-server.key     100% 3272     8.2MB/s   00:00
kube-api-server.crt     100% 2354     5.0MB/s   00:00
service-accounts.key    100% 3272     7.7MB/s   00:00
service-accounts.crt    100% 2004     3.9MB/s   00:00

Summary

Step 04 built out the cluster's PKI: a self-signed CA, then a key/cert pair for every component (admin, both worker nodes, kube-proxy, kube-scheduler, kube-controller-manager, kube-api-server, and service-accounts), distributed to the right machines.

DEV Community