Fixing Expired Certificates In Kubernetes for Worker Nodes

By default, when you setup your Kubernetes cluster, the certificates expires after one year.

The process to check is exactly like the process explained in https://dev.to/hatr0/fixing-expired-certificates-in-kubernetes-1oe2.
I have copied and paste all the needed commands here so no need to visit the post.

From the control plane node, if you issue the following command:

• kubectl get nodes

You'll notice some of your worker nodes have a STATUS of NotReady

Let's do some further checking. SSH to one of the worker nodes that has a status of NotReady.

The below commands requires 'root' privileged, so if you are login as non-root, you will need to prepend sudo before the commands.

Check your kubelet service, is it failing to start (systemctl status kubelet)?

Let's check logs related with kubelet (journalctl | grep kubelet), you'll noticed the belong error messages.

You should see an entry about bootstrap client certificate is expired.

Since the kubelet service on worker nodes talks to control plane, an expired license will prevent the kubelet from authenticating with the API server.

Let's verify our issue with the 'openssl' command.

Let's check kubelet client certificate with the following command:

• openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -text | grep 'Not After'

Not After : Feb 16 00:38:07 2023 GMT

If the date shown is in the past, the certificate has expired.

Let's renew certificates.

If your cluster has more than one worker node, be sure to check certificate expiration with the command above on all nodes. If certificate has expired, run the following commands on all expired worker nodes in the cluster.

First, let's backup our certificates just in case we need them.

• cp /etc/kubernetes/kubelet.conf /etc/kubernetes/kubelet.conf.backup
• cp -r /var/lib/kubelet/pki /var/lib/kubelet/pki.backup

Next, let's remove the expired certificates

• rm /etc/kubernetes/kubelet.conf
• rm /var/lib/kubelet/pki/kubelet-client* - Note: Wildcard might not work due to security restrictions, so delete each file individually.

After you delete the expired certificates, let's generate a new kubelet configuration.

On a working control plane node, let's first create a working config file since that is needed to create a new kubelet configuration file.

1. Create a file name temp-config.yaml
2. Add the following to the file and save the file. You can verify and match the entries with the current running cluster with the following command: kubectl get cm kubeadm-config -n kube-system -o=jsonpath="{.data.ClusterConfiguration}" This information can be found at https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/

# Temporary config.yaml file to generate new kubelet configuration file
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
# Cluster name in the kubeconfig 
clusterName: "kubernetes"
# The server IP/DNS name of the cluster in the kubeconfig
controlPlaneEndpoint: "IP or DNS name:6443"
# Cluster CA key and certificate will be loaded from this local directory
certficatesDir: "/etc/kubernetes/pki"

Now with the config.yaml file created, let's create the configuration file, substitute with the actual hostname of the worker node you are fixing, execute the following:

export NODE=<worker-node-name>
kubeadm kubeconfig user --org system:nodes --client-name system:node:${NODE} --config temp-config.yaml > kubelet.conf

Once the kubelet.conf file is generated, let's transfer this to the worker node. Substitute with the worker node IP or DNS name you are fixing.

scp kubelet.conf <worker-node>:/tmp/kubelet.conf
ssh <worker-node>
mv /tmp/kubelet.conf /etc/kubernetes/kubelet.conf
chown root:root /etc/kubernetes/kubelet.conf

Now still logged in on worker node, restart the kubectl service
systemctl restart kubelet

Verify by checking kubelet service and issue kubectl commands.