Fixing Expired Certificates In Kubernetes

By default, when you setup your Kubernetes cluster, the certificates expires after one year.

If it's been a while since you started up your Kubernetes cluster and you try to issue kubectl commands and notice connection refused errors.

The below commands requires 'root' privileged, so if you are login as non-root, you will need to prepend sudo before the commands.

Troubleshooting further, you noticed that your kubelet service is failing to start (systemctl status kubelet).

Upon checking logs related with kubelet (journalctl | grep kubelet), you noticed the belong error messages.

There are several ways to verify our issue, let's issue kubeadm certs check-expiration.

Bingo!

An alternative to using kubeadm would be to use 'openssl' command.
Let's check a specific certificate, in this case the API server certificate:

• "openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text | grep 'Not After'"

Not After : Nov 07 01:16:00 2024 GMT

As we see, the API server certificate has expired, you can also check the other certificates to verify:

• /etc/kubernetes/pki/front-proxy-client.crt
• /etc/kubernetes/pki/etcd/peer.crt
• /etc/kubernetes/pki/etcd/server.crt
• /etc/kubernetes/pki/etcd/healthcheck-client.crt
• /etc/kubernetes/pki/apiserver-etcd-client.crt
• /etc/kubernetes/pki/apiserver-kubelet-client.crt

Let's renew certificates.

If your cluster has more than one control-plane node, be sure to run the following commands on all control-plane nodes in the cluster.

First, let's backup our certificates just in case we need them.

• cp -R /etc/kubernetes/pki /etc/kubernetes/pki.backup

Manually renew certificates to fix our issues.

Now let's renew our certificates with

• kubeadm certs renew all

This will renew all certificates, if only a specific certificate has expired, you can renew the specific certificate with the following command:

• kubeadm certs renew <specific server/service> - where would be for example, apiserver, so kubeadm certs renew apiserver

Verify new certificate installed correctly.

• kubeadm certs check-expiration or use openssl command from above

Restart kubelet service

• systemctl restart kubelet

Once the renew process is complete, we will need to restart all the control plane pods. One way to restart is moving it's manifest file out and wait for for about 20 seconds before moving the file back in to the 'manifests' folder. This will will recreate the Pod to use the new certificates.

• cd /etc/kubernetes/manifests
• mv *.yaml .. - Move all yaml files one level up
• mv ../*.yaml . - After 20 seconds, move all yaml file back to current directory

Copy the administrator certificates

• sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
• sudo chown $(id -u):$(id -g) $HOME/.kube/config

This should get your cluster running again. You can verify with kubectl get pods and not get an error.

Update kubeconfig files - only if you renew certificates individually

If you updated individual certificates and after restarting kubelet service, things are still not working, you might need to update kubeconfig files. Do the following:

• kubeadm init phase kubeconfig admin - this will update admin user (admin.conf).
• kubeadm init phase kubeconfig controller-manager - this will update controller manager.
• kubeadm init phase kubeconfig scheduler - this will update the scheduler.

If this still doesn't work, before continuing to troubleshoot, reboot the system to make sure all services are restarted correctly.

• sudo reboot or sudo shutdown -r now
• systemctl status kubelet - After reboot, login and check status again. if fail, view section on "Additional Things To Do..." below.

Renew automatically via Kubernetes Upgrade
If you also need to update your Kubernetes version, you can apply the upgrade and all certificates will get renewed.

Let's run the following command to check current version and what available upgrades version.

• kubeadm upgrade plan

Find the version to upgrade to and issue the following to upgrade.

• kubeadm upgrade apply <version> - substitute the with the version number to update to.

Additional Things To Do If The Steps Above Doesn't Work

If for some reason the above doesn't work, you will need to do the following. Make a backup of /etc/kubernetes/pki/

• cp -R /etc/kubernetes/pki /etc/kubernetes/pki-backup

Delete the following files from /etc/kubernetes/pki

• rm apiserver.crt \ apiserver-etcd-client.key\ apiserver-kubelet-client.crt\ front-proxy-ca.crt\ front-proxy-client.crt\ front-proxy-client.key\ front-proxy-ca.key\ apiserver-kubelet-client.key\ apiserver.key\ apiserver-etcd-client.crt

Remove all .crt and .key files from /etc/kubernetes/pki/etcd

• rm /etc/kubernetes/pki/etcd/*.crt
• rm /etc/kubernetes/pki/etcd/*.key

Then create certs with

• kubeadm init phase certs all --apiserver-advertise-address <IP> --substitute your cluster IP.

Backup\Move conf files from /etc/kubernetes

• mkdir conf-backup
• mv admin.conf\ controller-manager.conf\ kubelet.conf\ scheduler.conf /etc/kubernetes/conf-backup/

Create new conf files

• kubeadm init phase kubeconfig all

Finally, restart kubelet service or reboot system.

• systemctl restart kubelet or reboot