Securing Kubernetes: Adding a new hostname or IP address to Kubernetes API Server

$:# kubectl get ns
Unable to connect to the server: x509: certificate is valid ...
$:# 

Introduction:

In the ever-evolving landscape of Kubernetes, securing connections to the API server is paramount. This article presents an approach on how to add a hostname or IP address to the TLS certificate used by the Kubernetes API server to allow continued secure access. Not assigning a static IP address to the control plane from get go, or using a new hostname or different hostname to access the API Server may result in the error highlighted above when attempting to access the cluster.

A quick workaround is to use the --insecure-skip-tls-verify command line argument with kubectl. However, it's crucial to note that this method skips TLS verification, rendering the connection insecure, and thus improper for Production use.

In a short delve into the technical background, it is important for know that the Kubernetes API server employs digital certificates to encrypt incoming and outgoing traffic and authenticate connections. For those interested in understanding the underlying processes, attempting to connect to the API server via kubectl with a hostname or IP address not included in the certificate's Subject Alternative Names (SAN) will result in an error. This error signifies that the certificate is not valid for the specified IP address or hostname.

To resolve this issue, it is necessary to update the certificate to include all relevant IP addresses or hostnames in the list of SANs. This ensures that the API server recognizes and validates connections initiated with the specified addresses or hostnames.

Before You Begin:

Before embarking on any updates, it's imperative to consider the production environment. For nodes actively serving in production, pull them out of the active node pool or gracefully drain live traffic using kubectl drain. This ensures a smooth transition without disrupting critical workloads.

 

Updating Kubernetes API Server Certificates:

Before proceeding with the certificate update, it is essential to ensure that the IP address of the node is accurately reflected across all pertinent manifest files. These Kubernetes component manifests are centrally located in the directory /etc/kubernetes/manifests/ and undergo real-time updates. Consequently, any modifications made to these files are promptly redeployed. The following steps outline the process:

 

Update Kubelet config:

$:# vi /etc/default/kubelet
KUBELET_EXTRA_ARGS=--node-ip=x.x.x.x #newIPaddress
$:# 

 

Update Kubernetes manifest

Edit the following Kubernetes manifests files - etcd.yaml and kube-apiserver.yaml. In the etcd.yaml - update the old IP address registered with the new IP address. You can execute the below, and it will update the address wherever it is present in the file.

$:# vi /etc/kubernetes/manifests/etcd.yaml
$:# 

Once the file is open, ensure you are escaped and run this command:

:%s/{oldIPaddress}/{newIPaddress}/gc

 
In the kube-apiserver.yaml file, update the IP address value of the advertise-address and save the file.

$:# vi /etc/kubernetes/manifests/kube-apiserver.yaml
$:# 

--
spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=x.x.x.x #newIPaddress

With the above steps executed, the manifest files are updated, and now you may proceed with updating the certificate accordingly.

---

Retrieve the kubeadm Configuration File:

export KUBECONFIG=/etc/kubernetes/admin.conf
kubectl -n kube-system get configmap kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' --insecure-skip-tls-verify > kubeadm.yaml

 

Fine-Tune the Configuration File:

Open the saved kubeadm.yaml in a text editor and navigate to the certSANs section under apiServer. Update the new IP address or hostname.

apiServer:
  certSANs:
  - "10.10.10.100"
  - "kubernetes.default"
  - "new-hostname"
  - "X.X.X.X" #newIPaddress
  extraArgs:
    ...

 

Recreate API Server Certificates:

This process will generate a new certificate and key for the API server, using the specified configuration file. Since the specified configuration file includes a certSANs list, kubeadm will automatically add those SANs when creating the new certificate.

mv /etc/kubernetes/pki/apiserver.{crt,key} ~
kubeadm init phase certs apiserver --config kubeadm.yaml

 

Restart the kubeapiserver Container:

Depending on the container runtime being used, follow either of the options below

Using Containerd as the runtime:

• Run crictl pods | grep kube-apiserver | cut -d' ' -f1 to get the Pod ID for the Kubernetes API server Pod.
• Run crictl stop <pod-id> to stop the Pod.
• Run crictl rm <pod-id> to remove the Pod.

Using Docker as the runtime:

• Run docker ps | grep kube-apiserver | grep -v pause to get the container ID for the container running the Kubernetes API server.
• Run docker kill <containerID> to kill the container.

Upon completion of these steps, Kubelet will autonomously initiate the restart of the container, facilitating the adoption of the updated certificate. Following the API server's restart, immediate connectivity will be established, allowing access through either of the newly-added IP addresses or hostnames.

---

Validation:

This is dual pronged:

• Use kubectl to access the cluster and confirm that the node and pod are all accessible.

export KUBECONFIG=/etc/kubernetes/admin.conf
kubectl get ns

• Alternatively, employ this verification method, leveraging OpenSSL on the Kubernetes control plane node. This step entails decoding the certificate and inspecting the list of Subject Alternative Names (SANs) embedded within the certificate:

openssl x509 -in /etc/kubernetes/pki/apiserver.crt -text

Inspect the "X509v3 Subject Alternative Name" line, which lists the DNS names and IP addresses incorporated as Subject Alternative Names (SANs) in the certificate. Executing this procedure should show the newly-added names and IP addresses specified in the modified kubeadm configuration file. If any error persists, revisit the process, checking for common errors such as overlooking the removal of the previous certificate and key or neglecting to include --config kubeadm.yaml in the kubeadm init phase certs command.

 

Updating the In-Cluster Configuration:

Assuming a smooth execution of all changes, the final step entails updating the kubeadm ConfigMap stored in the cluster. This guarantees that when kubeadm is utilized for a cluster upgrade later, the updated information seamlessly integrates into the cluster

kubeadm config upload from-file --config kubeadm.yaml

Verify the successful application of changes to the configuration with the command:

kubectl -n kube-system get configmap kubeadm-config -o yaml

---

Conclusion:

In the dynamic world of Kubernetes, ensuring secure access to the API server is a critical aspect of maintaining a robust and reliable infrastructure. By carefully updating and fine-tuning the TLS certificates used by the Kubernetes API server, administrators can seamlessly integrate new IP addresses or hostnames into the certificate's Subject Alternative Names (SANs), thus enabling secure and uninterrupted access. However, it is crucial to follow the outlined steps diligently and verify the changes to guarantee a smooth transition without compromising the security of the Kubernetes cluster.

Thank you for reading, and I hope someone finds this helpful. I am happy to chat further about new knowledge, opportunities and possible corrections. I can be reached on twitter via, @oluseyeo_

Happy Hacking 💥 💥

Comments

[Ahmed Ramadan]

Thanks, it helped me a lot. I need to add a note:

If using Kubernetes < v1.15:

kubeadm config upload from-file --config kubeadm.yaml

For Kubernetes version >= v1.15:

kubeadm init phase upload-config kubeadm --config kubeadm.yaml