Demystifying mTLS in Kubernetes: Certs, Components, and Cluster Security.

🧩 Introduction

When we talk about Kubernetes security, most developers think of Role-Based Access Control (RBAC) or Network Policies. But before these higher-level controls come into play, there's a foundational layer that silently ensures every component in the cluster is speaking to a trusted source — mutual TLS (mTLS).

In a self-managed Kubernetes cluster, unlike cloud-managed solutions like EKS or GKE, you're responsible for configuring, maintaining, and rotating these certificates. Understanding how mTLS works—and knowing where each certificate lives—gives you deeper visibility into the cluster's internal trust system and can help you troubleshoot or harden your deployment.

This post will walk you through:

• What mTLS means in the Kubernetes control plane
• Which components use certificates and why
• Where those certificates live in your filesystem
• How to inspect and manage them in your self-hosted cluster

---

🔐 What is mTLS in Kubernetes?

mTLS (Mutual TLS) is a mechanism where both client and server authenticate each other using certificates. In Kubernetes, this isn't just used to encrypt traffic; it's also a crucial part of authenticating various components within the control plane.

Let's look at a few examples:

• When the kubelet talks to the API server, it presents a certificate proving it is a legitimate node
• When the controller-manager communicates with etcd, both use TLS certificates to verify identity
• Even users interacting with the cluster via kubectl often use certificates behind the scenes (depending on your setup)

This secure, verified communication is enforced using X.509 certificates, which are generated during cluster setup (commonly via kubeadm) and stored in predictable locations in the filesystem.

---

🧩 Kubernetes Components Using Certificates (mTLS)

Here's a breakdown of the key components and where their TLS certificates typically live in a self-managed (kubeadm-based) cluster:

Component	Certificate Path	Purpose / Notes
API Server	/etc/kubernetes/pki/apiserver.crt & .key	Server cert for API requests
API Server CA	/etc/kubernetes/pki/ca.crt & .key	Root CA that signs other certs
Controller Manager	/etc/kubernetes/pki/controller-manager.crt & .key	Used to talk to API server
Scheduler	/etc/kubernetes/pki/scheduler.crt & .key	Talks to API server securely
Etcd (server)	/etc/kubernetes/pki/etcd/server.crt & .key	API server & etcd authenticate each other
Etcd (peer)	/etc/kubernetes/pki/etcd/peer.crt & .key	For peer-to-peer etcd clustering
Etcd CA	/etc/kubernetes/pki/etcd/ca.crt & .key	Separate CA from main API server
Kubelet	/var/lib/kubelet/pki/kubelet-client.crt & .key	Authenticates with API server
Kube-Proxy	/etc/kubernetes/pki/kube-proxy.crt & .key	Talks to API server for service updates
Front Proxy	/etc/kubernetes/pki/front-proxy-client.crt	For API aggregation layer
Admin User (kubectl)	/etc/kubernetes/admin.conf	Includes user certs for API access

---

🛠️ Who Issues These Certificates?

In self-managed Kubernetes clusters, especially those initialized with kubeadm, the certificate lifecycle is handled during setup — and it's your responsibility to maintain them over time.

🔐 CA (Certificate Authority)

When you run kubeadm init, it generates a root Certificate Authority (CA) stored at:

/etc/kubernetes/pki/ca.crt
/etc/kubernetes/pki/ca.key

This CA is then used to sign the server and client certificates for all core components (API server, controller manager, scheduler, etc.).

There's also a separate CA for etcd:

/etc/kubernetes/pki/etcd/ca.crt
/etc/kubernetes/pki/etcd/ca.key

⚙️ Inspecting the Certificates

You can list all certs created by kubeadm using:

sudo kubeadm certs list

And you can check expiration dates with:

sudo kubeadm certs check-expiration

You'll typically see that most certs expire in 1 year by default.

🔁 Renewing Certificates

To renew all expiring certs (except kubelet), you can run:

sudo kubeadm certs renew all

Note: The kubelet uses automatic rotation, but only if the RotateKubeletClientCertificate setting is enabled in /var/lib/kubelet/config.yaml.

---

🔁 mTLS in Action: Example Flow

Let's take a concrete example: how the kubelet authenticates with the API server.

1. The kubelet starts up and wants to register the node with the cluster

2. It presents its client certificate (/var/lib/kubelet/pki/kubelet-client.crt) to the API server

3. The API server checks:

• Is the cert signed by the CA (ca.crt) it trusts?
• Is it expired?
• Does it belong to a valid node?

1. If valid, the kubelet is authenticated and proceeds to perform actions like:
   • Registering the node
   • Posting status updates
   • Watching for Pod objects

This exact mTLS exchange also happens between:

• API server ↔ controller-manager
• API server ↔ scheduler
• API server ↔ etcd
• etcd ↔ etcd peers

Each time, both sides present certificates and verify identities before any communication happens — enforcing strict trust boundaries within the control plane.

---

🔎 How to Inspect Kubernetes Certificates Manually

Sometimes you'll want to manually inspect a certificate — whether for troubleshooting, verifying SANs, or checking expiration dates. You can use openssl for this.

📂 Example: Inspect the API Server certificate

sudo openssl x509 -in /etc/kubernetes/pki/apiserver.crt -text -noout

This will output details like:

• Issuer – usually kubernetes-ca
• Validity – Not Before / Not After (expiration)
• Subject – e.g., CN=kube-apiserver
• X509v3 Subject Alternative Name – e.g., IPs and DNS names like kubernetes, kubernetes.default, etc.

📂 Example: Check kubelet certificate

sudo openssl x509 -in /var/lib/kubelet/pki/kubelet-client.crt -text -noout

This helps confirm:

• Which node the cert is tied to
• If rotation has occurred
• Whether the cert has expired or is about to expire

---

🛡️ Security & Hardening Tips

When you manage your own Kubernetes control plane, here are a few things to keep in mind:

🔄 Rotate Certificates Regularly

• Certs expire after 1 year by default
• Use kubeadm certs renew or set up automation
• Monitor expiration with:

  kubeadm certs check-expiration

🔐 Protect Your CA Keys

• Keep /etc/kubernetes/pki/ca.key and /etc/kubernetes/pki/etcd/ca.key strictly restricted
• Never expose CA keys to unauthorized users or automation tools

🚫 Never Expose etcd Publicly

• etcd should never be exposed to the public internet
• Ensure it only listens on localhost or secure internal networks
• Always enforce client certs (--cert-file, --key-file, --client-cert-auth)

🔍 Audit TLS Flags in Static Pod Manifests

Check /etc/kubernetes/manifests/ for your API server pod config:

- --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
- --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
- --client-ca-file=/etc/kubernetes/pki/ca.crt

Ensure all these values are set and point to correct, non-expired certs.

---

✅ Conclusion

Mutual TLS is at the heart of Kubernetes' internal security model — silently ensuring that every component talks only to what it can trust. In cloud-managed clusters, this is handled for you, but in self-managed setups, you're the certificate authority, quite literally.

In this post, you learned:

• What mTLS is and why it matters in Kubernetes
• Which components use it and where the certs are stored
• How certificates are issued and rotated
• How to inspect and secure them

Understanding these mechanics gives you a strong edge as a DevOps engineer, especially when operating production clusters or preparing for certifications like the CKA.

---

What's your experience with managing certificates in Kubernetes? Have you encountered any certificate-related issues in your clusters? Share your thoughts in the comments below!