DEV Community

ThankGod Chibugwum Obobo
ThankGod Chibugwum Obobo

Posted on • Originally published at actocodes.hashnode.dev

Implementing mTLS in Kubernetes: How to Secure Service-to-Service Communication

#kubernetes #mtls #istio #linkerd

Introduction

In a microservices architecture running on Kubernetes, services communicate constantly, the orders-service calls the users-service, the payments-service calls the fraud-detection-service, and the API gateway routes to a dozen downstream services. By default, most of this traffic travels unencrypted inside the cluster, authenticated only by the fact that the caller has network access.

That assumption that anything inside the cluster boundary is trustworthy, is the network-level equivalent of leaving every internal door unlocked because the building has a lock on the front entrance. A single compromised pod, a misconfigured network policy, or a lateral movement attack can reach any service in the cluster with no additional authentication required.

Mutual TLS (mTLS) closes this gap. Unlike standard TLS where only the server presents a certificate, mTLS requires both parties, client and server, to present and verify certificates before any data is exchanged. Every service-to-service connection is cryptographically authenticated. A compromised pod cannot impersonate payments-service to fraud-detection-service because it cannot produce a valid certificate for that identity.

This guide covers how to implement mTLS in Kubernetes using a service mesh, comparing Istio and Linkerd, configuring automatic certificate issuance, enforcing strict mTLS policies, and validating that your network is actually secured.

How mTLS Works in Kubernetes

Standard TLS in a web context works like this: the client (browser) verifies the server's certificate, establishing that the server is who it claims to be, but the server accepts any client. mTLS adds the reverse, the server also verifies the client's certificate, establishing mutual identity before the connection proceeds.

In Kubernetes, the mechanics are:

  1. Each service is issued a certificate tied to its SPIFFE identity, a URI in the format spiffe://cluster.local/ns/{namespace}/sa/{service-account}.
  2. When orders-service connects to users-service, both services present their certificates.
  3. The receiving service verifies the caller's certificate against the cluster's trust anchor (a root CA).
  4. If verification passes, the connection proceeds. If not, wrong identity, expired certificate, or untrusted issuer, the connection is rejected at the TLS layer, before any application code runs.

The critical operational property: certificates are automatically rotated by the service mesh control plane, typically every 24 hours. No manual certificate management, no expiry incidents, no long-lived credentials.

Choosing a Service Mesh: Istio vs. Linkerd

Both Istio and Linkerd implement mTLS via sidecar proxies injected into each pod. The proxies intercept all inbound and outbound traffic, transparently handling TLS without requiring application code changes.

Feature Istio Linkerd
mTLS implementation Envoy sidecar Linkerd2-proxy sidecar
Certificate authority Istiod (built-in) or external Built-in or cert-manager
mTLS enforcement PeerAuthentication policy Server policy
Observability Rich (Kiali, Jaeger integration) Lightweight (built-in dashboard)
Resource overhead Higher (~200–350MB per sidecar) Lower (~10–20MB per sidecar)
Configuration complexity Higher Lower
FIPS 140-2 compliance Enterprise Enterprise
Multi-cluster support Native Native

Choose Istio if you need advanced traffic management (canary deployments, fault injection, circuit breaking), rich observability, or WebAssembly-based policy extensibility.

Choose Linkerd if you prioritize simplicity, low resource overhead, and want mTLS with strong defaults out of the box with minimal configuration.

This guide covers both installation and policy configuration for each.

Approach 1 - mTLS with Istio

Step 1 - Install Istio

Install Istio using the istioctl CLI with the minimal production profile:

# Install istioctl
curl -L https://istio.io/downloadIstio | sh -
export PATH=$PWD/istio-*/bin:$PATH

# Install Istio with production profile
istioctl install --set profile=default \
  --set values.pilot.env.PILOT_ENABLE_WORKLOAD_ENTRY_AUTOREGISTRATION=true \
  -y

# Verify installation
istioctl verify-install
Enter fullscreen mode Exit fullscreen mode

Step 2 - Enable Automatic Sidecar Injection

Label namespaces where you want mTLS enforced. Istio automatically injects the Envoy sidecar proxy into every pod in labeled namespaces:

# Enable sidecar injection for your application namespaces
kubectl label namespace orders-service istio-injection=enabled
kubectl label namespace users-service istio-injection=enabled
kubectl label namespace payments-service istio-injection=enabled

# Verify injection is active
kubectl get namespace -L istio-injection
Enter fullscreen mode Exit fullscreen mode

Rolling restart existing deployments to pick up sidecar injection:

kubectl rollout restart deployment -n orders-service
kubectl rollout restart deployment -n users-service
kubectl rollout restart deployment -n payments-service
Enter fullscreen mode Exit fullscreen mode

Verify sidecars are injected, each pod should now show 2/2 containers (application + Envoy proxy):

kubectl get pods -n orders-service
# NAME                             READY   STATUS    RESTARTS   AGE
# orders-api-7d8b9f5c4-xk2pq      2/2     Running   0          45s
Enter fullscreen mode Exit fullscreen mode

Step 3 - Configure PeerAuthentication for Strict mTLS

By default, Istio operates in permissive mode, it accepts both plaintext and mTLS traffic. This is safe for migration but provides no security guarantee. Switch to strict mode to enforce mTLS and reject all plaintext traffic:

# istio/peer-authentication-strict.yaml
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: istio-system   # mesh-wide policy
spec:
  mtls:
    mode: STRICT
Enter fullscreen mode Exit fullscreen mode 
kubectl apply -f istio/peer-authentication-strict.yaml
Enter fullscreen mode Exit fullscreen mode

This single resource enforces mTLS for all service-to-service communication across the entire mesh. Any pod that attempts to connect to another pod without a valid certificate, including debug containers, legacy services without sidecars, or external probes, will have their connection rejected.

Step 4 - AuthorizationPolicy for Fine-Grained Access Control

mTLS establishes who a caller is. AuthorizationPolicy controls what authenticated identities are allowed to do, combining mTLS identity with RBAC-style rules:

# istio/authorization-payments.yaml
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: payments-service-policy
  namespace: payments-service
spec:
  selector:
    matchLabels:
      app: payments-service
  action: ALLOW
  rules:
    - from:
        - source:
            principals:
              # Only orders-service can call payments-service
              - "cluster.local/ns/orders-service/sa/orders-service-sa"
      to:
        - operation:
            methods: ["POST"]
            paths: ["/payments/charge", "/payments/refund"]
Enter fullscreen mode Exit fullscreen mode 
# Deny all traffic not explicitly allowed, defense in depth
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: deny-all
  namespace: payments-service
spec:
  {}  # Empty spec = deny all by default
Enter fullscreen mode Exit fullscreen mode

Apply deny-all first, then layer specific ALLOW policies on top. This is the zero-trust principle applied at the service level: every inter-service call must be explicitly authorized.

Step 5 - Verify mTLS is Active with Kiali

Kiali, Istio's observability console, provides a visual confirmation of mTLS status:

# Install Kiali and supporting tools
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.20/samples/addons/kiali.yaml
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.20/samples/addons/prometheus.yaml

# Access Kiali dashboard
istioctl dashboard kiali
Enter fullscreen mode Exit fullscreen mode

In Kiali's service graph, a padlock icon on each edge confirms mTLS is active on that connection. Missing padlocks indicate a service communicating without mTLS, a misconfiguration or missing sidecar.

Approach 2 - mTLS with Linkerd

Step 1 - Install Linkerd 

# Install Linkerd CLI
curl --proto '=https' --tlsv1.2 -sSfL https://run.linkerd.io/install | sh

# Validate cluster compatibility
linkerd check --pre

# Install Linkerd control plane
linkerd install --crds | kubectl apply -f -
linkerd install | kubectl apply -f -

# Verify installation
linkerd check
Enter fullscreen mode Exit fullscreen mode

Step 2 - Inject Linkerd Proxy into Namespaces 

# Annotate namespaces for automatic injection
kubectl annotate namespace orders-service linkerd.io/inject=enabled
kubectl annotate namespace users-service linkerd.io/inject=enabled
kubectl annotate namespace payments-service linkerd.io/inject=enabled

# Restart deployments to pick up proxy injection
kubectl rollout restart deployment -n orders-service
kubectl rollout restart deployment -n users-service
kubectl rollout restart deployment -n payments-service
Enter fullscreen mode Exit fullscreen mode

Step 3 - Enforce mTLS with Server Policies

Linkerd enables mTLS by default in injected namespaces. To enforce authenticated traffic only (equivalent to Istio's strict mode), configure Server policies:

# linkerd/server-policy-payments.yaml
apiVersion: policy.linkerd.io/v1beta2
kind: Server
metadata:
  name: payments-server
  namespace: payments-service
spec:
  podSelector:
    matchLabels:
      app: payments-service
  port: 8080
  proxyProtocol: HTTP/2


apiVersion: policy.linkerd.io/v1beta2
kind: ServerAuthorization
metadata:
  name: payments-authz
  namespace: payments-service
spec:
  server:
    name: payments-server
  client:
    meshTLS:
      serviceAccounts:
        - name: orders-service-sa
          namespace: orders-service
Enter fullscreen mode Exit fullscreen mode 
kubectl apply -f linkerd/server-policy-payments.yaml

# Verify mTLS is active on connections
linkerd viz edges deployment -n orders-service
# Shows which connections are secured with mTLS (✓) vs plaintext (✗)
Enter fullscreen mode Exit fullscreen mode

Certificate Management with cert-manager

Both Istio and Linkerd can integrate with cert-manager for certificate issuance, enabling integration with external CAs (HashiCorp Vault, AWS ACM PCA, Let's Encrypt) rather than relying solely on the mesh's built-in CA.

# cert-manager/cluster-issuer-vault.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: vault-issuer
spec:
  vault:
    server: https://vault.internal:8200
    path: pki/sign/kubernetes-mesh
    auth:
      kubernetes:
        mountPath: /v1/auth/kubernetes
        role: cert-manager
        secretRef:
          name: cert-manager-vault-token
          key: token
Enter fullscreen mode Exit fullscreen mode

For Istio, plug the external CA into the control plane:

# Use cert-manager as Istio's certificate provider
istioctl install --set profile=default \
  --set values.pilot.env.EXTERNAL_CA=true \
  --set values.pilot.env.K8S_SIGNER=cert-manager.io/cluster-issuer/vault-issuer
Enter fullscreen mode Exit fullscreen mode

External CA integration provides:

  • Centralized audit trail of every certificate issuance across your mesh
  • Cross-cluster trust using a shared root CA for multi-cluster mTLS
  • Compliance alignment with PKI policies already in use in your organization

Migrating to Strict mTLS Without Downtime

Flipping directly to strict mode on a production cluster breaks any service that doesn't yet have a sidecar. Use a phased migration:

Phase 1 - Permissive mode (baseline). Install the mesh with mTLS in permissive mode. All traffic is accepted, both plaintext and mTLS. Use Kiali or linkerd viz to identify which connections are NOT using mTLS.

Phase 2 - Namespace-by-namespace injection. Enable sidecar injection one namespace at a time. Verify each namespace's workloads after injection before proceeding.

Phase 3 - Strict mode per namespace. Apply PeerAuthentication or Server policies with strict enforcement to already-injected namespaces:

# Namespace-scoped strict policy (safer than mesh-wide during migration)
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: strict-mtls
  namespace: payments-service   # scoped to one namespace
spec:
  mtls:
    mode: STRICT
Enter fullscreen mode Exit fullscreen mode

Phase 4 - Mesh-wide strict enforcement. Once all namespaces are injected and verified, apply the mesh-wide PeerAuthentication in istio-system and remove namespace-scoped policies.

This phased approach takes longer but eliminates the risk of a mesh-wide strict enforcement breaking services that weren't ready.

Validating mTLS Enforcement

After enabling strict mode, validate it is actually enforced, don't assume:

# Attempt to connect to payments-service from a pod WITHOUT a sidecar
kubectl run debug-pod --image=curlimages/curl --restart=Never -- \
  curl -s http://payments-service.payments-service.svc.cluster.local:8080/health

# Expected result: connection refused or reset, mTLS is enforced
# Unexpected result: 200 OK, mTLS is NOT enforced, investigate
Enter fullscreen mode Exit fullscreen mode 
# Verify certificate details on an active connection
kubectl exec -n orders-service deployment/orders-api -c istio-proxy -- \
  openssl s_client \
    -connect payments-service.payments-service.svc.cluster.local:8080 \
    -showcerts 2>/dev/null | grep -A2 "subject="

# Expected output shows SPIFFE URI:
# subject=URI:spiffe://cluster.local/ns/payments-service/sa/payments-service-sa
Enter fullscreen mode Exit fullscreen mode 
# Check Istio's view of mTLS status per service
istioctl x describe service payments-service.payments-service

# Output confirms:
# mTLS: yes
# Policy: STRICT
Enter fullscreen mode Exit fullscreen mode

Observability and Troubleshooting

mTLS adds authentication at the network layer, but it also adds debugging complexity. Keep these tools in your operational toolkit:

Istio proxy logs reveal TLS handshake failures with specific error codes:

# View Envoy access logs for a specific pod
kubectl logs -n orders-service deployment/orders-api -c istio-proxy | \
  grep -E "UF|UC|UH"  # Upstream connection failure codes
Enter fullscreen mode Exit fullscreen mode

Linkerd tap provides real-time traffic inspection with mTLS status:

linkerd viz tap deployment/orders-api -n orders-service \
  --to deployment/payments-service -n payments-service
Enter fullscreen mode Exit fullscreen mode

Certificate expiry monitoring even with automatic rotation, monitor certificate validity as a health signal:

# Prometheus alert for certificates expiring within 24 hours
- alert: MeshCertificateExpiringSoon
  expr: |
    (citadel_server_csr_count - citadel_server_success_cert_issuance_count) > 0
  for: 5m
  labels:
    severity: high
  annotations:
    summary: "Certificate issuance failures in Istio CA"
Enter fullscreen mode Exit fullscreen mode

Common Pitfalls to Avoid

Enabling strict mode before all services have sidecars. Services without sidecars cannot participate in mTLS and will have their connections rejected in strict mode. Always verify 100% sidecar coverage before enabling mesh-wide strict enforcement.

Forgetting health check probes. Kubernetes liveness and readiness probes originate from the kubelet, outside the mesh. Configure Istio to rewrite probe URLs or exclude probe ports from mTLS enforcement:

# Exclude health check port from mTLS, kubelet doesn't have a sidecar
spec:
  mtls:
    mode: STRICT
  portLevelMtls:
    8081:          # health check port
      mode: DISABLE
Enter fullscreen mode Exit fullscreen mode

Not testing AuthorizationPolicy before applying. A misconfigured deny-all policy can cut off legitimate service traffic in production. Test policies in a staging environment first, and use Istio's AUDIT action mode to log would-be denials before enforcing them.

Ignoring certificate rotation failures. The automatic certificate rotation is the operational backbone of mesh mTLS. Monitor certificate issuance metrics, a rotation failure that goes undetected leads to widespread connection failures when certificates expire.

Conclusion

Mutual TLS in Kubernetes transforms your cluster's internal network from an implicit trust zone into a zero-trust environment where every connection is cryptographically authenticated and explicitly authorized. A compromised pod cannot impersonate a legitimate service. A lateral movement attack cannot freely reach payment processing or user data services. Every inter-service call is auditable.

The operational cost, a service mesh, sidecar overhead, policy management, is real. But for systems where data sensitivity, regulatory requirements, or blast-radius concerns justify it, mTLS via a service mesh is the most complete implementation of zero-trust networking currently available for Kubernetes workloads.

Start with permissive mode, inject sidecars namespace by namespace, validate mTLS coverage with your mesh's observability tools, and graduate to strict enforcement once you're confident in coverage. The security guarantee at the end is worth the careful migration.

Running a multi-cluster Kubernetes setup? Extending mTLS across cluster boundaries with Istio's multi-cluster federation or Linkerd's cluster linking uses the same SPIFFE identity model, but adds a trust bundle exchange step between clusters.

Top comments (0)