DEV Community

Jeongwoo Kim
Jeongwoo Kim

Posted on

[Hands-on] Kubernetes Pod Certificate Request introduced in v1.35

#k8s #podcertificaterequests #athenz

Goal

[!NOTE]
In hurry? Jump to the result!

The goal of this document is to generate auto signed certificate for any pod with the following projected volumes:

volumes:
- name: creds
  projected:
    sources:
    - podCertificate:
        signerName: row-major.net/spiffe
        keyType: ED25519
        credentialBundlePath: service.crt
        keyPath: service.key
    - clusterTrustBundle:
        name: row-major.net:spiffe:primary-bundle
        path: ca.crt
Enter fullscreen mode Exit fullscreen mode

Table of Contents

Walkthrough

Here is the step-by-step record of how I achieved the goal.

Setup: Working Directory

Let's quickly create a test directory build:

test_name=pod_certificate_request
tmp_dir=$(date +%y%m%d_%H%M%S_$test_name)
mkdir -p ~/test_dive/$tmp_dir
cd ~/test_dive/$tmp_dir
Enter fullscreen mode Exit fullscreen mode

Setup: Kind Cluster with Cert Provisioning Enabled

[!NOTE]
Please note that the point of this blog's release already contains v1.35+ that includes the feature, so we do not need to set specific version.

Create a kind cluster locally with the following command:

_cluster_name="cert-provisioning"
_k8s_version="v1.35.0"

cat <<EOF | kind create cluster --name "$_cluster_name" --image kindest/node:$_k8s_version --config -
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
featureGates:
  PodCertificateRequest: true
  ClusterTrustBundle: true
  ClusterTrustBundleProjection: true
runtimeConfig:
  "certificates.k8s.io/v1beta1/podcertificaterequests": "true"
  "certificates.k8s.io/v1beta1/clustertrustbundles": "true"
nodes:
- role: control-plane
- role: worker
EOF
Enter fullscreen mode Exit fullscreen mode

Check new cluster created with version v1.35.0:

kubectl get nodes

# NAME                              STATUS   ROLES           AGE   VERSION
# cert-provisioning-control-plane   Ready    control-plane   29s   v1.35.0
# cert-provisioning-worker          Ready    <none>          19s   v1.35.0
Enter fullscreen mode Exit fullscreen mode

Setup: Mash Controller Deployed

Let's deploy the mesh-controller developed by @ahmedtd as a sample. First, clone the helper project:

git clone https://github.com/ahmedtd/mesh-example.git mesh_example
Enter fullscreen mode Exit fullscreen mode

Before we deploy the mesh-controller, we need to create a CA pool secret that the controller can use to sign the certificate requests. The mesh-controller expects two different CA pool secrets:

  • service-dns-ca-pool: for service DNS certificates between kubeapi server and mesh controller
  • spiffe-ca-pool: for ca certificate to be used to sign the auto distributed certificates to pods

The helper project provides the following command to create the CA pool & save as k8 secrets:

(cd mesh_example && go run ./cmd/meshtool make-ca-pool-secret --namespace mesh-controller --name service-dns-ca-pool --ca-id 1)
(cd mesh_example && go run ./cmd/meshtool make-ca-pool-secret --namespace mesh-controller --name spiffe-ca-pool --ca-id 1)
Enter fullscreen mode Exit fullscreen mode

Check secrets created:

kubectl get secrets -n mesh-controller

# NAME                  TYPE     DATA   AGE
# service-dns-ca-pool   Opaque   1      9s
# spiffe-ca-pool        Opaque   1      9s
Enter fullscreen mode Exit fullscreen mode

With secrets available as prerequisite, we can deploy the mesh controller:

kubectl apply -f ./mesh_example/controller-manifests

# namespace/mesh-controller configured
# clusterrole.rbac.authorization.k8s.io/meshtool-signer created
# clusterrolebinding.rbac.authorization.k8s.io/meshtool-is-a-meshtool-signer created
# role.rbac.authorization.k8s.io/meshtool-controller created
# rolebinding.rbac.authorization.k8s.io/meshtool-is-a-meshtool-controller created
# deployment.apps/mesh-controller created
Enter fullscreen mode Exit fullscreen mode

The command above will create a namespace mesh-controller and deploy the mesh-controller in it. mesh-controller is a custom controller and its current job is to:

  • Watch PodCertificateRequest resources, created by Kubernetes's kubelet with user's deployment's request
  • Sign the certificate request with desiginated CA pool injected as a secret
  • Return the signed certificate and key back to the kubelet

Check if the mesh controller is running:

kubectl get pods -n mesh-controller

# NAME                               READY   STATUS    RESTARTS   AGE
# mesh-controller-547cd7996b-blqsf   1/1     Running   0          20s
Enter fullscreen mode Exit fullscreen mode

Verify: Auto Distributed Certificate Feature on Sample Deployment

With all the prerequisites and the mesh controller in place, it's time to put it to the test. Let's deploy a sample application and verify if the certificates are automatically distributed and correctly mounted inside the pod.

First of all, let's create a namespace for our sample application:

kubectl create namespace podcert-demo

# namespace/podcert-demo created
Enter fullscreen mode Exit fullscreen mode

Next, deploy a k8s deployment:

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: spiffe-demo-deploy
  namespace: podcert-demo
spec:
  replicas: 2
  selector:
    matchLabels:
      app: spiffe-demo
  template:
    metadata:
      labels:
        app: spiffe-demo
    spec:
      serviceAccountName: default
      automountServiceAccountToken: false
      containers:
      - name: app
        image: alpine/openssl
        command: ["sh", "-lc", "sleep infinity"]
        volumeMounts:
        - name: creds
          mountPath: /var/run/tls
          readOnly: true
      volumes:
      - name: creds
        projected:
          sources:
          - podCertificate:
              signerName: row-major.net/spiffe
              keyType: ED25519
              credentialBundlePath: service.crt
              keyPath: service.key
          - clusterTrustBundle:
              name: row-major.net:spiffe:primary-bundle
              path: ca.crt
EOF

# deployment.apps/spiffe-demo-deploy created
Enter fullscreen mode Exit fullscreen mode

Note that the kubelet can notice that this deployment (or its pods) require the certificate by:

- name: creds
  projected:
    sources:
    - podCertificate:
        signerName: row-major.net/spiffe
        keyType: ED25519
        credentialBundlePath: service.crt
        keyPath: service.key
    - clusterTrustBundle:
        name: row-major.net:spiffe:primary-bundle
        path: ca.crt
Enter fullscreen mode Exit fullscreen mode

The mesh-controller by default is watching the signerName row-major.net/spiffe and row-major.net/service-dns. If we ever build our own signer, we can simply change the singerName in the deployment manifest.

Finally, let's check the certificate and key are mounted on /var/run/tls inside the pod:

kubectl exec -it deploy/spiffe-demo-deploy -n podcert-demo -- ls -al /var/run/tls

# total 4
# drwxrwxrwt    3 root     root           140 Apr  6 02:46 .
# drwxr-xr-x    1 root     root          4096 Apr  6 02:46 ..
# drwxr-xr-x    2 root     root           100 Apr  6 02:46 ..2026_04_06_02_46_52.3629103651
# lrwxrwxrwx    1 root     root            32 Apr  6 02:46 ..data -> ..2026_04_06_02_46_52.3629103651
# lrwxrwxrwx    1 root     root            13 Apr  6 02:46 ca.crt -> ..data/ca.crt
# lrwxrwxrwx    1 root     root            18 Apr  6 02:46 service.crt -> ..data/service.crt
# lrwxrwxrwx    1 root     root            18 Apr  6 02:46 service.key -> ..data/service.key
Enter fullscreen mode Exit fullscreen mode

Check the certificate auto distributed:

kubectl exec -it deploy/spiffe-demo-deploy -n podcert-demo \
  -- openssl x509 -in /var/run/tls/service.crt -noout -text

# Certificate:
#     Data:
#         Version: 3 (0x2)
#         Serial Number:
#             02:20:72:93:62:05:37:22:fe:41:4f:ad:3f:ce:c5:bd:54:31:9e:0c
#         Signature Algorithm: ED25519
#         Issuer:
#         Validity
#             Not Before: Apr  6 02:44:52 2026 GMT
#             Not After : Apr  7 02:44:52 2026 GMT
#         Subject:
#         Subject Public Key Info:
#             Public Key Algorithm: ED25519
#                 ED25519 Public-Key:
#                 pub:
#                     0e:bb:f2:f7:fc:ee:5f:21:83:c3:87:da:3c:eb:79:
#                     dd:fe:5c:41:7e:90:cb:aa:e0:96:a8:64:e0:c5:1c:
#                     11:90
#         X509v3 extensions:
#             X509v3 Key Usage: critical
#                 Digital Signature
#             X509v3 Extended Key Usage:
#                 TLS Web Client Authentication, TLS Web Server Authentication
#             X509v3 Basic Constraints: critical
#                 CA:FALSE
#             X509v3 Subject Alternative Name: critical
#                 URI:spiffe://cluster.local/ns/podcert-demo/sa/default
#     Signature Algorithm: ED25519
Enter fullscreen mode Exit fullscreen mode

Let's see the root CA with one year validity:

kubectl exec -it deploy/spiffe-demo-deploy -n podcert-demo \
  -- openssl x509 -in /var/run/tls/ca.crt -noout -text

# Certificate:
#     Data:
#         Version: 3 (0x2)
#         Serial Number:
#             70:f5:36:4e:2b:6f:9e:fc:fb:2d:cf:5d:32:77:65:a5:fe:5e:08:64
#         Signature Algorithm: ED25519
#         Issuer:
#         Validit
#             Not Before: Apr  6 02:39:27 2026 GMT
#             Not After : Apr  6 02:39:27 2027 GMT
#         Subject:
#         Subject Public Key Info:
#             Public Key Algorithm: ED25519
#                 ED25519 Public-Key:
#                 pub:
#                     87:6d:54:67:8f:0c:d7:ed:5a:e4:a5:fd:a0:fb:81:
#                     7d:e2:7e:84:44:dc:5f:2f:99:12:63:b4:08:2c:b0:
#                     0b:5b
#         X509v3 extensions:
#             X509v3 Key Usage: critical
#                 Digital Signature, Certificate Sign
#             X509v3 Basic Constraints: critical
#                 CA:TRUE
#             X509v3 Subject Key Identifier:
#                 A0:96:CE:C1:73:8E:63:5A:26:C2:0B:BE:45:46:4D:25:50:C4:7E:EB
#     Signature Algorithm: ED25519
Enter fullscreen mode Exit fullscreen mode

This is it! The kubelet automatically created a PodCertificateRequest and the mesh-controller signed it and returned the signed certificate and key back to the kubelet. The kubelet then mounted the certificate and key as a projected volume in the pod.

What's next?

The mesh-controller sample is a really good starting point to understand how the PodCertificateRequest works. However, if we want to generate a certificate signed by external CA, how can we do this? The next step is to create a controller that does detect the PodCertificateRequest, and instead of signing by itself with secret injected CA, it forwards the request to the external authentication/authorization service (like Athenz), and returns the signed certificate and key back to the kubelet.

Closing

If you enjoyed this deep dive, please leave a like & subscribe for more!

Also, leave comments if you have any questions or suggestions. Thank you in advance!

like_this_photo_cat

Top comments (0)