Bridging the Gap: Leveraging Secret Store CSI Drivers to Access Secrets from Google Secret Manager in GKE Cluster

Today, preserving and securely storing secrets like API keys, passwords, and certificates is an essential part of creating and deploying applications.

Google Secret Manager provides a powerful and centralized solution for secret management, as well as a secure vault to store and access sensitive information.

However, getting these secrets and seamlessly integrating them into applications might be difficult. This post will detail this process and demonstrate how simple it is to consume secrets from Google Secret Manager into a GKE cluster.

Before we get started there are couple of pre-requisites:

• Make sure the GKE cluster which you are creating should have WORKLOAD_IDENTITY enabled.
• Make sure you have a IAM SERVICE_ACCOUNT created with proper permissions.
• One which we are most interested in is "Secret Manager Secret Accessor(roles/secretmanager.secretAccessor). Refer this for more details.
• gcloud command line gives a great way to quickly create and destroy resources on GCP.
• In below command, replace CLUSTER_NAME with the name of the GKE cluster, PROJECT_ID with the name of the project where you want to create the GKE cluster, SERVICE_ACCOUNT with the name of the service account which contains the necessary permissions to interact with GCP resources (for example in this case it would be Google Secret Manager), MACHINE_TYPE with the size of machine to deploy your workloads since we would be installing couple of resources make sure you choose a medium machine, ZONE with the name of the zone, NUMBER_OF_NODES with an integer value stating minimum number of nodes per zone, CLUSTER_VERSION with the version of the GKE cluster you want to install.

gcloud container clusters create CLUSTER_NAME --workload-pool=PROJECT_ID.svc.id.goog --service-account=SERVICE_ACCOUNT@PROJECT_ID.iam.gserviceaccount.com --zone ZONE --machine-type MACHINE_TYPE --num-nodes=NUMBER_OF_NODES --cluster-version=CLUSTER_VERSION

gcloud container clusters create test-k8s \
    --workload-pool=proj-101.svc.id.goog \
    --service-account=[e-mail] \
    --zone us-central1-a \
    --machine-type e2-standard-4 \
    --num-nodes=1 \
    --cluster-version=1.23.17-gke.5600

• Install the Secret Store CSI Drivers, the simplest way is to get them installed via helm.
• Additionally, if you want to create secret k8s resource as well make sure to enable syncSecret flag.

helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts

helm upgrade --install csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver --namespace kube-system 

• Install the GCP provider plugin.

kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/secrets-store-csi-driver-provider-gcp/main/deploy/provider-gcp-plugin.yaml

• Make sure your service-account is binded with IAM GCP account
• Use below command to add/check IAM policy binding.
• Make sure to follow proper naming convention for service-account i.e. [NAMESPACE/SERVICE_ACCOUNT_NAME] for instance here we have [default/app-sa].

gcloud iam service-accounts add-iam-policy-binding \
    --role roles/iam.workloadIdentityUser \
    --member "serviceAccount:$proj-101.svc.id.goog[default/app-sa]" \
    [e-mail]

Once all these things are done you are ready to use the secrets from Google Secret Manager and consume them in your deployments.

• Use the same GCP IAM service account in k8s service account annotations.

---
apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    iam.gke.io/gcp-service-account: [e-mail]
  name: app-sa

• Create the k8s SecretProviderClass resource and have your path of secret appended to resourceName key (this is the one which will come from Google Secret Manager).
• In this example, I already created a secret called testpassword in GSM.

---
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: test-secret
spec:
  provider: gcp
  parameters:
    secrets: |
      - resourceName: "projects/proj-101/secrets/testpassword/versions/latest"
        fileName: "testpassword"
  secretObjects:
  - secretName: testpassword
    data:
    - key: testpassword
      objectName: testpassword
    type: Opaque

• Create the pod/deployment and use your secrets in your application.
• Here we are creating a volume item which is referring secrets-store.csi.k8s.io csi drivers and using the secretProviderClass which we created in above step.
• Further, we are mounting this as a volume to be consumed.
• Note: If you are planning to use the secret as an environment variable make sure the secret (k8s resource) is present.

---
apiVersion: v1
kind: Pod
metadata:
  name: test-pod
spec:
  serviceAccountName: app-sa
  containers:
  - image: gcr.io/google.com/cloudsdktool/cloud-sdk:slim
    name: mypod
    resources:
      requests:
        cpu: 100m
    tty: true
    volumeMounts:
      - mountPath: "/var/secrets"
        name: vol-secret
        readOnly: true
  volumes:
  - name: vol-secret
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: "test-secret"

I hope you find this post useful, feel free to add your thoughts or comments.

Happy learning!

References:

• https://docs.github.com/en/actions/deployment/deploying-to-your-cloud-provider/deploying-to-google-kubernetes-engine
• https://github.com/GoogleCloudPlatform/secrets-store-csi-driver-provider-gcp