(English) Configuring AKS to read secrets and certificates from Azure KeyVaults

Introduction

While building a kubernetes cluster to deploy your containers in there you will find that most of the times you need to pass information to those containers for them to work properly, this can be a connectio nstring to a certain resource (databases, redis cache, storage accounts, etc).
Usualy you will define an environment variable for the container and will set the value when running the deployment to kubernetes, this works just fine but the downside is that those values are visible to anyone who does a "describe" of the pod containing the containers.

This is a problem when you consider to version your yaml files, this will means that the values of those environment variables will be placed in the repository and more importantly if you deploy to multiple environments where you have different values for those secrets you can not have them hardcoded in your yaml file, instead, you need them to be dinamicaly adjusted to the environment where you are deploying them.

In this article you will learn how to configure an AKS cluster (Microsoft Azure offer for kubernetes) to consume secrets, keys and certificates from an Azure KeyVault (secure store offer from Microsoft Azure).

Initial setup

Infrastructure resources

The first thing you will need of course is the AKS cluster itself and a keyvault to store your sensitive data.

az aks create --resource-group "aks-demo" --name "aks-cluster" --network-plugin azure --enable-managed-identity

With this command you will create an AKS cluster called "aks-cluster" in the resource group "aks-demo". An important thing to note is the "--enabled-managed-identity" flag, this will create a managed identity that the cluster will use to manage it's interaction with Azure, this is needed for this whole article to work.

Then we will create a keyvault

az keyvault create --location westus2 --name "aks-keyvault" --resource-group "aks-demo"

Here we are creating the keyvault called "aks-keyvault" in the same resource group as our AKS cluster (please note that in that resource group you will only see the AKS resource but all the needed infrastructure for that cluster will be built in another resouce group which name starts with "MC_" this can be changed when creating the aks cluster if needed). We are choosing a location to deploy our keyvault as "westus2" but this is not mandatory to be the location.

Now we need to configure our access to the AKS cluster and we will do this using the Azure Cli as well!!

az aks install-cli

This will install kubectl which is a tool to manage our aks/k8s cluster from the command line.

Next we need to setup our kubectl to access our AKS cluster

az aks get-credentials --resource-group "aks-demo" --name "aks-cluster"

At this point we can test if the connection from kubectl to the AKS cluster is correct by simply running a "kubectl get nodes"

IMAGE getnodes.png

Configuring the AKS cluster

This whole solution works by adding a couple of demonsets and pods that will manage the access to the keyvault to retrieve the secreta/keys/certs and place them inside the pods you choose, thankfully we can do this by installing some helm charts in our AKS.

First we need to install the repos and the charts

helm repo add secrets-store-csi-driver https://raw.githubusercontent.com/kubernetes-sigs/secrets-store-csi-driver/master/charts

helm install csi-secrets-store-provider-azure/csi-secrets-store-provider-azure --generate-name

This will build a demonset and you will see it with this output (this was the name in my machine but will be different in your case as the name is autogenerated because of the --generate-name)

NAME: csi-secrets-store-provider-azure-1594729889
LAST DEPLOYED: Tue Jul 14 14:31:29 2020
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None

Our next step is to build a set of resources in our cluster that will manage the integration between the cluster, azure and the kevault it self, this set of resources will notice when a change is done in the pods (a redeploy) and will take care of going to the keyvault and extract the values to populate your pods configurations, they will also manage the selection of which secret/key or cert to extract from the keyvault.

kubectl apply -f https://raw.githubusercontent.com/Azure/aad-pod-identity/master/deploy/infra/deployment-rbac.yaml

the output should look something similar to this

serviceaccount/aad-pod-id-nmi-service-account created
customresourcedefinition.apiextensions.k8s.io/azureassignedidentities.aadpodidentity.k8s.io configured
customresourcedefinition.apiextensions.k8s.io/azureidentitybindings.aadpodidentity.k8s.io configured
customresourcedefinition.apiextensions.k8s.io/azureidentities.aadpodidentity.k8s.io configured
customresourcedefinition.apiextensions.k8s.io/azurepodidentityexceptions.aadpodidentity.k8s.io configured
clusterrole.rbac.authorization.k8s.io/aad-pod-id-nmi-role created
clusterrolebinding.rbac.authorization.k8s.io/aad-pod-id-nmi-binding created
daemonset.apps/nmi created
serviceaccount/aad-pod-id-mic-service-account created
clusterrole.rbac.authorization.k8s.io/aad-pod-id-mic-role created
clusterrolebinding.rbac.authorization.k8s.io/aad-pod-id-mic-binding created
deployment.apps/mic created

Now we will need a managed identity (User assigned) that will be used to read the values from our keyvault, we can do this using the azure cli as follows

az identity create -g "aks-demo" -n aksownidentity

The output will show you several lines of information about the configuration of this identity but the ones you will use in the next steps are the one called "principalId" and the "clientID"

With this identity created we will continue to assign the "Reader" role at the keyvault level

az role assignment create --role Reader --assignee "PRINCIPALID OF IDENTITY" --scope /subscriptions/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX/resourcegroups/aksdemo/providers/Microsoft.KeyVault/vaults/aks-keyvault

Next we need to provide access to the identity to read secrets, keys and certificates, you don't need to include the tree, if you only need to read secrets, only add the "get" permission to "secrets"

# set policy to access keys in your keyvault
az keyvault set-policy -n aks-keyvault --key-permissions get --spn CLIENT_ID

# set policy to access secrets in your keyvault
az keyvault set-policy -n aks-keyvault --secret-permissions get --spn CLIENT_ID

# set policy to access certs in your keyvault
az keyvault set-policy -n aks-keyvault --certificate-permissions get --spn CLIENT_ID

After this step all the infrastructure permissions and deployments are done, you only need to provision some items inside your kubernetes cluster and you are all set, let's see those yaml files

apiVersion: "aadpodidentity.k8s.io/v1"
kind: AzureIdentity
metadata:
  name: aksuseridentity
spec:
  type: 0
  resourceID: /subscriptions/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX/resourcegroups/aksdemo/providers/Microsoft.ManagedIdentity/userAssignedIdentities/aksownidentity
  clientID: CLIENT ID OF YOUR IDENTITY

Store this yaml as "aadpodidentity.yaml" and execute it in your aks with

kubectl apply -f aadpodidentity.yaml

Please note the name of this resource, you will need it in the next yaml file

apiVersion: "aadpodidentity.k8s.io/v1"
kind: AzureIdentityBinding
metadata:
  name: aksidentitybinding
spec:
  azureIdentity: aksuseridentity
  selector: aksidentitybindingselector

• Note in the azureIdentity we are using the same as the name of our previous resource "aksuseridentity" in this example.

• The selector is the tag with which you will identify this resource when accesing the keyvault from a pod.

Store this file as "aadpodidentitybinding.yaml" and do

kubectl apply -f aadpodidentitybinding.yaml

Now our last step is to deploy a pod and consume a secret from our keyvaul, first go to your keyvault in Azure and create a secret, a key and a certificate (depending on which permissions you granted previuosly you might want only to create the one you provided access for)

I will be using a simple linux container running alpine to present the skeleton of how should look our deployment definition to consume the keyvault but please feel free to use the one that fits the best for you

apiVersion: apps/v1
kind: Deployment
metadata:
  name: alpine
  labels:
    app: alpine
spec:
  replicas: 1
  template:
    metadata:
      name: alpine
      labels:
        app: alpine
        aadpodidbinding: "aksidentitybindingselector"
    spec:
      nodeSelector:
        "beta.kubernetes.io/os": linux
      containers:
      - image: nginx
        name: nginx
        volumeMounts:
        - name: secrets-store-inline
          mountPath: "/mnt/secrets-store"
          readOnly: true
      volumes:
        - name: secrets-store-inline
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              providerName: "azure"
              usePodIdentity: "true"          # [OPTIONAL] if not provided, will default to "false"
              keyvaultName: "aks-keyvault"                # the name of the KeyVault
              cloudName: ""          # [OPTIONAL for Azure] if not provided, azure environment will default to AzurePublicCloud
              objects:  |
                array:
                  - |
                    objectName: secret1
                    objectType: secret        # object types: secret, key or cert
                    objectVersion: ""         # [OPTIONAL] object versions, default to latest if empty
              resourceGroup: "aks-demo"               
              subscriptionId: "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX"              # [REQUIRED for version < 0.0.4] the subscription ID of the KeyVault
              tenantId: "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX"                    # the tenant ID of the KeyVault
  selector:
    matchLabels:
      app: alpine

Again store this yaml as "mypod.yaml" and execute

kubectl apply -f mypod.yaml

Some details of this last yaml

• The "aadpodbinding" should be the same as the "selector" you put in the aadpodidentitybinding.yaml file

• The mountPath is the place where the secrets, keys and certificates will be stored, they will be represented as files in that directory

• For each item (secret, key or certificate) that you want to access from the keyvault you will need an entry in "objects"

• The tenantId is the directory id where you have the subscription that contains the keyvault.

After you complete this steps you can go inside your container and check that directory (/mnt/secrets-store) to verify if your secret is in there.

kubectl exec -it PODNAME /bin/bash
ls -l /mnt/secrets-store

you should see one file per secret/key/certificate that you defined in the objects in the deployment yaml that you executed.

An important note is that if you added a new item in the keyvault, it will be only visible if you add it in the objects in the deployment and redeploy it also if you change the value of the secret in the keyvault, you need to force a redeploy in kubernetes to read the new value, this can be done adding a label in the deployment that you can change and apply the deployment again to aks, this will force a reconfiguration in kubernetes that will update the value of the file inside the pods.

References and documentation

I decided to create this article as I found there is a lot of information on this topic but it took me some time to understand how to configure everything properly to make it to work, it is not complicated but some times the documentation links other documentation and following that chain is complex so I decided to put together the minimum needed steps to make this work (also tested all of them myself)

I will place here the links to the documentation of each part that makes this happen as the people working in this deserves the kudos and love, please go and share your love with them

LINKS

Main article explaining how all this works

secrets-store-csi-driver installation

Deploy your aks in Azure

Final notes

If at any step of this guide you can't continue or find something confussing, please don't hesitate to let me know and reach back to me I will try to help you the best as I can, we are all in the same boat constantly learning ;)

Thank you for reading the article and if you like it or found it useful please let me know with the hearts in this article so I know what to write next.

Have a wonderful day!!