Purpose of this post
The purpose of this post is to show you how to access secrets from AKS cluster that are stored in Azure Key Vault.
In one of my previous blog posts, i have shown how to access keys from Key vault from Azure DevOps, where i have configured the release pipeline to fetch the secret from key vault and substitute it during runtime for the pipeline.
we have many other ways of accessing keys from key vault from any of the Azure resources we deploy, Using managed identities is one of the secure and easy ways to access to keep our app secure.
What are Managed Identities?
There are a lot of posts that are there which helps you understand what a managed identity is. If you go to Microsoft docs, here is the definition of managed identities you will get.
"Managed identities provide an identity for applications to use when connecting to resources that support Azure Active Directory (Azure AD) authentication. Applications may use the managed identity to obtain Azure AD tokens. For example, an application may use a managed identity to access resources like Azure Key Vault where developers can store credentials in a secure manner or to access storage accounts."
Definition credits: Microsoft docs
In simple words, any azure resource that supports azure ad authentication can have managed identities. Once we enable managed identity for an azure, a service principal will be created in active directory on behalf of that azure resource you create.
With this, you can grant access to the an Azure resource that has managed identity enabled on the target azure resource you want to access.
For example, if you want to have a webapp access your key vault, all you need to do is to enable managed identity on your webapp and grant access to the managed identity of your webapp in the access policies of the key vault.
Without managed identities, in the above mentioned scenario you would need a service principal and a client secret to be created for your application (webapp in above scenario), and that service principal has to be granted permission on the target azure resource (key vault in above scenario). You need to configure your webapp to use the client id and secret to make calls to key vault to fetch the secrets.
You would have to manage the client id and secret by yourself. Incase if the service principal credentials are compromised, you need to change the secret every time and update the application code to consume the new secret. This is not only a bit insecure, but also tedious to update client secrets in multiple places.
Managed Identities to rescue
With managed identities you no longer have to create a service principal for your app, but when the feature is enabled on the azure resource, it will not only create an SP for you, but also it would manage rotation of keys by itself. You no longer need to keep client id and client secret of your service principal in your source code to access the target resource.
Kindly note that we are removing the burden of maintaining the service principal credentials in your code.
But you still need to have appropriate libraries and respective code to access the target resource. For example, if your app is going to access key vault and if the app is going to be on a webapp with managed identity enabled, you no longer need to pass the service principal credentials to call the key vault api endpoint. You can call the key vault api endpoint directly from your webapp as it has managed identity enabled and that managed identity is granted permission on the key vault.
With that, Lets dive into some demo.
Here are the steps we are going to follow:
- Create an AKS cluster
- Enable managed identity to AKS cluster
- Create a key vault with a secret in it.
- Enable access to managed identity of AKS via access policies in key vault.
- Access the secret in the key vault from a Pod in AKS.
We are going to create 2 resources in this demo.
- AKS Cluster
- Azure Key Vault
In this demo, i have created a sample AKS cluster using following commands after i have logged in to Azure from my azure cli
az group create --name yourresourcegroupname --location uksouth
az aks create -g yourresourcegroupname -n MyAKS --location uksouth --generate-ssh-keys
As we are discussing about managed identities and not about AKS, the above should suffice for creating an AKS cluster.
Once AKS cluster is created, you should see a new resource group created with name "MC_" this is for the underlying resource your AKS cluster needs to function.
Once its created, click on the VMSS that was created for your AKS cluster.
Once in the VMSS blade, click on the identity and notice that we have option for System Assigned managed identity.
Enable it by click on "on".
Once its enabled, you should see a new managed identity resource created in the "MC_" resource group.
Next create and Azure key vault resource and secret in it.
az group create --name "yourresourcegroupname" -l "locationofyourresorucegroup"
I have created below key vault and a secret.
As of now, we have created an AKS cluster, enabled system assigned managed identity and created a Key Vault with a new secret in it.
Next, we are going to add permission to AKS to access key vault. To do so, go to access policies of Key vault and click on "Add access policy" option.
Select "secret management" in the configure from template option.
Note that i have selected "secret management" for the sake of this POC. In real production environment, get, list permissions should be enough.
In the "Select principal" option, click on "none selected" to select one and choose "AKS Service principal" Object ID and "add".
You should see the access policy added in the list of access policies and click 'save'.
Once done, connect to AKS Cluster using below commands
az aks get-credentials --resource-group yourresourcegroupname --name youraksclustername --overwrite-existing
Once done, spin up an nginx pod using below commands.
use following command to login to the pod interactively
kubectl exec -i -t nginx --container nginx -- /bin/bash
To access the secret key in azure key vault, we need to hit the api to obtain the access token as described in this document.
Once the token is obtained, you can access the secret key value in the key vault using below command.
curl 'https:///secrets/?api-version=2016-10-01' -H "Authorization: Bearer "
This way we can access the values in the key vault from AKS with managed identities enabled.
In this blog post, we have seen what managed identies are in a nut shell and seen how to enabled managed identity for AKS cluster and access the key vault from AKS cluster with help of access policy granted to managed identity of AKS cluster.
System Assigned managed identity lives as long as the resource is in Azure. Once the resource is deleted, the corresponding managed identity and its service principal are also deleted from Azure AD.
We also have whats called an User identity which exists even after a resource is deleted and you can assign it to one or more instances of an Azure service. In the case of user-assigned managed identities, the identity is managed separately from the resources that use it and you are responsible for cleaning it up after use.
Hope you enjoyed reading this blog post.
Thanks for reading!!
Top comments (5)
Hey. This was pretty clear compared to other posts I have seen on this topic.
Does this work for RBAC instead of Vault access policies?
Hi Sorry for the late reply, yes it does work with RBAC roles as well. whether its key vault polices or RBAC is a way of accessing the key vault. So, accessing the secrets should work with either of the approaches as long as the MI has access to AKV.
Thanks @vivekanandrapaka for the detailed article. We have pretty much done the same thing and used the AKS's system assigned managed identity and provided it access to the key vault, however our pods got error in CrashLoop, which to rectify further we used the VMSS system assigned identity and then the deployment was successful. However the secret used in key vault is a Azure SQL server connection string which our deployed code using DefaultAzureCredential class is not able to pick up the login.. Do we have to assign the VMSS id to sql server as well or if anything we are missing here. please suggest
Hi Chetan, Sorry for the late response. If the connection string is stored in AKV and if you are able to fetch it via the POD, then connectivity between AKV and POD is established, if you are unable to connect to SQL server via connection string, i suggest you to check if the connection string stored in AKV has proper syntax, you can test it manually from your local machine...
The thing is it works fine in local ,as the managed identity fetches the credentials via either the Visual Studio IDE or local AZ CLI's exe or the least used environment variables. However when deployed the same on AKS , even after providing the access for AKS's system assigned managed identity to access the key vault, deployment got failed and then we read somewhere to use the VMSS's identity which made it work.. Microsoft recommends to use POD level security or some AKS CSI driver, however i still think the simple way should be AKS's system assigned identity connecting to AKV and further connecting to the Azure SQL server should resolve the issue. The issue with VMSS seems to be when auto-scale kicks in this would create a problem? or it should remain as it is with no issues? To summarize should we use the VMSS's managed id or the cluster's managed ID