The perfect combination for Azure Key Vault integration with AKS (External Secret Operator + Reloader + Workload Identity)

Umut — Mon, 17 Feb 2025 19:13:27 +0000

Hello everyone !

I will show a quick demonstration of how we can use this integration with simple steps. I believe this is one of the most secure and easy to use method to read secrets from Azure key vault. In particular, if you're using IaC tools and creating SQL servers, app registrations and so on, you're passing secrets to the Key Vault during creation and reading them in kubernetes. There are several other methods and possibilities. But we won't compare them in this article.

Let's get started

Essentially, these are the requirements for this example,

Azure Kubernetes Service with Workload Identity enabled
External Secret Operator
Reloader

What are these tools ? Here are the official for deep dive links.

The flow will be like this,

to make it more secure, private endpoint and internal ci/cd agent should be used. I will be acting as an agent in the article.

All of this will be documented as code on this GitHub repo as well,

https://github.com/Learn4Ops/esowir

git clone https://github.com/Learn4Ops/esowir
cd esowir/infrastracture
az login --use-device-code
terraform init
terraform apply --auto-approve -var "subscription_id=xxxxx" -var "tenant_id=xxxxx"

Subscription id and tenant id are required

The key point is to manage the subject part for managed identity,

subject = "system:serviceaccount:testapp:external-secret-service-account"

And this managed Identity will need 'Get' secrets permission.

We will create service account and use the namespace as above.

Let's connect kubernetes and install required objects.

az aks get-credentials --resource-group aks-resource-group --name aks-cluster --overwrite-existing

We will work on these namespaces,

kubectl create ns external-secrets
kubectl create ns testapp

Helm Charts,

ESO,
helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets -n external-secrets

Reloader,
helm repo add stakater https://stakater.github.io/stakater-charts
helm install reloader stakater/reloader -n external-secrets

Key vault secrets should exist (terraform already created), 'test-password' , 'test-username'

Okey, now everything is ready to test the flow, with the following command we will install each kubernetes object, so let's check it first.

Create a service account to use as the workload identity
Create a CRD named SecretStore that will use the service account and connect to Azure Key Vault
Create CRD named ExternalSecret that will create kubernetes secret using SecretStoreRef
Create deployment which will use kubernetes secret

kubectl apply -f ../application

Check service account, If you are following don't forget to change related fields

kubectl get sa -n testapp

You can check SecretStore health, Detailed API guide: https://external-secrets.io/latest/api/secretstore/

kubectl get secretstore -n testapp

Also External Secret, Detailed API guide: https://external-secrets.io/latest/api/externalsecret/

kubectl get externalsecret -n testapp

Looks good, so secret should be created

kubectl get secret -n testapp

Then I will check the secrets values from the pod,

kubectl exec -it -n testapp testapp-deployment-65f7dc494c-xztjw -- sh
echo $TEST_USERNAME
or
echo $TEST_PASSWORD

Same as azure key vault values,

What about changing the secret? Reloader takes over here with the annotation adjustment,

After the creation of new value,

Then wait 30 seconds because the interval is set to this value.

Amazing ! I don't need to worry about new secrets for rollout deployments.

Let's clear the environment.
terraform destroy --auto-approve -var "subscription_id=xxxxx" -var "tenant_id=xxxxx"

I mentioned earlier that to make it more secure, you can use Microsoft's internal network, Kubernetes should access KeyVault through a private endpoint, and creating the secret can be managed on the Azure side with an agent deployed in the container or virtual machine.

Let's recap important advantages,
-> Workload identity is really great way to use Azure resources, in our case we didn't give any secret to use service account as identity and we filter related service account with namespace option.
-> External Secret Operator has a lot features, everything is documented on official site, not so complex to use and define configuration.
-> Reloader is very handy, if you have a lot of pods using secret and need to refresh this value, no worries, you just wait for new secret to restart pods automatically.

Register Azure DevOps Agents with Service Principal Secret !

Umut — Mon, 25 Nov 2024 20:23:20 +0000

Here are the short instructions to add Azure DevOps agents using an app registration secret instead of your PAT!

As a standard you can register your agent by following this documentation

However, when it comes to using the Service Principal (SP) option with the script, things get a little bit more complex.

These other documentations ( 1, 2 ) are very useful, but I just want to provide a clear example of how to use it.

Thanks to these comments on GitHub and Developer Community, I figured out how to use the Service Principal.

Basically, it will look like this when using the script,

./config.sh --unattended \
  --agent "${AZP_AGENT_NAME:-$(hostname)}" \
  --url "${AZP_URL}" \
  --auth "SP" \
  --clientid "yourclientid") \
  --clientsecret "yourclientsecret") \
  --tenantid "yourtenantid") \
  --pool "${AZP_POOL:-Default}" \
  --work "${AZP_WORK:-_work}" \
  --replace \
  --acceptTeeEula & wait $!

Note: The auth parameter should be set to "SP", and you need to provide clientid, clientsecret, and tenantid.

If you are using agent in Docker, you'll also need to adjust the Dockerfile. Here are the parts that should be removed or modified,

if [ -z "${AZP_TOKEN_FILE}" ]; then
if [ -z "${AZP_TOKEN}" ]; then
echo 1>&2 "error: missing AZP_TOKEN environment variable"
exit 1
fi

AZP_TOKEN_FILE="/azp/.token"
echo -n "${AZP_TOKEN}" > "${AZP_TOKEN_FILE}"
fi

~~unset AZP_TOKEN~~

~~./config.sh remove --unattended --auth "PAT" --token $(cat "${AZP_TOKEN_FILE}") && break~~

~~export VSO_AGENT_IGNORE="AZP_TOKEN,AZP_TOKEN_FILE"~~

AZP_AGENT_PACKAGES=$(curl -LsS \
-u user:$(cat "${AZP_TOKEN_FILE}") \
-H "Accept:application/json" \
"${AZP_URL}/_apis/distributedtask/packages/agent?platform=${TARGETARCH}&top=1")

~~AZP_AGENT_PACKAGE_LATEST_URL=$(echo "${AZP_AGENT_PACKAGES}" | jq -r ".value[0].downloadUrl")~~

./config.sh --unattended \
--agent "${AZP_AGENT_NAME:-$(hostname)}" \
--url "${AZP_URL}" \
--auth "PAT" \
--token $(cat "${AZP_TOKEN_FILE}") \
--pool "${AZP_POOL:-Default}" \
--work "${AZP_WORK:-_work}" \
--replace \
--acceptTeeEula & wait $!

Important Notes:

Pay attention to the VSO_AGENT_IGNORE variable, which helps prevent the client secret from being visible in Azure DevOps (and on the portal). This provides a more secure description of the agent in the portal.
Be sure to remove the lines associated with the personal access token (PAT) and adjust the configuration for the Service Principal authentication.

DEV Community: Umut

The perfect combination for Azure Key Vault integration with AKS (External Secret Operator + Reloader + Workload Identity)

Register Azure DevOps Agents with Service Principal Secret !