DEV Community

Goodluck Ekeoma Adiole
Goodluck Ekeoma Adiole

Posted on

Encrypt secrets stored in etcd with a Key Vault key

  • AKS uses the KMS (Key Management Service) plugin to encrypt secrets stored in etcd with a Key Vault key (your CMK). See Microsoft’s KMS docs. (Microsoft Learn)
  • KMS requires a user-assigned managed identity (system-assigned identities are not supported for KMS). You must grant that identity key permissions on the Key Vault before enabling KMS. (Microsoft Learn)
  • DO NOT delete the Key or Key Vault once KMS is turned on — doing so will make secrets unrecoverable. (Microsoft Learn)

Quick variable setup (copy/paste and edit) 

# your cluster info (already provided)
RG="MiddleWare-Dev-RG"
CLUSTER="hil-middleware-staging-cluster"
LOCATION="<your-azure-location>"         # e.g. eastus2
# choose names
KV_NAME="<your-keyvault-name>"
KEY_NAME="<your-key-name>"
IDENTITY_NAME="<your-user-identity-name>"
Enter fullscreen mode Exit fullscreen mode

Step 1 — Ensure Azure CLI is new enough 

az --version   # need recent az (see docs), update if old
Enter fullscreen mode Exit fullscreen mode

Step 2 — Create Key Vault + Key (public example)

(If you prefer a private Key Vault, I list the extra args afterwards.)

# Create key vault (public network access)
az keyvault create \
  --name $KV_NAME \
  --resource-group $RG \
  --location $LOCATION

# Create an RSA key in the vault
az keyvault key create \
  --vault-name $KV_NAME \
  --name $KEY_NAME \
  --kty RSA
Enter fullscreen mode Exit fullscreen mode

Save the key id:

KEY_ID=$(az keyvault key show --vault-name $KV_NAME --name $KEY_NAME --query 'key.kid' -o tsv)
echo $KEY_ID   # you'll pass this to az aks update
Enter fullscreen mode Exit fullscreen mode

(Private vault: az keyvault create --public-network-access Disabled and you’ll need API Server VNet Integration on AKS — see docs. (Microsoft Learn))

Step 3 — Create a user-assigned managed identity (required for KMS) 

az identity create --resource-group $RG --name $IDENTITY_NAME --location $LOCATION

IDENTITY_PRINCIPAL_ID=$(az identity show --resource-group $RG --name $IDENTITY_NAME --query principalId -o tsv)
IDENTITY_RESOURCE_ID=$(az identity show --resource-group $RG --name $IDENTITY_NAME --query id -o tsv)

echo $IDENTITY_PRINCIPAL_ID
echo $IDENTITY_RESOURCE_ID
Enter fullscreen mode Exit fullscreen mode

Step 4 — Grant the identity permissions on the Key Vault

Choose the correct method depending on whether your Key Vault uses Access Policies (classic) or RBAC.

If the vault uses access policies (default):

az keyvault set-policy \
  --name $KV_NAME \
  --object-id $IDENTITY_PRINCIPAL_ID \
  --key-permissions decrypt encrypt
Enter fullscreen mode Exit fullscreen mode

If the vault uses RBAC (you created it with --enable-rbac-authorization true):

KEYVAULT_RESOURCE_ID=$(az keyvault show --name $KV_NAME --resource-group $RG --query id -o tsv)
az role assignment create \
  --assignee-object-id $IDENTITY_PRINCIPAL_ID \
  --assignee-principal-type ServicePrincipal \
  --role "Key Vault Crypto User" \
  --scope $KEYVAULT_RESOURCE_ID
Enter fullscreen mode Exit fullscreen mode

(Documentation shows both approaches — pick the one matching your Key Vault config.) (Microsoft Learn)

Step 5 — Attach the user-assigned identity to the AKS cluster

KMS requires a user-assigned identity; attach it to the control plane:

az aks update \
  --resource-group $RG \
  --name $CLUSTER \
  --enable-managed-identity \
  --assign-identity $IDENTITY_RESOURCE_ID
Enter fullscreen mode Exit fullscreen mode

You can confirm the identity was attached:

az aks show -g $RG -n $CLUSTER --query identity
Enter fullscreen mode Exit fullscreen mode

(If your cluster already used system-assigned identity, switching to / adding user-assigned can cause the control plane to reconfigure; monitor the cluster resource state via az aks show.) (Microsoft Learn)

Step 6 — Enable KMS (turn on Azure KeyVault KMS for etcd)

For a public Key Vault:

az aks update \
  --resource-group $RG \
  --name $CLUSTER \
  --enable-azure-keyvault-kms \
  --azure-keyvault-kms-key-vault-network-access "Public" \
  --azure-keyvault-kms-key-id "$KEY_ID"
Enter fullscreen mode Exit fullscreen mode

For a private Key Vault (requires API server VNet integration and vault resource id):

KEYVAULT_RESOURCE_ID=$(az keyvault show --name $KV_NAME --resource-group $RG --query id -o tsv)

az aks update \
  --resource-group $RG \
  --name $CLUSTER \
  --enable-azure-keyvault-kms \
  --azure-keyvault-kms-key-vault-network-access "Private" \
  --azure-keyvault-kms-key-id "$KEY_ID" \
  --azure-keyvault-kms-key-vault-resource-id "$KEYVAULT_RESOURCE_ID"
Enter fullscreen mode Exit fullscreen mode

After this command completes, KMS will be enabled and AKS will use that Key Vault key for encrypting secrets in etcd. See the official KMS doc for the exact flags. (Microsoft Learn)

Step 7 — Re-encrypt existing secrets (important)

Secrets created before KMS was enabled remain encrypted with the old mechanism. Run this to update them (you can scope to namespaces if desired):

kubectl get secrets --all-namespaces -o json | kubectl replace -f -
Enter fullscreen mode Exit fullscreen mode

If you have many secrets, consider doing namespace-by-namespace to reduce blast radius. The docs recommend re-replacing secrets after enabling KMS. (Microsoft Learn)

Step 8 — Verify status via CLI (recommended)

Check the AKS resource for the KMS block:

az aks show --resource-group $RG --name $CLUSTER --query "securityProfile.azureKeyVaultKms" -o json
Enter fullscreen mode Exit fullscreen mode

Expected output when enabled (example):

{
  "keyId": "https://<vault>.vault.azure.net/keys/<key>/<version>",
  "networkAccess": "Public"
}
Enter fullscreen mode Exit fullscreen mode

You can also inspect the whole cluster JSON:

az aks show -g $RG -n $CLUSTER -o json | jq '.securityProfile'
Enter fullscreen mode Exit fullscreen mode

If the azureKeyVaultKms object appears, KMS is active. (This matches guidance in the Microsoft KMS doc.) (Microsoft Learn, GitHub)

Step 9 — Verify in Azure Portal (if you still want the Portal)

  • The Portal UI may not show a dedicated "Encryption" blade for older clusters. If a blade appears for newer clusters it will be under the cluster Settings → Security/Encryption area.
  • The reliable method is CLI/JSON as above, or open the cluster resource in the Portal, click “JSON View” (or “Export template”), and inspect properties.securityProfile.azureKeyVaultKms. If present, CMK/KMS is configured. (Microsoft Learn)

Safety & operational cautions (read these)

  • Do not delete the Key or Vault while KMS is enabled — you will break access to secrets and potentially make the cluster unusable. The docs warn strongly about this. (Microsoft Learn)
  • If you rotate the key (new version), update the cluster with az aks update --enable-azure-keyvault-kms --azure-keyvault-kms-key-id $NEW_KEY_ID and re-replace secrets as noted. (Microsoft Learn)
  • If your Key Vault uses firewall rules / private endpoints, follow the API Server VNet Integration guidance in the docs before enabling KMS. (Microsoft Learn)

Helpful one-liners (copy/paste)

  • Get the current KMS block: 
az aks show -g $RG -n $CLUSTER --query "securityProfile.azureKeyVaultKms" -o json
Enter fullscreen mode Exit fullscreen mode
  • Re-encrypt secrets (all namespaces): 
kubectl get secrets --all-namespaces -o json | kubectl replace -f -
Enter fullscreen mode Exit fullscreen mode

If you want, I can:

  • produce the exact command sequence filled with specific names (I used your RG/cluster; tell me KV_NAME, KEY_NAME, IDENTITY_NAME, and LOCATION and I’ll print the full script), or
  • give the equivalent ARM template / Terraform snippet to bake this into infra-as-code.

Which one do you want me to produce next?

Top comments (0)