Azure Pipelines secret management with Key Vault

Summary

In this article, I would like to introduce an example to input secrets as variables for the IaC pipeline in Azure Pipelines and then keep the secrets solely in Azure Key Vault securely. Once you deploy the Azure resources, you do not need to keep the secrets as Pipeline Variable any more. This makes your secret management easier and more effective.

Sample code: api-management-sample

Context

Azure Pipeline Library allow you to use Variable Group to input secrets to the IaC pipeline. You can Link secrets from an Azure key vault to secure them. However, obviously you need a Key Vault in the first run that stores secrets, and it does not work for setting secrets in for the first time run of IaC pipeline.

This does not work

Another option is Azure Pipeline Secret Variable where you can securely set secrets for the IaC pipeline first run. However, when you run the IaC pipeline the second time or later for the purpose of infrastructure configuration, you still need the same secret in the Pipeline Secret Variable. This is not what I want to do because you have to keep the secrets in two locations, the Key Vault and Pipeline Secret Variable for the system operation. The most secure and effective secret management is to store secrets solely in the Key Vault for the operation.

Keeping secrets in two locations is not what I want to do

What I want to do is to set secrets in Pipeline Secret Variable for the first time run of IaC deployment, and to remove the secrets from Pipeline Secret Variable and store them only in the Key Vault. The IaC pipeline faces errors without secret inputs from Pipeline Secret Variable for the second run. What I need to make this work for the second run is to download the existing secrets from the Key Vault and set them as Azure Pipeline variables.

This is what I want to do

IaC pipeline

I am trying to explain how to make this work with the sample code api-management-sample. For the sample code, you can refer to those articles.

Sequence

Key Vault stage

Two secret inputs of BASIC_AUTH_PASS and CLIENT_SECRET have to be set as Azure DevOps Pipeline Secret Variable.
In Key Vault stage, it uses for each statement for the repeatable tasks with two secret variables.
Parameters for the repeatable tasks have name, secret, kvsecretname.
secret is important to set as parameter so the pipeline can dynamically use the secret value in the powershell script.
In the powershell script, if ( '${{ PipelineSecret.secret }}' -like '*${{ PipelineSecret.name }}*' ) sees if secrets exist in Pipeline Secret Variable. This syntax is tricky, but if Pipeline Secret Variable does not exist, PipelineSecret.secret becomes $(BASIC_AUTH_PASS) or $(CLIENT_SECRET). That is why like '*${{ PipelineSecret.name }}*' syntax is used.
Write-Host "##vso[task.setvariable variable=$env:SecretVariableName;issecret=true;isOutput=true]$secretValue" sets secrets as variables in this pipeline, which can be used in the later stage. issecret=true makes the variable as secure string, and isOutput=true allows to be used in the later stage.

iac_pipeline_subscription.yml

parameters:
- name: PipelineSecrets
  type: object
  default:
    - name: BASIC_AUTH_PASS
      secret: $(BASIC_AUTH_PASS)
      kvsecretname: $(KVSECRET_NAME_BASIC_PASS)
    - name: CLIENT_SECRET
      secret: $(CLIENT_SECRET)
      kvsecretname: $(KVSECRET_NAME_CLIENTSECRET)

stages:
- stage: KeyVault
  jobs:
  - ${{ each PipelineSecret in parameters.PipelineSecrets }}:
    - job: ${{ PipelineSecret.name }}
      variables:
        - name: SecretVariableName
          value: ${{ PipelineSecret.name }}
      steps:
      - task: AzurePowerShell@5
        name: ${{ PipelineSecret.name }}
        displayName: Set ${{ PipelineSecret.name }} variable
        env:
          SecretValue: ${{ PipelineSecret.secret }}
        inputs:
          azureSubscription: $(AZURE_SVC_NAME)
          azurePowerShellVersion: latestVersion
          ScriptType: InlineScript
          Inline: |
            if ( '${{ PipelineSecret.secret }}' -like '*${{ PipelineSecret.name }}*' ){
              $secretValue = Get-AzKeyVaultSecret -VaultName $env:KeyVaultName -Name '${{ PipelineSecret.kvsecretname }}' -AsPlainText
              Write-Host "##vso[task.setvariable variable=$env:SecretVariableName;issecret=true;isOutput=true]$secretValue"
              Write-Host "Input variable $env:SecretVariableName does not exists. $secretValue is set for the later stage."
            }
            else{
              Write-Host "##vso[task.setvariable variable=$env:SecretVariableName;issecret=true;isOutput=true]'${{ PipelineSecret.secret }}'"
              Write-Host "Input variable $env:SecretVariableName already exists. Use this variable for the later stage."
            }

Bicep stage

This stage needs value: $[ stageDependencies.KeyVault.BASIC_AUTH_PASS.outputs['BASIC_AUTH_PASS.BASIC_AUTH_PASS'] ] to use variables set in the previous stage. You can refer the syntax to Use output variables from tasks
Secret variables should be set with env:. The syntax is described in Set secret variables

iac_pipeline_subscription.yml

- stage: Deploy
  jobs:
  - job: AzureResourceGroupDeployment
    timeoutInMinutes: 0
    variables:
      - name: BasicAuthPass
        value: $[ stageDependencies.KeyVault.BASIC_AUTH_PASS.outputs['BASIC_AUTH_PASS.BASIC_AUTH_PASS'] ]
      - name: ClientSecret
        value: $[ stageDependencies.KeyVault.CLIENT_SECRET.outputs['CLIENT_SECRET.CLIENT_SECRET'] ]
      - name: CertThumbprint
        value: $[ stageDependencies.KeyVault.Certificate.outputs['TaskCertificate.CertThumbprint'] ]
    steps:
    - task: AzureCLI@2
      displayName: Deploy Azure resources
      env:
        BasicAuthPass: $(BasicAuthPass)
        ClientSecret: $(ClientSecret)
        CertThumbprint: $(CertThumbprint)
      inputs:
        azureSubscription: $(AZURE_SVC_NAME)
        scriptType: ps
        scriptLocation: inlineScript
        inlineScript: |
          az group create --name $(ResourceGroupName) --location $(LOCATION)
          az deployment group create --resource-group $(ResourceGroupName) --template-file $(BicepFilePath) `
            --parameters $(BicepParameterFilePath) `
            base_name=$(BASE_NAME) `
            environment_symbol=$(ENVIRONMENT_SYMBOL) `
            aad_objectid_svc=$(AAD_OBJECTID_SVC) `
            aad_appid_client=$(AAD_APPID_CLIENT) `
            aad_appid_backend=$(AAD_APPID_BACKEND) `
            aad_tenantid=$(AAD_TENANTID) `
            basic_auth_user=$(BASIC_AUTH_USER) `
            kvcert_name_api=$(KVCERT_NAME_API) `
            kvsecret_name_basic_pass=$(KVSECRET_NAME_BASIC_PASS) `
            kvsecret_name_cert_thumbprint=$(KVSECRET_NAME_CERT_THUMBPRINT) `
            kvsecret_name_clientsecret=$(KVSECRET_NAME_CLIENTSECRET) `
            kvsecret_name_subscription_key=$(KVSECRET_NAME_SUBSCRIPTION_KEY) `
            apim_api_name_ad=$(APIM_API_NAME_AD) `
            apim_api_path_ad=$(APIM_API_PASH_AD) `
            apim_api_name_basic=$(APIM_API_NAME_BASIC) `
            apim_api_path_basic=$(APIM_API_PASH_BASIC) `
            apim_api_name_cert=$(APIM_API_NAME_CERT) `
            apim_api_path_cert=$(APIM_API_PASH_CERT) `
            basic_auth_pass=$(BasicAuthPass) `
            client_secret=$(ClientSecret) `
            cert_thumbprint=$(CertThumbprint)