Helm 3 - CI/CD with Azure DevOps using Azure Container Registry (ACR) and Azure Kubernetes Service (AKS)

I have been bit late to Kubernetes world, but ever since I have started using it, I have been part of a great teams building awesome applications and using Helm package manager. With Helm, you package your Kubernetes application as charts, which are then stored in Helm chart repo. Helm also has a templating engine allowing you to set values in your charts dynamically allowing you to manage your applications more easily. Azure Container Registry (ACR) currently supports publishing Helm 3 charts to ACR and it is currently in preview.

In this post we will see how we can publish a sample Helm chart to ACR and also deploy the application to Azure Kubernetes Service (AKS) by consuming the published chart from ACR. We will also use ACR’s repository scoped tokens - a preview feature which offer great benefits.<!--more-->

We will cover following things in this post.

1) Prerequisites

• ACR
  • Setting up the ACR to use repository scoped tokens
  • Create scope-maps
  • Create a token
• AKS
  • Create AKS instance on Azure

For CI and CD, I am going to use Azure DevOps YAML multi-stage pipelines in this post and our pipeline will be divided as below.

2) CI Stage

• Get the latest chart from GitHub
• Publish to ACR

3) CD Stage

• Pull the latest version of chart from ACR
• Deploy the pulled to AKS
• View the deployed service in Azure DevOps

The code I am using is on

1. Prerequisites

You can authenticate with ACR in multiple ways.

1. Using a service principal - You can create separate service principals. This gives access to whole ACR and all the repositories inside that ACR instance. More info here

2. Using managed identity - Using user-assigned or system-assigned managed identity for easily consuming charts on an Azure VM. More info here

3. Using image pull secret - For your Kubernetes (K8s) cluster (including unmanaged AKS instance), you can also define a image pull secret, which then lets K8s cluster do pull images and run the application. More info on how you can do this with AKS is here

4. Using a repository scoped token - This feature of ACR is currently in preview and only available on Premium container registry service tier. This allows you to define scope maps and repository specific tokens for your ACR.

Setting up the ACR to use repository scoped tokens

I am going to use Tokens to authenticate ourselves with ACR. The biggest advantage is that, as a registry owner, you can have a single instance of ACR across your organisation and provide access to certain teams only selected repositories. There are other interesting scenarios highlighted in the documentation, mainly…

• Allow IoT devices with individual tokens to pull an image from a repository
• Provide an external organisation with permissions to a specific repository
• Limit repository access to different user groups in your organisation. For example, provide write and read access to developers who build images that target specific repositories, and read access to teams that deploy from those repositories.

If you have an existing ACR instance (which in my case I do), the first step is to upgrade your ACR instance to use Premium tier.

Then select Premium and click Save.

Creating scope maps

Once, you are on Premium tier, you will be able to to create a tokens. But before we can create tokens, let us define two scope maps. Scope maps when associated with tokens sets permissions on what can be done on ACR.

1. chartpull - a scope limiting read permission to helmdemo/vote-app repo. I am using content/read and metadata/read permissions.
2. chartpush - a scope limiting write permission inside helmdemo/vote-app repo. I am using content/write and content/read permissions.

The idea here is that, development team pushes the charts to ACR and some other team is consuming the charts/application. But you can extend this idea to let users external to your organisation to pull images only from specific repositories.

So to create scope map, go to Scope maps under Repository permissions sections and then click + Add. In my case, I am limiting these scopes only to helmdemo repo.

I follow the same process to create another scope named chartpull scope with content/read, metadata/read scopes.

• Notice that in chartpush scope we also allow content/read permission. This is because, you need content/read scope along with content/write scope if you are going to push charts to ACR (for the repo defined in the scope).
• You can also use Azure CLI to do the same. Use az acr token command. Refer the documentation for more information.

Creating token

Once you defined scopes, you are ready to create tokens. Go to Tokens in Repository permissions section and click + Add. Then name the token (e.g. helmdemopull) and select the scope map you created above. In my case I am creating two tokens, one for pull and another one for push.

You can repeat the process to create another token. Refreshing the screen you will see something like this below.

You will notice that these tokens (helmdemopull and helmdemopush) do not yet have passwords generated. Lets create passwords. Click on the token, click the icon under Actions column for either password1 or password2. You will also have an option to set an expiration date for the password if you planning to allow token to be valid only for specific days.

2. CI Stage

Get the latest chart from GitHub

To get the latest source from the GitHub repo, in our YAML pipeline we add a resources section and point to the repo on GitHub. We will also need to pass the endpoint parameter with the name of the GitHub service connection. More info about how to do this is here and here.

resources:
 repositories:
   - repository: helmrepo
     type: github
     name: utkarshpoc/azure-vote-helm-chart
     endpoint: github

Publish chart to ACR

Before we write steps to publish to ACR, we need to ensure we store the tokens we generated in the pipeline as variables. Here I am adding tokens as variables using the variables UI of the pipeline and marking passwords as secret.

You can also define these in Azure Pipelines Library if you intend to use these tokens in multiple pipelines.

I also have few static and non-secret variables in YAML itself so that they are source controlled using variables as below.

variables:
  acr.name: acrdemoutkarsh
  acr.repo.name: helmdemo/vote-app

The steps involved for publishing our Helm chart are simple.

1. Installing Helm 3 on the agent
2. Login to the ACR using Helm
3. Save and push the chart

1. Installing Helm 3 on the agent

You can do this using OOB HelmInstaller task. The YAML is as below. Here I am using latest for helmVersion, but you can stick specific to a version (3.x and above).

- task: HelmInstaller@0
  displayName: install helm
  inputs:
    helmVersion: 'latest'
    installKubectl: false

2. Login to the ACR using Helm

We can use simple script step to login to the registry.

- script: |
    helm registry login $(acr.name).azurecr.io --username $(acr.push.username) --password $(acr.push.password)
  displayName: login to acr using helm

But if you run this, at the time of writing you will get an error as below

Error: this feature has been marked as experimental and is not enabled by default. 
Please set HELM_EXPERIMENTAL_OCI=1 in your environment to use this feature

As the error states, this is because, publishing Helm charts which follow Open Container Initiative (OCI) standard is experimental and you are required to set HELM_EXPERIMENTAL_OCI to 1. We can do this by updating our variables section in the YAML. The updated yaml looks as below.

variables:
  acr.name: acrdemoutkarsh
  acr.repo.name: helmdemo/vote-app
  HELM_EXPERIMENTAL_OCI: 1

3. Save and push the chart to ACR

Next step is to save this chart locally and create an alias for the chart with our ACR registry URL. So our YAML looks as below.

- script: |
    helm chart save $(build.sourcesdirectory)/src/azure-vote-helm-chart/ $(acr.name).azurecr.io/$(acr.repo.name):latest
  displayName: save the chart and set the alias

And finally, we push the Helm chart to ACR using helm chart push command.

- script: |
    helm chart push $(acr.name).azurecr.io/$(acr.repo.name):latest
  displayName: push the chart to acr

So our CI stage is now complete.

2. CD Stage

Pull the latest version of chart from ACR

The first step is to pull the latest version of the chart (one with tag latest) to our agent machine and extract it in a folder. We can do that in pipeline as below. Notice I have added a new stage name cd and its dependent of stage ci.

- stage: cd
    displayName: CD
    dependsOn: ci
    jobs:
    - deployment: helm_publish_aks
      displayName: deploy to aks
      environment: 
        name: PROD
        resourceName: helmdemo
        resourceType: Kubernetes
      strategy:
        runOnce:
          deploy:
            steps:
            - task: HelmInstaller@0
              displayName: install helm
              inputs:
                helmVersion: 'latest'
                installKubectl: false

            - script: |
                echo "$(acr.pull.password)" | helm registry login $(acr.name).azurecr.io --username $(acr.pull.username) --password-stdin
              displayName: login to acr using helm

            - bash: |
                  helm chart pull $(acr.name).azurecr.io/$(acr.repo.name):latest
              displayName: get helm chart on agent

            - bash: |
                helm chart export $(acr.name).azurecr.io/$(acr.repo.name):latest --destination $(build.stagingdirectory)
              displayName: export the chart to folder

Because each job in Azure DevOps run in a separate agent, I have to ensure agent has Helm tool, so install Helm tool again in the first step. Also, note that I am using token with scope-map permission set only to pull charts.

Deploy the chart to AKS

Now that the latest chart is pulled from our ACR and is available on agent, only step remaining is to deploy to AKS. We can do that using HelmDeploy task.

- task: HelmDeploy@0
  displayName: deploy chart to aks
  inputs:
    connectionType: 'Azure Resource Manager'
    azureSubscription: '$(azure.service.connection)'
    azureResourceGroup: 'demos'
    kubernetesCluster: 'aksdemoutkarsh'
    namespace: 'helmdemo'
    command: 'upgrade'
    chartType: 'FilePath'
    chartPath: '$(build.stagingdirectory)/azure-vote/'
    releaseName: 'helmdemo'
    arguments: '--create-namespace --install'

View the deployed service in Azure DevOps

If you have added your Kubernetes cluster as a resource in Azure DevOps environment, you can now view deployed services directly from Azure DevOps.

You can also view deployed services in YAML and also see any ports exposed.

Browsing the exposed port, you will see application.

Conclusion

That is it! If you have read this far, thank you 🙏🏼. As we saw, with the support of tokens and OCI artifacts, ACR is one of the best in class container registry. Also, with deep integration with AKS (features like browsing services, logs, yaml), Azure DevOps brings in lots of productivity for your teams. Hope you found this post informative. If so, please share and tweet!

Comments

[Andres Gomez]

Hi, how are you? I have a question, when I make a second deployment AKS does not detect a new image and deletes the pod to be created with the new image automatically, I have to delete the pod manually. How did you solve this?

[Smyja]

Great article, would check it out. I just started playing around with helm.