Use Container App Jobs for scaling Devops Build Agents

#azure #devops #containerapps #cicd

Setting up Build Agents in Projects, where from different Reasons (networking, capacity, cost, ...) hosted Agents are no option has been quite unpleasing for some time. But now, with the Container App Jobs, this can be done quite easily and with the scaling that we all wanted.

First, a short explanation of why Container Apps Jobs was needed to achieve this. The standard Container Apps supported scaling up with the KEDA Rule so it was not a problem to provision enough Revisions according to the Build Agent Queue. The Issue came up as soon as the first Builds finished. Scaling down Replicas was some kind of random, as soon as the Queue was empty and not all Replicas had a Job, any Replica could get removed, so often a running Container got shut down and therefore the scaling option was not usable. With that said, Container App Jobs solves this issue.

Preparing your Container

To Prepare your container, you can follow this guide . This article will only cover deploying the Build Agents from an Image that is already in a Container Registry.

Create a Build Agent Pool and a Personal Access Token

To create a Build Agent Pool navigate to Organization Settings (if you don't have the Permissions to do it on Organization Level just jump into the Settings of a Project) in DevOps, there in the Category Pipelines>Agent Pools click on "Add pool" and select "Self-hosted" as Pool Type. You can also just add the Agents to an existing Agent Pool of type "Self-hosted".

If you now open your Agent Pool (this time on Organization Level, it's important) you can see the Agent Pool Id in the URL:
https://dev.azure.com//_settings/agentpools?poolId=<poolId>&view=jobs
Note that Id somewhere.

Next step is to create a Personal Access Token, which the Containers can use to authenticate against the Devops API to register themselves as Build Agents. Therefore click on User Settings (top right in Devops) and jump to Personal Access Tokens. The Token should have permission to Read & Manage Agent Pools. Generate the PAT and store it somewhere save (e.g. Password Manager).

Setup a Container Apps Environment

If you already have one, you can use an existing one. A very handy fact about the Container Apps Jobs is that high-level Infrastructure Configuration mostly concerns the Container Apps Environment, therefore if you have an Environment that is already integrated into your networking infrastructure, just use this one to deploy your Container Apps Jobs, and they will automatically be integrated into the Network.

For this article, we're going to stick to a simple Environment with no specific configuration. Because the Container Apps Jobs are in Public Preview and only available with CLI or ARM I am going to stick with Azure CLI. Depending on if you run the CLI Commands in Powershell or Bash, you might need to replace the ""(powershell) with "\"(bash).

az containerapp env create \
    --name "<name>" \
    --location "westeurope" \
    --resource-group "<rg-name>" \
    --logs-workspace-id "<workspace-id>" \
    --logs-workspace-key "<workspace-key>"

You can leave off the workspace id and key, then it will provision you a Log Analytics workspace automatically. Just specify it if you already got one to avoid duplication. You can get the Key with:

az monitor log-analytics workspace get-shared-keys \
   --name "<law-name>" \
   --resource-group "<rg-name>"

Deploy the Container App Job

Now you can deploy the Container App Job like that:

az containerapp job create \
    --name "<name>" \
    --resource-group "<rg-name>" \
    --environment "<containerappenvironment-name>" \
    --trigger-type "Event" \
    --replica-timeout 7200 \  #timeout in seconds
    --replica-retry-limit 0 \
    --replica-completion-count 1 \
    --parallelism 1 \
    --polling-interval 1 \
    --image "<path to your image>" \
    --registry-server "<your container registry server>" \
    --registry-username "<cr-username>" \
    --registry-password "<cr-password>" \
    --cpu "0.25" --memory "0.5Gi" \
    --scale-rule-name "azure-pipelines" \
    --scale-rule-type "azure-pipelines" \
    --scale-rule-metadata "poolID=<poolId>" \
                          "organizationURLFromEnv=AZP_URL" \
                          "personalAccessTokenFromEnv=AZP_TOKEN" \
                          "poolName=AZP_POOL" \
    --secrets "azp-pool=<yourAgentPoolName>" \
              "azp-token=<yourPatToken>" \
              "azp-url=<yourDevopsOrgUrl>" \
    --env-vars "AZP_POOL=secretref:azp-pool" \
               "AZP_TOKEN=secretref:azp-token" \
               "AZP_URL=secretref:azp-url"

This is first a lot, but we'll go through the important things.

First of all, is the "replica-timeout". This needs to be balanced so you don't overspend on stuck build agents, but your pipeline runs also need to have enough time to finish. Just put a bit longer than your longest expected run in there. One tip, since we now have this awesome option: Create a Pool for short-running tasks and one for the long ones. Since they really scale to zero it might save some money.

The Polling interval might also be interesting, there you also have to find the balance between faster scaling -> shorter Queue Times, or lower cost -> fewer retries.

Then to configure the KEDA scaling you need to create Secrets for azp-pool, azp-token and azp-url. These you need to reference with the secretref: notation in your environment variables and pass those into the scale-rule-metadata. This kind of configuration does not look very pleasing but for a public preview, this will do.

Now all there is left to say is thank god for the option of using Build Agent Containers that don't require yourself to manage a K8s Cluster or draw carts of money to Microsoft because of missing scaling capabilities.

Edit:
You can now also find a template to deploy the Container Apps Jobs Build Agents using Powershell and Bicep. Check out this Link.