What's the difference between these options and how do they work?
Hosting options
When we want to build code, run code or even deploy some IaC in an automated way, we need somewhere for these commands to run. This is usually just some compute to run CLI commands, and the best choice is to use pipelines. With pipelines we can integrate source control, event triggers and security scanning.
With Azure DevOps we have a few options to host our pipelines, basically we have:
Microsoft hosted Agents
Self-Hosted Agents
These options represent the compute that runs our jobs and tasks.
An Azure DevOps pipeline is broken down into tasks, jobs and stages — each job requires an agent to run tasks, which can be inline scripts or prebuilt tasks we can consume like AzureCLI@2 which can run az cli commands.
Jobs are an extremely important concept for agent hosting. Microsoft provides a free job with up to 1,800 minutes a month, but the catch is only one job can be run simultaneously. If you want to run different jobs at the same time you need to purchase this — so you purchase concurrency. This is also the case for self-hosted agents, which we will go into more detail on.
This is similar to GitHub repositories which provide free GitHub runners, but only if your project is a public one where people can browse to the repo from the internet. Microsoft have recently removed the option to have public Azure DevOps projects, so the table above only really applies if you already have publicly visible ADO projects.
Microsoft hosted build agents
How do I set this up? — Create a pipeline and specify a well-known image in the pipeline, then run.
trigger: none
pool:
vmImage: ubuntu-latest
The first type of agent hosting is the Microsoft-hosted agent, where Microsoft manages the infrastructure, networking and agent provisioning, and you pass the agent your job. The Microsoft documentation states that each time you run a pipeline you get a fresh virtual machine for each job in a pipeline. This is probably achieved by pre-warmed virtual machine scale set instances which are waiting for job allocation after an instance runs the job it’s then destroyed and replaced by a new instance.
We can do some exploring of a Microsoft hosted image by creating a new pipeline in an azure devops project and running some commands to see what it tells us.
trigger: none
pool:
vmImage: ubuntu-latest
jobs:
- job: forensicInspection
displayName: "inspecting microsoft hosted agents"
steps:
- bash: |
echo "env with grep"
env | sort | grep -E 'SYSTEM_|BUILD_|AGENT_|GITHUB_' || true
displayName: "environment variables filtered with grep"
- bash: |
echo "env no grep"
env
displayName: "environment variables"
The output will give some additional insight into the agent
AGENT_ACCEPTTEEEULA=True
AGENT_BUILDDIRECTORY=/home/vsts/work/1
AGENT_ID=8
AGENT_ISSELFHOSTED=0
SYSTEM_ISAZUREVM=1
SYSTEM_ISDOCKERCONTAINER=0
SYSTEM_TASKINSTANCENAME=Bash1
VSTSAGENT_CONTINUE_AFTER_CANCEL_PROCESSTREEKILL_ATTEMPT=true
VSTSAGENT_DOCKER_ACTION_RETRIES=true
BUILD_ADVANCEDSECURITYENABLED=false
BUILD_QUEUEDBY=Jonny Diamond
BUILD_REPOSITORY_PROVIDER=TfsGit
BUILD_SOURCEBRANCH=refs/heads/main
BUILD_SOURCEBRANCHNAME=main
The different environment variables have prefixes which give insight into what they are for. For example, BUILD_xx supplies information like the repository, who queued the job and whether Advanced Security is enabled (that’s probably GitHub Advanced Security). SYSTEM_TASKINSTANCENAME tells the agent you are running bash tasks, so expect bash commands. We can also see AGENT_ISSELFHOSTED , SYSTEM_ISAZUREVM and SYSTEM_ISDOCKERCONTAINER — these tell us that we aren’t using a self-hosted agent, it’s an Azure virtual machine, and that it’s not a Docker container. These variables are provided to an agent at execution runtime; it’s a bit like saying “here is the repo, here is the code and this is what you’re doing.”
Managed DevOps Pools
How do I set this up? — Deploy network resources, Dev Center, Dev Center project, then the Microsoft.DevOpsInfrastructure/pools resource. Add the pool in ADO.
The other flavour of Microsoft-managed agents is Managed DevOps Pools. These are agents which are managed on behalf of the customer by Microsoft. The compute is actually hosted in a Microsoft subscription, with the option to VNet-inject into a subnet of your choice by delegating the subnet to the Microsoft.DevOpsInfrastructure/pools service.
When a pipeline is triggered and Azure DevOps requests an agent, a virtual machine agent is provisioned in a Microsoft-managed subscription. The VM’s NIC is injected into your delegated subnet as below.
We can see this in our lab by viewing the connected devices in our delegated subnet.
The subnet is delegated to the resource provider Microsoft.DevOpsInfrastructure/pools. This allows virtual machine agents in Microsoft’s managed subscription to have their NICs injected into this subnet, effectively surfacing the Managed DevOps Pool resource inside our virtual network.
In fact, if we click on the virtual machine in the portal under connected devices, we can see some more interesting information. We can see the Microsoft subscription it lives in and the tenant too. I am unable to list the resource as the issuer of the access token is not the correct tenant:
{
“code”: “InvalidAuthenticationTokenTenant”,
“message”: “The access token is from the wrong issuer ‘https://sts.windows.net/my_tenant_id/'. It must match the tenant ‘https://sts.windows.net/microsoft_tenant_id/' associated with this subscription. Please use the authority (URL) ‘https://login.windows.net/microsoft_tenant_id' to get the token. Note, if the subscription is transferred to another tenant there is no impact to the services, but information about new tenant could take time to propagate (up to an hour). If you just transferred your subscription and see this error message, please try back later.”
}
This demonstrates the VM exists in Microsoft’s subscription, but its network interface is injected into our subnet. All traffic flows through our network infrastructure, giving us full control over routing, NSGs and firewall rules. It also demonstrates that Managed DevOps Pools run on virtual machine scale sets.
Self-Hosted agents
How do I set this up? — In ADO > Project Settings > Add Agent Pool > Self-Hosted. Generate the zip and put it on your compute of choice.
The third option is self-hosted agents. In this scenario you host the agent yourself on your compute of choice: virtual machines, VMSS, Azure Container Apps or even AKS. The key difference between Microsoft-hosted or Managed DevOps Pools and self-hosted is that, as opposed to job sessions being provisioned on demand, self-hosted agents are instead long-running processes which poll Azure DevOps for jobs — they are always running and always listening for jobs.
Azure Virtual Machine Scale Sets
How do I set this up? — Deploy a VMSS, navigate to “Add Pool” in ADO, select the VMSS.
The final option is self-hosted on VMSS. This was Microsoft’s preferred choice for self-hosted agents, which allowed agents to be provisioned on an existing VMSS service to simplify deployment — no manual deploying of the agent. It’s also the inspiration behind Managed DevOps Pools. In this model, ADO manages the scaling when there are no agents waiting to pick up jobs or there are fewer agents than there should be. Like other self-hosted options, you can decide on the size and image you are running too, unlike the Microsoft-hosted agents which probably use Dasv5-size VMs.
Summary
So to host our ADO pipeline agents we have two main options: Microsoft-hosted and self-hosted. Managed DevOps Pools provide managed-on-behalf-of infrastructure injected into your network. Microsoft-hosted agents are fast to provision and useful in most cases, but they don’t have line of sight to private resources. Self-hosted agents maximise customisability but come with obvious management overhead, and you will still need to pay to run more than two jobs.
Top comments (0)