DEV Community

Cover image for Docker & Azure Pipelines: Better Together!
Davide 'CoderDave' Benvegnù
Davide 'CoderDave' Benvegnù

Posted on • Edited on

Docker & Azure Pipelines: Better Together!

#cicd #azuredevops #devops #docker

If you want to use an Azure Pipelines Self Hosted agents, you have some different options. But only few of them allow you to be truly flexible and scalable.

In this post we will see how to run the Azure Pipelines Agents in Docker, so you can take advantage of orchestrators like Kubernetes, Azure Container Instances, and many more.

Let's get into it.

Intro

Today I wanna show you how you can run your Azure Pipelines agents in Docker so you can then take advantage of orchestrators like Kubernetes, Azure Container Instances, and others to make your CI CD agents truly scalable and flexible.

I will walk you through a complete container example.

Video

As usual, if you are a visual learner, or simply prefer to watch and listen instead of reading, here you have the video with the whole explanation and demo, which to be fair is much more complete than this post.

(Link to the video: https://youtu.be/rO-VKProMp8)

If you rather prefer reading, well... let's just continue :)

Supported Platforms

First things first: Let's talk about the supported platforms.

Platforms

The good news is that both Windows and Linux are supported as container hosts. If your CICD process requires tools that are platform specific there is no problem.

Prerequisites

Second thing, prerequisites. On Windows you would need to remember to enable Hyper-V, which is disabled by default, while on Linux it just works.

Of course you would need to have Docker installed on both platforms, and that could be either Docker Community Edition or Docker Enterprise Edition.

Container Image Creation

Alright, let see now how to create a basic image. The process is slightly different between Windows and Linux, so I will cover both.

I will start with Windows, but you can skip to the Linux section if you want.

Windows

Click here to check the video and see this explained in much greater details.

To create a container image for Windows you need a Docker file, of course.

FROM mcr.microsoft.com/windows/servercore:ltsc2019

WORKDIR /azp

COPY start.ps1 .

CMD powershell .\start.ps1
Enter fullscreen mode Exit fullscreen mode

We derive from the Server Core image, which contains already all the dependencies we need.

Next we need to download the PowerShell script Microsoft provides (here):

if (-not (Test-Path Env:AZP_URL)) {
  Write-Error "error: missing AZP_URL environment variable"
  exit 1
}

if (-not (Test-Path Env:AZP_TOKEN_FILE)) {
  if (-not (Test-Path Env:AZP_TOKEN)) {
    Write-Error "error: missing AZP_TOKEN environment variable"
    exit 1
  }

  $Env:AZP_TOKEN_FILE = "\azp\.token"
  $Env:AZP_TOKEN | Out-File -FilePath $Env:AZP_TOKEN_FILE
}

Remove-Item Env:AZP_TOKEN

if ($Env:AZP_WORK -and -not (Test-Path Env:AZP_WORK)) {
  New-Item $Env:AZP_WORK -ItemType directory | Out-Null
}

New-Item "\azp\agent" -ItemType directory | Out-Null

# Let the agent ignore the token env variables
$Env:VSO_AGENT_IGNORE = "AZP_TOKEN,AZP_TOKEN_FILE"

Set-Location agent

Write-Host "1. Determining matching Azure Pipelines agent..." -ForegroundColor Cyan

$base64AuthInfo = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(":$(Get-Content ${Env:AZP_TOKEN_FILE})"))
$package = Invoke-RestMethod -Headers @{Authorization=("Basic $base64AuthInfo")} "$(${Env:AZP_URL})/_apis/distributedtask/packages/agent?platform=win-x64&`$top=1"
$packageUrl = $package[0].Value.downloadUrl

Write-Host $packageUrl

Write-Host "2. Downloading and installing Azure Pipelines agent..." -ForegroundColor Cyan

$wc = New-Object System.Net.WebClient
$wc.DownloadFile($packageUrl, "$(Get-Location)\agent.zip")

Expand-Archive -Path "agent.zip" -DestinationPath "\azp\agent"

try
{
  Write-Host "3. Configuring Azure Pipelines agent..." -ForegroundColor Cyan

  .\config.cmd --unattended `
    --agent "$(if (Test-Path Env:AZP_AGENT_NAME) { ${Env:AZP_AGENT_NAME} } else { ${Env:computername} })" `
    --url "$(${Env:AZP_URL})" `
    --auth PAT `
    --token "$(Get-Content ${Env:AZP_TOKEN_FILE})" `
    --pool "$(if (Test-Path Env:AZP_POOL) { ${Env:AZP_POOL} } else { 'Default' })" `
    --work "$(if (Test-Path Env:AZP_WORK) { ${Env:AZP_WORK} } else { '_work' })" `
    --replace

  Write-Host "4. Running Azure Pipelines agent..." -ForegroundColor Cyan

  .\run.cmd
}
finally
{
  Write-Host "Cleanup. Removing Azure Pipelines agent..." -ForegroundColor Cyan

  .\config.cmd remove --unattended `
    --auth PAT `
    --token "$(Get-Content ${Env:AZP_TOKEN_FILE})"
}
Enter fullscreen mode Exit fullscreen mode

This script checks the environment, create some folders and install and configure the agent.

If you want a fresh agent container for every pipeline job, pass the --once flag to the run command. You must also use a container orchestration system, like Kubernetes or Azure Container Instances, to make sure you start new copies of the container when the work completes.

Finally, let's just create the image:

docker build -t dockeragentwin:latest -f Dockerfile.windows .
Enter fullscreen mode Exit fullscreen mode

Once again, you may want to check the video and see this explained in much greater details.

Linux

Click here to check the video and see this explained in much greater details.

For Linux, the Dockerfile looks a little different:

FROM ubuntu:18.04

# To make it easier for build and release pipelines to run apt-get,
# configure apt to not require confirmation (assume the -y argument by default)
ENV DEBIAN_FRONTEND=noninteractive
RUN echo "APT::Get::Assume-Yes \"true\";" > /etc/apt/apt.conf.d/90assumeyes

RUN apt-get update \
&& apt-get install -y --no-install-recommends \
        ca-certificates \
        curl \
        jq \
        git \
        iputils-ping \
        libcurl4 \
        libicu60 \
        libunwind8 \
        netcat \
        libssl1.0

WORKDIR /azp

COPY ./start.sh .
RUN chmod +x start.sh

CMD ["./start.sh"]
Enter fullscreen mode Exit fullscreen mode

The main differences here are that we use a Linux base image, and that we need to install the dependencies manually using apt-get.

Then we need the startup script as well, but this time is a bash script (also provided by Microsoft here)

#!/bin/bash
set -e

if [ -z "$AZP_URL" ]; then
  echo 1>&2 "error: missing AZP_URL environment variable"
  exit 1
fi

if [ -z "$AZP_TOKEN_FILE" ]; then
  if [ -z "$AZP_TOKEN" ]; then
    echo 1>&2 "error: missing AZP_TOKEN environment variable"
    exit 1
  fi

  AZP_TOKEN_FILE=/azp/.token
  echo -n $AZP_TOKEN > "$AZP_TOKEN_FILE"
fi

unset AZP_TOKEN

if [ -n "$AZP_WORK" ]; then
  mkdir -p "$AZP_WORK"
fi

rm -rf /azp/agent
mkdir /azp/agent
cd /azp/agent

export AGENT_ALLOW_RUNASROOT="1"

cleanup() {
  if [ -e config.sh ]; then
    print_header "Cleanup. Removing Azure Pipelines agent..."

    ./config.sh remove --unattended \
      --auth PAT \
      --token $(cat "$AZP_TOKEN_FILE")
  fi
}

print_header() {
  lightcyan='\033[1;36m'
  nocolor='\033[0m'
  echo -e "${lightcyan}$1${nocolor}"
}

# Let the agent ignore the token env variables
export VSO_AGENT_IGNORE=AZP_TOKEN,AZP_TOKEN_FILE

print_header "1. Determining matching Azure Pipelines agent..."

AZP_AGENT_RESPONSE=$(curl -LsS \
  -u user:$(cat "$AZP_TOKEN_FILE") \
  -H 'Accept:application/json;api-version=3.0-preview' \
  "$AZP_URL/_apis/distributedtask/packages/agent?platform=linux-x64")

if echo "$AZP_AGENT_RESPONSE" | jq . >/dev/null 2>&1; then
  AZP_AGENTPACKAGE_URL=$(echo "$AZP_AGENT_RESPONSE" \
    | jq -r '.value | map([.version.major,.version.minor,.version.patch,.downloadUrl]) | sort | .[length-1] | .[3]')
fi

if [ -z "$AZP_AGENTPACKAGE_URL" -o "$AZP_AGENTPACKAGE_URL" == "null" ]; then
  echo 1>&2 "error: could not determine a matching Azure Pipelines agent - check that account '$AZP_URL' is correct and the token is valid for that account"
  exit 1
fi

print_header "2. Downloading and installing Azure Pipelines agent..."

curl -LsS $AZP_AGENTPACKAGE_URL | tar -xz & wait $!

source ./env.sh

print_header "3. Configuring Azure Pipelines agent..."

./config.sh --unattended \
  --agent "${AZP_AGENT_NAME:-$(hostname)}" \
  --url "$AZP_URL" \
  --auth PAT \
  --token $(cat "$AZP_TOKEN_FILE") \
  --pool "${AZP_POOL:-Default}" \
  --work "${AZP_WORK:-_work}" \
  --replace \
  --acceptTeeEula & wait $!

print_header "4. Running Azure Pipelines agent..."

trap 'cleanup; exit 130' INT
trap 'cleanup; exit 143' TERM

# To be aware of TERM and INT signals call run.sh
# Running it with the --once flag at the end will shut down the agent after the build is executed
./run.sh & wait $!
Enter fullscreen mode Exit fullscreen mode

Although it looks different, this script does exactly the same things of the Windows one: prepare the system, and install and configure the agent.

Let's build the image:

docker build -t dockeragentlin:latest -f Dockerfile.linux .
Enter fullscreen mode Exit fullscreen mode

Once again, if you want a fresh agent container for every pipeline job, pass the --once flag to the run command.

Remember to check the video and see this explained in much greater details.

Run the container

That was not too bad right? And the fact that, as we have seen, Microsoft provides the full script for both platforms makes it even easier.

If you are interested in how you can Drastically improve you Docker Build Performance, I have a video which talks just about that.

To run the container, of course we can use the docker run command:

docker run -e AZP_URL=<Azure DevOps instance> -e AZP_TOKEN=<PAT token> -e AZP_AGENT_NAME=<Agent name> dockeragentlin:latest
Enter fullscreen mode Exit fullscreen mode

As you've probably noticed, there are some environment variables we can use to control the execution. Needless to say, those works in both a local Docker instance as well as any orchestrator.

Environment Variables

We can use the _AZP_URL_ variable to specify the URL of the instance of Azure DevOps we want to connect to. This parameter, as you may imagine, is required.

We also need to pass the _AZP_TOKEN_ variable and its value must be a Personal Access Token with the Read and Manage Agent Pools scope that is valid on the instance of Azure DevOps we are connecting to.

The other 3 variables are instead optional.

  • _AZP_AGENT_NAME_ lets you set a name for the agent and it defaults to the container hostname
  • _AZP_POOL_ is used to specify the Agent Pool that you want that specific agent to be part of
  • And finally _AZP_WORK_ lets you cange the working directory of the agent, which by default is _work.

Extend the basic image

At this point we have a fully working basic build agent.

You can of course extend the Dockerfile to include additional tools and their dependencies, or build your own container by using this one as a base layer.

Just make sure that the start script is called and it is the last command in the Dockerfile, and ensure that any of the derivative containers don't remove any of the dependencies of the original one.

The agent in Azure DevOps

Check out this section of the video to see how a new agent appears in Azure DevOps.

Important Notes

Before we close, just a few notes.

First of all, as we have seen before, the startup script installs the agents every time the image is run from scratch. If you want the startup to be quicker, you can consider installing the agent as part of your docker image creation instead.

Also, remember that each agent automatically updates itself when it runs a task that requires a newer version of the agent.

Second thing to note is that in order to use Docker from within a Docker container, you have to bind-mount the Docker socket.

Warning

As this very noticeable warning message on the official documentation site says, remember that doing so it is not recommended because it has serious security implications. The code inside the container can inf cat run as root on your Docker host.

Lastly, if you are trying to run this on Windows and you experience an error like the one below, you need to install Git Bash by downloading and installing git-scm.

Error

Conclusions

Alright, that's it for today. Now you can take those container images and run them in any orchestration of your choice, to have a truly flexible and scalable CICD platform with Azure Pipelines.

Let me know in the comment section below what you think about this, and if you'd like me to make a follow-up post/video on this showing how to deploy the containerized Azure Pipeline agents to Kubernetes.

Top comments (10)

Collapse
 
viniciodavid profile image
ViNiCiO PaZmiñO •

Hi, I want to make a pipeline in azure devops to automate the deployment of those commands, apart I use an ACR (Azure Container Registry of Azure) image and I am using my own agent within the agent pool to launch those commands, I would have a skin like this:

pool:
name: MYPOOLAGENT
demands:

agent.name -equals $(AGENT_TO_DEPLOY) # My Agent WINDOWS
steps:

task: DockerInstaller@0
displayName: Docker Installer
inputs:
dockerVersion: 17.09.0-ce
releaseType: stable

task: Docker@2
displayName: Login to ACR
inputs:
command: login
containerRegistry: 'DockerRegistrySREDevSecOps'

script: |
docker run -d -e AZP_URL="$(azure-url)" \
-e AZP_TOKEN="$(PERSONAL_ACCESS_TOKEN)" \
-e AZP_AGENT_NAME="$(AGENT_NAME)" \
-e AZP_POOL="MYPOOLAGENT" $(acr-image)
displayName: Deploy Agent in Azure

I get the following error:
error during connect: Get http://%2F%2F.%2Fpipe%2Fdocker_engine/v1.32/version: open //./pipe/docker_engine: The system cannot find the file specified. In the default daemon configuration on Windows, the docker client must be run elevated to connect. This error may also indicate that the docker daemon is not running.

the agent in which I compile is Windows so I have added install docker tasks, but I think the docker does not initialize, how can I add some task in my pieline to correct the error?

Collapse
 
n3wt0n profile image
Davide 'CoderDave' Benvegnù •

When you use your own agens (aka Self-hosted Agents) you need to install docker on your agent manually before executing any pipeline to make sure they run with the proper permissions.

Collapse
 
ashun_77 profile image
ASHUN KOTHARI • • Edited

How can I specify a custom path for docker.sock while mentioning the resources or while mentioning container inside the task?

I am running the agent as a service on a linux machine which has docker installed but the socket is not at its default location. I want to run jobs inside a docker but the build fails as I am not able to specify the docker.sock path.

Collapse
 
n3wt0n profile image
Davide 'CoderDave' Benvegnù •

In order to use Docker from within a Docker container, you need to [bind-mount}(docs.docker.com/storage/bind-mounts/) the Docker socket.

you will need to add 2 parameters to your docker run command line which are -v /var/run/docker.sock:/var/run/docker.sock -v /usr/bin/docker:/usr/bin/docker (Change it with your path)

Collapse
 
ashun_77 profile image
ASHUN KOTHARI •

Thank you for the reply Davide! :)

The way I have configured my pipeline is somewhat like this

resources:
  containers:
  - container: u14
    image: ubuntu:14.04

  - container: u16
    image: ubuntu:16.04

  - container: u18
    image: ubuntu:18.04

jobs:
- job: RunInContainer
  pool:
    vmImage: 'ubuntu-18.04'
  container: u18

  steps:
  - script: printenv
Enter fullscreen mode Exit fullscreen mode

My agent is a running as a service. The docker installed on the agent is having its docker.sock file at a different place. As I am not running the docker run command manually is there a way I could specify it in the above configurations?

I tried giving it under resources > containers > container > options but that didn't work.

Thread Thread
 
n3wt0n profile image
Davide 'CoderDave' Benvegnù •

Well, that should be in the docker run of your agent container (in the script you use to start the container that has the agent in it)

Collapse
 
viniciodavid profile image
Info Comment hidden by post author - thread only accessible via permalink
ViNiCiO PaZmiñO • • Edited

Hi, I want to make a pipeline in azure devops to automate the deployment of those commands, apart I use an ACR (Azure Container Registry of Azure) image and I am using my own agent within the agent pool to launch those commands, I would have a skin like this:

pool:
name: MYPOOLAGENT
demands:

  • agent.name -equals $(AGENT_TO_DEPLOY) # My Agent WINDOWS

steps:

  • task: DockerInstaller@0
    displayName: Docker Installer
    inputs:
    dockerVersion: 17.09.0-ce
    releaseType: stable

  • task: Docker@2
    displayName: Login to ACR
    inputs:
    command: login
    containerRegistry: 'DockerRegistrySREDevSecOps'

  • script: |
    docker run -d -e AZP_URL="$(azure-url)" \
    -e AZP_TOKEN="$(PERSONAL_ACCESS_TOKEN)" \
    -e AZP_AGENT_NAME="$(AGENT_NAME)" \
    -e AZP_POOL="MYPOOLAGENT" $(acr-image)
    displayName: Deploy Agent in Azure

I get the following error:
error during connect: Get http://%2F%2F.%2Fpipe%2Fdocker_engine/v1.32/version: open //./pipe/docker_engine: The system cannot find the file specified. In the default daemon configuration on Windows, the docker client must be run elevated to connect. This error may also indicate that the docker daemon is not running.

the agent in which I compile is Windows so I have added install docker tasks, but I think the docker does not initialize, how can I add some task in my pieline to correct the error?

Collapse
 
dsfrederic profile image
Frédéric De Smet •

How can i provide tools to build an SQL Database project for example?

Collapse
 
n3wt0n profile image
Davide 'CoderDave' Benvegnù •

Not sure I got your question. But if you are referring to the content of the article, you can add all the tools you need to the base image, together with the agent.

This way every time your pipeline runs, you have all the tools you need already there to use

Collapse
 
n3wt0n profile image
Davide 'CoderDave' Benvegnù •

Made some small changes. Let me know what you think!

Do you use already the Azure Pipelines Agents inside Docker Containers?

Some comments have been hidden by the post's author - find out more

Read next

hanzalawebdev profile image

BashBlaze Day 4: Building a System Monitoring Script with Bash

Muhammad Hanzala Ali -

techmaharaj profile image

How To Run Static Analysis On Your CI/CD Pipelines Using AI

Atulpriya Sharma -

monarene profile image

Designing a fault-tolerant etcd cluster

Michael Mekuleyi -

574n13y profile image

Networking Concepts for DevOps and Cloud Engineers

Vivesh -