I regularly use Azure for my day-job or other software work, and depending on the task at hand, it's necessary to setup self-hosted agents to get your work done. For those of you uninitiated, Azure Pipelines are a way of automating CI/CD and, once you've gotten over the initial learning curve, are genuinely great. At the time of writing this, open-source software projects could have up to 10 simultaneous "agents", hosted for free on Azure's cloud. However, for private projects you're a bit more limited. You get a single agent for a few thousand hours and this is generally insufficient for all but most basic projects. This is where self-hosted agents come in. You setup your own environment and plug it into Azure's job scheduler to get sorta the best of both worlds.
Being Saturday and COVID, I decided to work on making a private pipeline more efficient by adding a self-hosted agent to the existing pipeline. Since I'm cheap, I'm only willing on shelling out the $60 it takes to have two simultaneous, unlimited MS agents and waiting for the CI tasks to complete was really getting on my nerves. Since I have a home server/NAS that's mostly sitting there, I might as well make use of its processor. Should be easy right?
Setting Up the Agent
Let me prerequisite this by first saying that I'm no fan of Microsoft Windows. The cumulative time developers have spent in the past decade dealing with the OS idiosyncrasies, inefficiencies, and bad business practices enabled by a past corporate culture is a crime against humanity. That being said, the MS of today has genuinely pivoted to the cloud and open-source and rather than killing GitHub after it's acquisition, I would argue that it's improved under their leadership. Not something I'd expect from Microsoft. While I genuinely think that Windows, or at least the kernel, will be replaced with Linux one day as Windows slides into obsolescence, but think that Microsoft has a bright future ahead in the areas of cloud and software development. And honestly, there's far more money there than just in the OS anyway.
Windows has great documentation for setting up the agent and installing it locally at Self-hosted Linux agents, and there's very little I can add to the setup process. Note that for the agent to be started automatically to enable the service with ./svc.sh install.
Pipeline Configuration
With the agent installed, the next step is to modify your azure-pipelines.yml. For example, I added my agent, ds9, to the pool ds9_pool and added that into the yml with:
- job: Linux
pool: ds9_pool
From the old configuration of:
pool:
vmImage: 'ubuntu-latest'
Next, create a branch with these changes and submit push it. Provide you already have the pipeline setup, this "Linux" job will automatically run on your new agent. I say "run" but I really mean, "fail almost instantly" because if you're migrating from a Microsoft-hosted agent, you'll need to do some extra work getting it compatible.
Pipeline Migration to Self-Hosted Agent
First, a little background on my pipeline. My pipeline compiles a python module that has several C/C++ extensions that are built using Cython. Overall, it's a great setup, I have the choice to write in Cython or to write pure C/C++ and then interface with it from Cython. For my project this saves a lot of time writing C extension interface code in pure C, which is often brittle and difficult to code.
However, using C extensions means that I can't just compile the package as a pure-python and ship it. I've got to build it on the environment and platform matching where it will be deployed. In this case, that's Windows and Linux for Python 3.6 - Python 3.8. In the past I would have to manually build it for each release, and all the build scripts and unit testing would take a minimum of a week per release (regardless of patch or major). Creating a pipeline is absolutely critical to getting the real work done of making better software.
The beginning of my "Linux" job looks like:
jobs:
- job: Linux
strategy:
matrix:
Python36:
python.version: '3.6'
python.architecture: 'x64'
Python37:
python.version: '3.7'
python.architecture: 'x64'
Python38:
python.version: '3.8'
python.architecture: 'x64'
pool: ds9_pool
steps:
- template: .ci/checkout.yml
- task: UsePythonVersion@0
inputs:
versionSpec: $(python.version)
displayName: 'Use Python $(python.version)'
This is where the first issue migrating will crop up. The UsePythonVersion@0 task requires Python to be installed on the self-hosted machine. For example, assuming you're using the default _work agent directory, your directory structure should look like:
$AGENT_TOOLSDIRECTORY/
Python/
{version number}/
{platform}/
{tool files}
{platform}.complete
Microsoft provides documentation for setting this up at Use Python Version task, but it's not complete. For example, what's in {tool files}? Can I use 3.8 in place of 3.8.5 in {version number}. Is there an automated way of installing Python?
Turns out it takes a bit of work to figure out, but it's actually quite straightforward once you know a few tricks:
Installing and Setting up Python
The directory structure of Python 3.8 should look like:
$AGENT_TOOLSDIRECTORY/
Python/
3.8/
x64/
bin/
include/
lib/
share/
x64.complete
You could manually download, install, and copy over Python to this directory, but it's far better to simply use a Python virtual environment as this is repeatable and far more reliable. First, ensure you have python3.8-venv installed with:
sudo apt install python3.8-venv
Next, cd into your _tool directory and create a Python directory with mkdir Python. I need three versions of Python, 3.6, 3.7, and 3.8. Since the machine already has 3.8 installed from Ubuntu 20.04, it's easy to create a virtual environment since the base version is already installed. However, first you'll need to create a 3.8 directory as this is how Azure expects the directories to be setup. However, since you might wish to reference the exact version of python, it's best to create a directory of the full version of Python and then symlink it to the directory without the patch version. For example, the version installed after running python3 -V is Python 3.8.5, so you would create a symlink with ln -s 3.8.5 3.8. Your directory structure should now look like:
$AGENT_TOOLSDIRECTORY/
Python/
3.8/ -> 3.8.5/
3.8.5/
cd into 3.8.5 and create your virtual enviornment with:
python3.8 -m venv x64
It's x64 because that's how Azure expects it to be setup (i.e. {version}/{architecture}). Finally, touch x64.complete to add the x64.complete file within the 3.8.5 directory. I'm not sure why this is necessary given that all the information is there in the directory structure, but alas, some poor programmer probably had a reason to do this.
The directory structure should now look like:
$AGENT_TOOLSDIRECTORY/
Python/
3.8 --> 3.8.5
3.8.5/
x64/
bin/
include/
lib/
share/
x64.complete
This has the advantage of using the system Python as a symlink, not as a copy, yet retaining separate directories for all Azure installed packages. For all intensive purposes, it's a virtual environment setup just for the Azure agent.
The Other Python Versions
Ubuntu 20.04 comes with Python3.8, but for the older versions of Python, you'll need to add the deadsnakes repository and install them from there. Do this with:
sudo add-apt-repository ppa:deadsnakes/ppa
Since my pipeline requires the Python headers, I need to install the -dev version, and in order to create the virtual environment, you'll need to install -venv, so for the other two versions of Python, the full installation command is:
sudo apt install python3.7-dev python3.7-venv python3.6-dev python3.6-venv
Next, starting in the Python directory, create the virtual environments for 3.6 and 3.7 with:
PY36_VER=$(python3.6 -c "import sys; print('.'.join([f'{val}' for val in sys.version_info[:3]]))")
mkdir $PY36_VER
ln -s $PY36_VER 3.6
cd $PY36_VER
python3.6 -m venv x64
touch x64.complete
cd ..
PY37_VER=$(python3.7 -c "import sys; print('.'.join([f'{val}' for val in sys.version_info[:3]]))")
mkdir $PY37_VER
ln -s $PY37_VER 3.7
cd $PY37_VER
python3.7 -m venv x64
touch x64.complete
cd ..
The directory structure should look like:
$AGENT_TOOLSDIRECTORY/
Python/
3.6 --> 3.6.13
3.6.13/
x64/
bin/
...
x64.complete
3.7 --> 3.7.10
3.7.10/
x64/
bin/
...
x64.complete
3.8 --> 3.8.5
3.8.5/
x64/
bin/
...
x64.complete
Note that if your pipeline does not include a step to update pip, you may have to add it in. The version included with Python 3.6 was incompatible with my workflow and had to be manually upgraded.
Installing other Packages and Requirements
Azure hosted images contain several popular software packages, and if you're migrating to a fresh environment, it's necessary to install quite a few of them manually. In my case, I needed the following:
sudo apt install docker zip libglu1-mesa
sudo snap install docker
Zip was for the ArchiveFiles@2 step, libglu1-mesa was for vtk, and docker was for the building the Python wheels on quay.io/pypa/manylinux2010_x86_64.
Performance Improvements and Next Steps
Using Microsoft hosted agents is a great way to run CI/CD on the cloud without owning hardware; it scales well and is well suited to building software. However, in my case, I had extra hardware and spare time, so it was worth the time investment to add an additional agent to the pipeline. My builds run about 3x faster using the additional agent, and I could probably find another (perhaps Windows) computer to run the build.
However, I wasn't just going after performance improvements. There's some data that I'd rather not to be on the "cloud", and it's important to run these functional tests pre-release. I've been neglecting them because they're not automated, and there's usually an edge case that crops up because I failed to run them. The next step is to add in these functional tests, save the benchmarks (using pytest-benchmark), and add it in as a pre-release step.
Top comments (2)
Thank you for this fun and helpful write-up.
Hi Alex, Thanks for putting this out here, it provided some good ideas for me. Everything works as expected, except I cannot get the agent's system capabilities to update and list the additional versions of python. I can create a user defined capability in the devops portal, but I would like for it to be automatically detected. Did you attempt to do this? Thanks!