Hey! You broke our pipeline!

One of the advantages of Azure Pipelines is that it automatically updates your tasks to the latest minor version. That way you don't have anything to stay up-to-date. But this also has disadvantages. Both Microsoft and Extension Authors can accidentally break your pipelines.

I've personally shipped a version of my Variable Toolbox extension that broke 10.000 developers at a single corporation that relied on my pipeline tasks.

And yesterday Microsoft had a world-wide outage due to the npmAuthenticate@0 task suffering from an unexpected breaking change.

Most people don't know what to do when something like this happens. But luckily there are a few things you can do to fix thing yourself when you find yourself in this situation.

Pin to an older version

If you're using YAML pipelines, then the simplest way to get unblocked, is to pin your workflow to a specific version of the task. This is one of the big advantages of YAML based pipelines. By default Azure Pipelines will reference a task by its major version @0 in the case of the npmAuthenticate task, but you can specify @0.208.1 or any other previous version instead:

# Broken
- step: npmAuthenticate@0
  inputs:
    workingDirectory: $(Build.SourcesDirectory)

# Fixed
- step: npmAuthenticate@0.208.1
  inputs:
    workingDirectory: $(Build.SourcesDirectory)

The easiest way to find the version to enter here is to look at the last pipeline that succeeded, the Initialize Job step will list the exact versions used:

2022-12-23T09:05:03.7682146Z ##[section]Starting: Initialize job
2022-12-23T09:05:03.7684106Z Current agent version: '2.213.2'
2022-12-23T09:05:03.7719717Z Current image version: '20221215.2'
2022-12-23T09:05:03.8121128Z Download all required tasks.
...
2022-12-23T09:05:03.8228407Z Downloading task: npmAuthenticate (0.208.1)
2022-12-23T09:05:06.6642168Z ##[section]Finishing: Initialize job

Overwrite the broken task

But if you're using classic build and release pipelines (UI based), then you can't pin to a minor version. In that case you'll need to upload a version of the task that works. You can use tfx-cli to do that:

TFS Cross Platform Command Line Interface v0.12.0
Copyright Microsoft Corporation

Syntax:
tfx build tasks upload --arg1 arg1val1 arg1val2[...] --arg2 arg2val1 arg2val2[...]

Command: upload
Upload a Build Task.

Arguments:
  --task-path Local path to a Build Task.
  --task-zip-path Local path to an already zipped task
  --overwrite Overwrite existing Build Task.

But first you'll need to get a copy of the task that works. Your Azure DevOps instance already has a copy of that task, you just need to know where to find it:

$pat = "{{AZURE DEVOPS PAT TOKEN}}"
$org = "{{AZURE DEVOPS ORG NAME}}"
$url = "https://dev.azure.com/$org"
$header = @{authorization = "Basic $([Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes(".:$pat")))"}

$taskName = "npmAuthenticate"
$taskid = "ad884ca2-732e-4b85-b2d3-ed71bcbd2788"
$taskversion = "0.208.1"
$taskZip = "$taskName.$taskid-$taskversion.zip"

Invoke-WebRequest -Uri "$url/_apis/distributedtask/tasks/$taskid/$taskversion" -OutFile "$taskZip" -Headers $header

Or you can grab a copy of an older version from my azure-pipelines-tasks-zips repository:

Now in order to "fix" the broken version we'll have to overwrite it. To do so we're going to patch the older working version with the broken version's version number.

First extract the zip file, then update the task.json file and set the version number to the one you want to overwrite (in our case 0.214.1):

Put the task.json back into the zip file.

We're now ready to upload the task. We can use tfx-cli to do that.

We first need to delete the existing task using tfx build task delete --task-id (warning this will delete all older versions):

And then we can upload the patched task tfx build tasks upload --overwrite --task-zips-path file-we-just-patched.zip:

Instead of deleting the existing task and re-uploading it, we can also patch the taskzip with a version number that's newer than the broken task, in our case 0.214.2 and then upload that.

As you can see here, that's what the Azure Pipelines team does too, but they roll out the task through their own provisioning process, they either hotfix the task, or simply roll back their changes:

Comparing master...releases/m215 · microsoft/azure-pipelines-tasks

Temporarily replace the broken task

Instead of overwriting, we can also install a temporary version of the task with a different id and name, reconfigure our pipelines to use the temporary replacement and switch back when the Azure Pipelines team has fixed theirs.

The process is very similar to replacing the existing task, but in this case we need to make a few more changes, we need to replace the id (choose any guid), name and update the friendlyName properties in both task.json and task.loc.json:

Or you can download a pre-patched (-sxs) version from the azure-pipelines-tasks-zips repository.

Then we can upload the patched file:

But now you'll need to go through all of your pipelines and replace the old task with the temporary replacement. A bit of scripting against the REST API would make that a quick change.