Building VSTS YAML definitions with Sparrowdo

YAML build definitions

Today VSTS provides YAML DSL to create build definitions instead of creating them in Web UI.

I like the idea, though maintaining YAML code for more than one projects quickly becomes tedious. Microsoft offers some helpers on this road, like templates to reuse your build definitions, but let me show reasonable alternative.

Welcome to Sparrowdo VSTS-* family

Sparrowdo::VSTS is a subset of Sparrowdo modules to boostrap/generate YAML code for your VSTS build definitions. Sparrowdo itself is a flexible and powerful task-runner/configuration management tool built on shoulders of Perl5/Perl6.

Turing Sparrowdo into task runner

Initially Sparrowdo was written as configuration management tool to run remote scenarios by ssh, however Sparrowdo could be easily converted into local task runner, by setting a couple of options:

$ sparrowdo --local_mode --no_sudo

This makes Sparrowdo client run in local mode ( on local host ) and omit sudo when executing commands as this is not required to the tasks of code generation.

Let write code to create a code

Main idea of the approach is to create code generator for YAML build definitions, instead on creating YAML manually. Using high level Perl6 abstractions it is possible to generate YAML code definition (build.yaml) and then commit changes to a Git repository and finally run a build using generated build definition file.

Following a schematic picture of overall design:

1. Sparrowdo generator ( sparrowfile ):


+-----------------------------------------+
|                                         |
|  < build header >                       |
|      agent-info                         |
|      repo-info                          |
+-----------------------------------------+
|  < tasks >                              |
|                                         |
|      task1 <===> Sparrowdo Module Call  |
|      task2 <===> Sparrowdo Module Call  |
|      task3 <===> Sparrowdo Module Call  |
|      task4 <===> Sparrowdo Module Call  |
|      task5 <===> Sparrowdo Module Call  |
|      ... so on                          |
+-----------------------------------------+


2. Outcome => build.yaml file ( build definition )

Build header

Every YAML build defintion should contain essential info about where to run the build, we call it build header, he Sparrowdo representation of a such is Sparrowdo::VSTS::YAML::Build module:

$ nano sparrowfile


#!perl6

module_run "VSTS::YAML::Build", %(
  build-dir => "cicd/build",
  agent-name => "BuildServer01", # default value is not set
  queue => "Queue01",
  timeout => 10, # build timeout 10 minutes, this is default value
);

We define all required and optional parameters just like Perl6 function, other steps would be expressed the same way. Here we want to tun our build on agent named "BuildServer01", in queue named "Queue01", with execution maximum time of 10 minutes.

Build tasks

As I said every VSTS task is just a proper Sparrowdo::VSTS::* module call, consider simple scenario to build dotnet project and publish artifacts to VSTS storage, this implies following steps:

1. checkout source code ( ensured by VSTS itself )
2. Install nuget and nuget dependencies
3. Build solution
4. Publish artifacts

Following are implementation details:

Install nuget and nuget dependencies

$ nano sparrowfile

#!perl6

module_run "VSTS::YAML::Nuget", %(
  build-dir => "cicd/build",
  working-folder => ".", # path to project
  solution => "app.sln", # path to solution file, default value
);

Build solution

$ nano sparrowfile


#!perl6

module_run "VSTS::YAML::Solution", %(
  vs-version  => '15.0', # visual  studio version, default value
  build-dir => "cicd/build",
  solution => "app.sln", # path to solution file, default is "**\*.sln"
  platform => "x86",
  configuration => "debug",
  restore-solution => "app.sln", # path to NugetRestore solution file
  test-assemblies => True, # run tests, default value is False
  publish-symbols => False, # publish symbols, this is default value
);

Publish artifacts

$ nano sparrowfile

#!perl6

module_run "VSTS::YAML::Artifact", %(
  build-dir => "cicd/build",
  artifact-name => "drop",
  path => "'\$(build.artifactstagingdirectory)'",
  publish-location => "Container" # default value
);

Final version

Here is how we create YAML build definition from the scratch:

$ git clone $repo-url app && cd app # cloning source code of application to build

$ zef install Sparrowdo::VSTS::YAML::Build Sparrowdo::VSTS::YAML::Nuget Sparrowdo::VSTS::YAML::Solution Sparrowdo::VSTS::YAML::Artifact # Install proper Sparrowdo modules


$ nano sparrowfile # create build scenario mentioned in previous paragraphs


$ mkdir -p cicd/build # create directory to hold internal build files


$ sparrowdo --local_mode --no_sudo # generate build definition


$ echo ".cache" >> .gitingore # ask git to skip .cache files inside cicd/build dir


$ git add sparrowfile cicd/ .gitingore  && git commit -a -m "YAML build definition" # commit outcome

$ git push # push changes

The you may go to VSTS Web UI and create YAML based pipeline pointing cicd/build/build.yaml as build scenario, we are all set!