Develop, test, and deploy your extensions for all popular CIs from a single codebase

💡 The feature image shows a typical CI/CD pipeline in action partly drawn by OpenAI DALL-E, but in this article, we are going to develop something beneficial

Table of Contents

• Start from the official templates
  • GitHub Actions
  • Azure Pipelines
• Create the monorepo
• Share code between actions and tasks
• Build and publish
• CircleCI?

This is a relatively short tutorial on how to develop, test, and deploy your CI extensions for GitHub Actions, Azure Pipelines, and CircleCI from a single monorepo and is based on the experience of creating the Qodana CI extensions.

Start from the official templates

Let's pick the technology stack for our CI extensions.

OK, I will not pick. I'll just tell you why I used TypeScript and node.js for the extensions.

Pros for using JS-based actions:

• More flexible than bash/Dockerfile-based approaches
  • Different libraries (like actions/toolkit and microsoft/azure-pipelines-task-lib) with more accessible and easy-to-use APIs are available out-of-box
• Writing tests is relatively simple

Cons

• JavaScript

So let's write a TypeScript-based action!

GitHub Actions

I found the GitHub actions documentation easier to read than Azure, so I would recommend starting writing and testing your extensions on GitHub by using the official template actions/typescript-action. The mentioned template provides a good starting point; I won't repeat the steps here. Play with it, write some simple stuff, and then return here for the next steps.

Azure Pipelines

GitHub Actions are built on Azure infrastructure, so porting your GitHub action to Azure Pipelines should be relatively easy.

So,

• the "action" becomes the "task"
• it's packed a bit differently, distributed, and installed the other way

And the definition of a task task.json is the same as the action one action.yml.

For example, having the following action.yml:

name: 'Your name here'
description: 'Provide a description here'
author: 'Your name or organization here'
inputs:
  milliseconds: # change this
    required: true
    description: 'input description here'
    default: 'default value if applicable'
runs:
  using: 'node16'
  main: 'dist/index.js'

"Easily" translates to the following Azure task:

{
  "$schema": "https://raw.githubusercontent.com/Microsoft/azure-pipelines-task-lib/master/tasks.schema.json",
  "id": "822d6cb9-d4d1-431b-9513-e7db7d718a49",
  "name": "YourTaskNameHere",
  "friendlyName": "Your name here",
  "description": "Provide a description here",
  "helpMarkDown": "Provide a longer description here",
  "author": "Your name or organization here",
  "version": {
    "Major": 1,
    "Minor": 0,
    "Patch": 0
  },
  "instanceNameFormat": "YourTaskNameHere",
  "inputs": [
    {
      "name": "milliseconds",
      "type": "string",
      "label": "label name here",
      "defaultValue": "default value if applicable",
      "required": true,
      "helpMarkDown": "input description here"
    }
  ],
  "execution": {
    "Node10": {
      "target": "index.js"
    }
  }
}

From such a simple example, one can see why I suggested starting with GitHub Actions. But let's continue.

To start developing your new shiny Azure Pipelines task, I suggest just copying the action directory and then implementing steps from the official Azure documentation – it's pretty straightforward.

1. Create vss-extension.json
2. Create task.json and place it into your dist directory (actually better to name it after the task name)
3. If you used any methods from @actions/core or @actions/github in your action, you need to replace them with the corresponding methods from azure-pipelines-task-lib (e.g. core.getInput -> tl.getInput)

The API of azure-pipelines-task-lib is similar to @actions/core and other @actions/* libraries. For example, we have a method for obtaining the input parameters:

export function getInputs(): Inputs {
  return {
    milliseconds: core.getInput('milliseconds'),
  }
}

And the same for Azure Pipelines:

export function getInputs(): Inputs {
  return {
    milliseconds: tl.getInput('milliseconds'),
  }
}

For more real cases, feel free to explore our Qodana GitHub Actions codebase utils and Azure Pipelines task utils.

Create the monorepo

We are going to use npm workspaces to manage the monorepo.
Place your action and task code into subdirectories (e.g. github) of your newly created monorepo. And then create a package.json file in the root directory.

{
  "name": "@org/ci",
  "version": "1.0.0",
  "description": "Common code for CI extensions",
  "license": "Apache-2.0",
  "workspaces": [
    "github",
    "azure"
  ],
  "devDependencies": {
    "typescript": "latest",
    "eslint": "latest",
    "eslint-plugin-github": "latest",
    "eslint-plugin-jest": "latest",
    "prettier": "latest",
    "ts-node": "latest"
  }
}

So the monorepo structure looks like this:

...
├── action.yaml
├── github/
├── azure/
└── package.json

After implementing the workspace setup, you can run tasks and actions from the root directory. For example, to run the build task from the github directory, you can use the following command:

npm run -w github build 

Share code between actions and tasks

The most valuable part from using the monorepo approach starts here: you can share the code between your actions and tasks.

We are going to do the following steps:

1. Create a common directory in the root of the monorepo, a subproject for shared code
2. Update tsconfig.json compiler configurations from all sub-dirs for proper project builds

At first, let's create the base tsconfig – tsconfig.base.json with the base settings that are going to be used in all subprojects:

{
  "compilerOptions": {
    "target": "es6",
    "module": "commonjs",
    "strict": true,
    "esModuleInterop": true,
    "skipLibCheck": true,
    "forceConsistentCasingInFileNames": true,
    "resolveJsonModule": true,
    "composite": true
  },
  "exclude": ["node_modules", "**/*.test.ts", "*/lib/**"]
}

Then create a simple tsconfig.json in the project root:

{
  "references": [
    { "path": "common" },
    { "path": "azure" },
    { "path": "github" }
  ],
  "files": []
}

Then common/tsconfig.json:

{
  "extends": "../tsconfig.base.json",
  "compilerOptions": {
    "outDir": "./lib",
    "rootDir": "."
  },
  "files": ["include your files here or use typical include/exclude patterns"]
}

And finally, update the tsconfig.json files in the subprojects (they are basically the same, e.g. github/tsconfig.json):

{
  "extends": "../tsconfig.base.json",
  "compilerOptions": {
    "outDir": "./lib",
    "rootDir": "./src"
  },
  "references": [
    { "path": "../common" }
  ]
}

Now you can use the shared code from the common directory in your actions and tasks. For example, we have a qodana.ts file in the common directory that contains function getQodanaUrl that returns the URL to the Qodana CLI tool. And we use it in both actions and tasks.

Build and publish

You already have GitHub workflows from the template configured to publish your actions to your repository releases.
For automated releases, we use GH CLI, and we have a simple script that publishes a changelog to the repository releases:

#!/usr/bin/env bash
previous_tag=0
for current_tag in $(git tag --sort=-creatordate)
do

if [ "$previous_tag" != 0 ];then
    printf "## Changelog\n"
    git log ${current_tag}...${previous_tag} --pretty=format:'* %h %s' --reverse | grep -v Merge
    printf "\n"
    break
fi
previous_tag=${current_tag}
done

And the GitHub workflow that runs it:

name: 'Release'
on:
  push:
    tags:
      - '*'
permissions:
  contents: write

jobs:
  github:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
        with:
          fetch-depth: 0
      - run: |
          ./changelog.sh > changelog.md
          gh release create ${GITHUB_REF##*/} -F changelog.md
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

For Azure Pipelines task releases, you can use the official approach from Azure. Still, also you can do the same on GitHub actions infrastructure as their publisher tool can be installed anywhere. So, in our case, it's solved by a simple GitHub workflow job:

  azure:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Set Node.js 12.x
        uses: actions/setup-node@v3.6.0
        with:
          node-version: 12.x
      - name: Install dependencies
        run: npm ci && cd vsts/QodanaScan && npm ci && npm i -g tfx-cli
      - name: Package and publish
        run: |
          cd vsts && npm run azure
          mv JetBrains.qodana-*.vsix qodana.vsix
          tfx extension publish --publisher JetBrains --vsix qodana.vsix -t $AZURE_TOKEN
        env:
          AZURE_TOKEN: ${{ secrets.AZURE_TOKEN }}

With this set up, each release happens automatically on each tag push.

git tag -a v1.0.0 -m "v1.0.0" && git push origin v1.0.0

CircleCI?

Ah, yes, this article mentioned the CircleCI orb also... CircleCI setup is straightforward but does not support TypeScript extensions, so you have to pack your code into a Docker image or a binary and run it there. The only reason it's included in this post is that we build our orb with the monorepo approach, which works well.

Just implement the official orb template and place it in your monorepo, so the structure looks like this:

...
├── action.yaml
├── github/
├── azure/
├── src/            # orb source code here
└── package.json

And don't forget to commit .circleci/ directory to your repository to make CircleCI lint, test, and publish your orb.

If this post gets ten reactions here, I'll add a second part about publishing and testing Azure Pipelines and CircleCI orbs