AUTOMATING NPM PACKAGE PRERELEASES

Akalanka Perera — Sun, 07 May 2023 21:09:05 +0000

What I built

We at the SLIIT FOSS Community are a passionate and committed team dedicated to promoting the use and development of open source software. It has been quite some time since we have shifted focus on building NPM and Dart libraries for use by fellow developers. The subject of this post however will be the former where we'll be exploring the use of a plethora of tools and concepts to build a fully automated prerelease mechanism for these libraries.

Category Submission:

DIY Deployments

App Link

https://www.npmjs.com/~sliit.foss

Screenshots

Description

NPM Catalogue isn't exactly an app on it's own. It's just a development environment for our NPM libraries. The main idea behind this project was to have everything required for the development, testing and release of an NPM library from step 1 in one single place making it as easy as possible for developers to contribute to these libraries allowing them to focus on the actual development itself rather than the surrounding processes. Simply, it's one of the many ways we at the SLIIT FOSS Community are trying to make open source development easier for everyone and we hope that this project will be of use to many developers out there.

Link to Source Code

Repository - https://github.com/sliit-foss/npm-catalogue

Full list of changes which were made under this development cycle - https://github.com/Akalanka47000/npm-catalogue/pull/3/files

Note that this is just a fork and the actual workflow runs can be found in the parent repository

Permissive License

MIT

Background

As of today, we have over a dozen packages which have been released publicly to the NPM registry which you can find over here and till now we only released these as either major, minor or patch updates. However as of late, with more complex developments, we had identified the need for prereleases which would primarily help us with a smoother and safer way of getting things done. There were instances that a library was completed to a certain extent, was ready for internal testing but not just yet ready for a full release.

At the time, we had already automated the existing release process with the use of a GitHub Actions Workflow that would be run on a push to the main branch of the repository. This workflow took care preparing the project, executing the required scripts for versioning, releasing the packages to the NPM registry and finally syncing the newly published package versions back into the repository by execting a commit within the workflow itself.

All of our packages are housed within this single repository and it is bootstrapped with Turborepo which was already making things quite easy for us. For example, we could run a single command to execute a script across all packages within the repository and in our case, we had a release script defined within all of our packages and a simple process of running pnpm release would take care of the rest. The release script was organized as follows:-

Root package.json

{
    "scripts": {
        ...
        "release": "turbo run release --concurrency=1",
        ...
    }
}

Workspace package.json

{
    "scripts": {
        ...
        "build": "node ../../scripts/esbuild.config.js",
        "bump-version": "pnpm build && npx automatic-versioning --name=@sliit-foss/<package-name> --no-commit --recursive",
        "release": "bash ../../scripts/release.sh",
        ...
    }
}

release.sh

pnpm run bump-version && pnpm publish --access=public --tag=latest || true

Our versioning logic was quite simple, we decided which type of release it was based on the prefix of the commit which triggered the workflow run, for example:-

Fix: <commit message> would trigger a patch release
Feat: <commit message> would trigger a minor release
Feat!: <commit message> would trigger a major release

This process was quite easy since we anyways follow conventional commits while checking in changes. We have Husky setup with Commitlint which ensures that this safe-zone is never crossed. The version bumping itself is handled by a library of our own making called @sliit-foss/automatic-versioning which evaluates the commit history and
determines the next version based on the type of release.

While this process was working fine, it did have these following problems:-

What if there was a prerelease that needed to be done? This would mean that we would have to manually bump the version of the package, publish it to the NPM registry and then sync the version back into the repository. This was quite a tedious process and we wanted to automate this as well. Further to this, all releases were done from the main branch itself which meant that we had to be extra careful when merging pull requests to the main branch since we didn't want to accidentally release a package which wasn't ready for release.
There was no development branch or anything which matched it. All changes had to be held within the feature branches itself to avoid polluting the main branch. This was quite a hassle since we had to keep track of which feature branches were ready for release and which weren't.
The release process was quite slow. As you might have already seen, the release script at the root level was limited to a concurrency of 1 and the scripts have sequential dependencies on previous scripts. For example bump-version calls pnpm build within itself. This was primarily to ensure that the scripts were run in the order of build ---> bump-version ---> release. This was unnecessary since Turborepo already has more efficient ways to manage this.

Here is a diagram of the release process as it was before and after this development cycle.

Fun fact!

The prerelease tag blizzard was actually named after the Witcher potion Blizzard which is a potion that slows down time and increases perception. We thought it was quite fitting since prereleases are meant to be a slow down of the release process and a way to increase perception of the changes that are being made, a concept which is quite similar to Canary Releases.

How I built it

First off, we had two approaches to solve this:-
Quick way
- Add Prerelease commit prefix evaluation to automatic-versioning and increment the version accordingly. This would mean that we would be able to merge in prerelease changes to the main branch with the associated commit prefix and let the existing release workflow take care of the rest. But this obviously will be polluting the main branch and doesn't solve the problem of every release being published with the latest tag which can be quite dangerous.

Complete way

Extract the existing release workflow into a custom Composite GitHub Action. This would make the exact same release steps reusable for both types of releases. The variables which in this case consists of only the release tag were provided as inputs into this action along with the needed repositry secrets.
Modify the existing release workflow to consume this new action and add in a second prerelease workflow which gets triggered on pushes to development branch. The final workflow files and the action looks like this:-

.github/actions/release/action.yml

name: release
description: Base package release action
inputs:
    npm_token:
        description: "Token to authenticate with the npm registry"
        required: true
runs:
    using: composite
    steps:
      - name: Setup Node
        uses: actions/setup-node@v3
        with:
            node-version: '16.x'
            registry-url: 'https://registry.npmjs.org'

      - name: Configure git
        run: |
            git config --global user.email "github-actions[bot]@users.noreply.github.com"
            git config --global user.name "github-actions[bot]"
        shell: bash

      - run: git fetch --prune --unshallow
        shell: bash

      - name: Install dependencies
        run: npm install -g pnpm@8 && pnpm install --production --ignore-scripts
        shell: bash

      - name: Create .npmrc
        run: echo "//registry.npmjs.org/:_authToken=${{ inputs.npm_token }}" > .npmrc
        shell: bash

      - run: echo "git-checks=false" >> .npmrc
        shell: bash

      - name: Sync submodules
        run: pnpm sync-submodules
        shell: bash

      - name: Populate prerequisities
        run: |
            echo "{\"release_tag\":\"$TAG\"}" > cache-control.json
            for dir in packages plugins; do
            cd "$dir" && for p in */; do
                cp ../{.npmignore,LICENSE,cache-control.json} "$p"
            done && cd ..
            done
        shell: bash

      - name: Publish packages on NPM
        run: |
            pnpm --filter @sliit-foss/automatic-versioning build && npm i -g ./packages/automatic-versioning
            pnpm release
        shell: bash

      - name: Cleanup
        run: rm -rf cache-control.json && rm -rf p*/**/cache-control.json
        shell: bash

      - name: Update release info
        run: |
            git config pull.ff true
            git add . && git commit -m "CI: @sliit-foss/automatic-versioning - sync release" || true
            git pull --rebase && git push origin
        shell: bash

.github/workflows/release.yml

name: CI Release
on:
    push:
        branches:
            - main
    workflow_dispatch:

jobs:
    release:
        runs-on: ubuntu-latest
        env:
            TURBO_TOKEN: ${{ secrets.TURBO_TOKEN }}
            TURBO_TEAM: ${{ secrets.TURBO_TEAM }}
            TAG: latest
        steps:
            - uses: actions/checkout@v3
            - uses: ./.github/actions/release
            with:
                npm_token: ${{ secrets.NPM_TOKEN }}

.github/workflows/prerelease.yml

name: CI Prerelease
on:
    push:
        branches:
            - development
    workflow_dispatch:

jobs:
    release:
        runs-on: ubuntu-latest
        env:
            TURBO_TOKEN: ${{ secrets.TURBO_TOKEN }}
            TURBO_TEAM: ${{ secrets.TURBO_TEAM }}
            TAG: blizzard
        steps:
            - uses: actions/checkout@v3
            - uses: ./.github/actions/release
            with:
                npm_token: ${{ secrets.NPM_TOKEN }}

This on its own was not enough, we had to add in a couple of new features to our library automatic-versioning to be able to support this process. The newly added features are as follows:-
- The ability to specify a prerelease tag as a command line argument when invoking the script.
- The ability to specify a prerelease branch as a command line argument when invoking the script which will essentially change the commit prefix evaluation as follows while in that branch:-
  - Feat! --> Premajor
  - Feat --> Preminor
  - Fix --> Prepatch
- The ability to designate a list of prefixes as ignored prefixes which will not be considered when evaluating the commit prefix. This was useful as we needed to ignore the prerelease version sync commit from the development branch which is prefixed with CI: from being considered when evaluating the commit prefix and to avoid it being considered as an invalid value for version incrementing.
- The ability to recongize the commit history for a specific workspace instead of the whole repository as a whole
- Support for commit scopes. For example we now can add in commits with prefixes as follows -> Feat(automatic-versioning): commit scope support
Further in the midst of this release, we added in the following bugfix to automatic-versioning
- Invalid versioning if the commit contains a url in the commit message such as a merge commit with a source branch url

Problem two got automatically solved with the above. The availablity of a development branch with its own release cycle meant that we could now merge pull requests into the development branch and keep them there as long as we wanted to until they were ready to be merged into the main branch.

Finally to address our third and final problem, we removed the concurrency limit from the turbo script and structured the pipeline in our turbo.json as follows:-

{
    ...
    "pipeline": {
        "build": {
            "dependsOn": [],
            "outputs": ["dist/**"]
        },
        "bump-version": {
            "dependsOn": ["build"],
            "outputs": ["package.json"]
        },
        "release": {
            "dependsOn": ["bump-version"],
            "outputs": []
        }
    }
    ...
}

This indefinitely sped up the release process since the scripts were now run in parallel and the dependencies were handled by Turborepo itself. Further to this, we integrated Vercel Remote Caching in our CI pipeline which meant that all of the steps were now cached and the subsequent runs were much faster. This process turned out be quite simple since we were already using Vercel for a few of our other projects and the integration was quite seamless. Jared Palmer has done a quite a good job of making this process as simple as possible as it just requires 2 environment variables to be set in the CI environment which we added as repository secrets referenced in our workflows. For more information, refer the following docs.

Overall these changes reduced the release time from 1-2 mins to 30-45 seconds which was a huge improvement.

There however was a catch, since we were using Vercel Remote Caching, we had to ensure that the cache was invalidated whenever a new release was made in the main branch. Say for example I'm adding a patch to a package with a version 1.0.0 and merging it with the development branch. This package will be released as 1.0.1-blizzard-0 during the prerelease workflow run of this branch. Now if I merge the same branch to the main branch, in the eyes of Turborepo, the inputs to the release script have not changed since the time it was run in the development branch which will cause Turbo to replay the cache from it thus skipping the actual release of the package to version 1.0.1. Fortunately, dealing this was quite simple, we just needed turbo to maintain two caches for the 2 release types which we did by adding a simple cache-control.json file to each of the project workspaces at the time of the CI run as in the action steps below where the tag is different for the two workflow runs:-

    - name: Populate prerequisities
      run: |
        echo "{\"release_tag\":\"$TAG\"}" > cache-control.json
        for dir in packages plugins; do
          cd "$dir" && for p in */; do
            cp ../{.npmignore,LICENSE,cache-control.json} "$p"
          done && cd ..
        done
      shell: bash

    # Release step

    - name: Cleanup
      run: rm -rf cache-control.json && rm -rf p*/**/cache-control.json
      shell: bash

As an added bonus, we also updated our existing testing workflow to incorporate linting along with unit tests. We could've just added in a separate workflow for linting or another job itself within the existing test.yaml file but that would have been a massive repetition of code. After all only the run command would have been different for both jobs which is why we decided to use the strategy feature of Github Actions to run the same job twice with different run commands as follows:-

.github/workflows/lint-test.yml

name: CI Code Quality + Tests
on:
  pull_request:
    branches:
      - main
      - development

jobs:
  scripts:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        command: ['lint', 'test']
    env:
      TURBO_TOKEN: ${{ secrets.TURBO_TOKEN }}
      TURBO_TEAM: ${{ secrets.TURBO_TEAM }}
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-node@v3
        with:
          node-version: 16

      - name: Install dependencies
        run: npm install -g pnpm@8 && pnpm install --ignore-scripts

      - name: Run checks
        shell: bash
        run: |
          pnpm --filter @sliit-foss/eslint-config-internal build
          pnpm ${{ matrix.command }}
        env:
          GITHUB_ACCESS_TOKEN: ${{ secrets.ACCESS_TOKEN_GITHUB }}
          FIREBASE_CONFIG: ${{ secrets.FIREBASE_CONFIG }}

Additional Resources/Info

Tools

Interesting Links
- Released Packages
- Automation in Action

DEV Community: Akalanka Perera