What I built
We at the SLIIT FOSS Community are a passionate and committed team dedicated to promoting the use and development of open source software. It has been quite some time since we have shifted focus on building NPM and Dart libraries for use by fellow developers. The subject of this post however will be the former where we'll be exploring the use of a plethora of tools and concepts to build a fully automated prerelease mechanism for these libraries.
Category Submission:
DIY Deployments
App Link
https://www.npmjs.com/~sliit.foss
Screenshots
Description
NPM Catalogue isn't exactly an app on it's own. It's just a development environment for our NPM libraries. The main idea behind this project was to have everything required for the development, testing and release of an NPM library from step 1 in one single place making it as easy as possible for developers to contribute to these libraries allowing them to focus on the actual development itself rather than the surrounding processes. Simply, it's one of the many ways we at the SLIIT FOSS Community are trying to make open source development easier for everyone and we hope that this project will be of use to many developers out there.
Link to Source Code
Repository - https://github.com/sliit-foss/npm-catalogue
Full list of changes which were made under this development cycle - https://github.com/Akalanka47000/npm-catalogue/pull/3/files
Note that this is just a fork and the actual workflow runs can be found in the parent repository
Permissive License
MIT
Background
As of today, we have over a dozen packages which have been released publicly to the NPM registry which you can find over here and till now we only released these as either major, minor or patch updates. However as of late, with more complex developments, we had identified the need for prereleases which would primarily help us with a smoother and safer way of getting things done. There were instances that a library was completed to a certain extent, was ready for internal testing but not just yet ready for a full release.
At the time, we had already automated the existing release process with the use of a GitHub Actions Workflow that would be run on a push to the main branch of the repository. This workflow took care preparing the project, executing the required scripts for versioning, releasing the packages to the NPM registry and finally syncing the newly published package versions back into the repository by execting a commit within the workflow itself.
All of our packages are housed within this single repository and it is bootstrapped with Turborepo which was already making things quite easy for us. For example, we could run a single command to execute a script across all packages within the repository and in our case, we had a release script defined within all of our packages and a simple process of running pnpm release would take care of the rest. The release script was organized as follows:-
Root package.json
{
"scripts": {
...
"release": "turbo run release --concurrency=1",
...
}
}
Workspace package.json
{
"scripts": {
...
"build": "node ../../scripts/esbuild.config.js",
"bump-version": "pnpm build && npx automatic-versioning --name=@sliit-foss/<package-name> --no-commit --recursive",
"release": "bash ../../scripts/release.sh",
...
}
}
release.sh
pnpm run bump-version && pnpm publish --access=public --tag=latest || true
Our versioning logic was quite simple, we decided which type of release it was based on the prefix of the commit which triggered the workflow run, for example:-
-
Fix: <commit message>would trigger a patch release -
Feat: <commit message>would trigger a minor release -
Feat!: <commit message>would trigger a major release
This process was quite easy since we anyways follow conventional commits while checking in changes. We have Husky setup with Commitlint which ensures that this safe-zone is never crossed. The version bumping itself is handled by a library of our own making called @sliit-foss/automatic-versioning which evaluates the commit history and
determines the next version based on the type of release.
While this process was working fine, it did have these following problems:-
What if there was a prerelease that needed to be done? This would mean that we would have to manually bump the version of the package, publish it to the NPM registry and then sync the version back into the repository. This was quite a tedious process and we wanted to automate this as well. Further to this, all releases were done from the main branch itself which meant that we had to be extra careful when merging pull requests to the main branch since we didn't want to accidentally release a package which wasn't ready for release.
There was no development branch or anything which matched it. All changes had to be held within the feature branches itself to avoid polluting the main branch. This was quite a hassle since we had to keep track of which feature branches were ready for release and which weren't.
The release process was quite slow. As you might have already seen, the release script at the root level was limited to a concurrency of 1 and the scripts have sequential dependencies on previous scripts. For example
bump-versioncallspnpm buildwithin itself. This was primarily to ensure that the scripts were run in the order ofbuild ---> bump-version ---> release. This was unnecessary since Turborepo already has more efficient ways to manage this.
Here is a diagram of the release process as it was before and after this development cycle.
Fun fact!
- The prerelease tag
blizzardwas actually named after the Witcher potion Blizzard which is a potion that slows down time and increases perception. We thought it was quite fitting since prereleases are meant to be a slow down of the release process and a way to increase perception of the changes that are being made, a concept which is quite similar to Canary Releases.
How I built it
First off, we had two approaches to solve this:-
-
Quick way
- Add Prerelease commit prefix evaluation to
automatic-versioningand increment the version accordingly. This would mean that we would be able to merge in prerelease changes to the main branch with the associated commit prefix and let the existing release workflow take care of the rest. But this obviously will be polluting the main branch and doesn't solve the problem of every release being published with thelatesttag which can be quite dangerous.
- Add Prerelease commit prefix evaluation to
-
Complete way
- Extract the existing release workflow into a custom Composite GitHub Action. This would make the exact same release steps reusable for both types of releases. The variables which in this case consists of only the release tag were provided as inputs into this action along with the needed repositry secrets.
- Modify the existing release workflow to consume this new action and add in a second prerelease workflow which gets triggered on pushes to development branch. The final workflow files and the action looks like this:-
.github/actions/release/action.yml
name: release description: Base package release action inputs: npm_token: description: "Token to authenticate with the npm registry" required: true runs: using: composite steps: - name: Setup Node uses: actions/setup-node@v3 with: node-version: '16.x' registry-url: 'https://registry.npmjs.org' - name: Configure git run: | git config --global user.email "github-actions[bot]@users.noreply.github.com" git config --global user.name "github-actions[bot]" shell: bash - run: git fetch --prune --unshallow shell: bash - name: Install dependencies run: npm install -g pnpm@8 && pnpm install --production --ignore-scripts shell: bash - name: Create .npmrc run: echo "//registry.npmjs.org/:_authToken=${{ inputs.npm_token }}" > .npmrc shell: bash - run: echo "git-checks=false" >> .npmrc shell: bash - name: Sync submodules run: pnpm sync-submodules shell: bash - name: Populate prerequisities run: | echo "{\"release_tag\":\"$TAG\"}" > cache-control.json for dir in packages plugins; do cd "$dir" && for p in */; do cp ../{.npmignore,LICENSE,cache-control.json} "$p" done && cd .. done shell: bash - name: Publish packages on NPM run: | pnpm --filter @sliit-foss/automatic-versioning build && npm i -g ./packages/automatic-versioning pnpm release shell: bash - name: Cleanup run: rm -rf cache-control.json && rm -rf p*/**/cache-control.json shell: bash - name: Update release info run: | git config pull.ff true git add . && git commit -m "CI: @sliit-foss/automatic-versioning - sync release" || true git pull --rebase && git push origin shell: bash.github/workflows/release.yml
name: CI Release on: push: branches: - main workflow_dispatch: jobs: release: runs-on: ubuntu-latest env: TURBO_TOKEN: ${{ secrets.TURBO_TOKEN }} TURBO_TEAM: ${{ secrets.TURBO_TEAM }} TAG: latest steps: - uses: actions/checkout@v3 - uses: ./.github/actions/release with: npm_token: ${{ secrets.NPM_TOKEN }}.github/workflows/prerelease.yml
name: CI Prerelease on: push: branches: - development workflow_dispatch: jobs: release: runs-on: ubuntu-latest env: TURBO_TOKEN: ${{ secrets.TURBO_TOKEN }} TURBO_TEAM: ${{ secrets.TURBO_TEAM }} TAG: blizzard steps: - uses: actions/checkout@v3 - uses: ./.github/actions/release with: npm_token: ${{ secrets.NPM_TOKEN }}-
This on its own was not enough, we had to add in a couple of new features to our library
automatic-versioningto be able to support this process. The newly added features are as follows:-- The ability to specify a prerelease tag as a command line argument when invoking the script.
-
The ability to specify a prerelease branch as a command line argument when invoking the script which will essentially change the commit prefix evaluation as follows while in that branch:-
- Feat! --> Premajor
- Feat --> Preminor
- Fix --> Prepatch
The ability to designate a list of prefixes as ignored prefixes which will not be considered when evaluating the commit prefix. This was useful as we needed to ignore the prerelease version sync commit from the development branch which is prefixed with
CI:from being considered when evaluating the commit prefix and to avoid it being considered as an invalid value for version incrementing.The ability to recongize the commit history for a specific workspace instead of the whole repository as a whole
Support for commit scopes. For example we now can add in commits with prefixes as follows -> Feat(automatic-versioning): commit scope support
-
Further in the midst of this release, we added in the following bugfix to
automatic-versioning- Invalid versioning if the commit contains a url in the commit message such as a merge commit with a source branch url
- Problem two got automatically solved with the above. The availablity of a development branch with its own release cycle meant that we could now merge pull requests into the development branch and keep them there as long as we wanted to until they were ready to be merged into the main branch.
- Finally to address our third and final problem, we removed the concurrency limit from the turbo script and structured the pipeline in our turbo.json as follows:-
{
...
"pipeline": {
"build": {
"dependsOn": [],
"outputs": ["dist/**"]
},
"bump-version": {
"dependsOn": ["build"],
"outputs": ["package.json"]
},
"release": {
"dependsOn": ["bump-version"],
"outputs": []
}
}
...
}
This indefinitely sped up the release process since the scripts were now run in parallel and the dependencies were handled by Turborepo itself. Further to this, we integrated Vercel Remote Caching in our CI pipeline which meant that all of the steps were now cached and the subsequent runs were much faster. This process turned out be quite simple since we were already using Vercel for a few of our other projects and the integration was quite seamless. Jared Palmer has done a quite a good job of making this process as simple as possible as it just requires 2 environment variables to be set in the CI environment which we added as repository secrets referenced in our workflows. For more information, refer the following docs.
Overall these changes reduced the release time from 1-2 mins to 30-45 seconds which was a huge improvement.
There however was a catch, since we were using Vercel Remote Caching, we had to ensure that the cache was invalidated whenever a new release was made in the main branch. Say for example I'm adding a patch to a package with a version 1.0.0 and merging it with the development branch. This package will be released as 1.0.1-blizzard-0 during the prerelease workflow run of this branch. Now if I merge the same branch to the main branch, in the eyes of Turborepo, the inputs to the release script have not changed since the time it was run in the development branch which will cause Turbo to replay the cache from it thus skipping the actual release of the package to version 1.0.1. Fortunately, dealing this was quite simple, we just needed turbo to maintain two caches for the 2 release types which we did by adding a simple cache-control.json file to each of the project workspaces at the time of the CI run as in the action steps below where the tag is different for the two workflow runs:-
- name: Populate prerequisities
run: |
echo "{\"release_tag\":\"$TAG\"}" > cache-control.json
for dir in packages plugins; do
cd "$dir" && for p in */; do
cp ../{.npmignore,LICENSE,cache-control.json} "$p"
done && cd ..
done
shell: bash
# Release step
- name: Cleanup
run: rm -rf cache-control.json && rm -rf p*/**/cache-control.json
shell: bash
As an added bonus, we also updated our existing testing workflow to incorporate linting along with unit tests. We could've just added in a separate workflow for linting or another job itself within the existing test.yaml file but that would have been a massive repetition of code. After all only the run command would have been different for both jobs which is why we decided to use the strategy feature of Github Actions to run the same job twice with different run commands as follows:-
.github/workflows/lint-test.yml
name: CI Code Quality + Tests
on:
pull_request:
branches:
- main
- development
jobs:
scripts:
runs-on: ubuntu-latest
strategy:
matrix:
command: ['lint', 'test']
env:
TURBO_TOKEN: ${{ secrets.TURBO_TOKEN }}
TURBO_TEAM: ${{ secrets.TURBO_TEAM }}
steps:
- uses: actions/checkout@v3
- uses: actions/setup-node@v3
with:
node-version: 16
- name: Install dependencies
run: npm install -g pnpm@8 && pnpm install --ignore-scripts
- name: Run checks
shell: bash
run: |
pnpm --filter @sliit-foss/eslint-config-internal build
pnpm ${{ matrix.command }}
env:
GITHUB_ACCESS_TOKEN: ${{ secrets.ACCESS_TOKEN_GITHUB }}
FIREBASE_CONFIG: ${{ secrets.FIREBASE_CONFIG }}
Additional Resources/Info
-
Interesting Links
Top comments (0)