Yarn workspaces and change detection

Hello dev.to community!

Perviously I wrote a short article about “How to create your own modern yarn cli bundle” where I explained how we can build own CLI containing our plugins set. The one important plugin from my own CLI - enhanced workspace plugin which can help to provide more predictable and controllable change detection mechanism helping to optimise CI/CD and reduce additional operations on developer machines. But before…a short history and motivation to create another one bicycle in npm repo.

Motivation

The long time ago in the one Galaxy my R&D team where I’m participating decided to choose Node.js + TypeScript as the main platform for FE & BE Web products in order to have the one efficient and scalable environment. Good idea! But after creation the many product our ecosystem we faced to the problem: we have many repositories and is really time consuming thing to manage all PRs over them, link them between each other, organise version management, request our CI/CD sequently and etc.
The one obvious solution is to reduce repositories count and group them by the one monorepo. Fortunately we have many tools solving this problem: Bazel, Lerna, NX, Rush and Yarn. I my third article I have a plan to compare them by details and examples, but for now I just shortly say that after long discussions we decided to start from something quickest, easiest and extendable for us - yarn from the second version which provides flexible and good API for many cases.

We updated our Jenkins Pipeline to support Yarn V2 workspaces, designed a project transaction plan to monorepos and everything was good except performance. Yarn workspace plugin has (till 3.1.1 version) has two critical issues for us:

• Wildcard operation has problem with appropriate topological ordering. In rare cases parent dependencies can be built only after children.
• It doesn’t support the partial detection only for changed from some snapshot but it’s really important feature for any environment! (I know that is working from NX under the box)

In this time we still wanted to support Yarn and decided to start a creation of our Magic tool doing everything and specially:

• Provide robust dependencies tree control for monorepo dependencies.
• Help to determine what workspace was changes after previous commit or specific branch (master/main).
• Build graphs helping to organise parallel CD/CD jobs.
• Speedup development and CI/CD life by above points.

Great ambitious! And I can say that they were been achieved in this plugin: https://github.com/RuBAN-GT/yarn-plugin-enhanced-workspaces, hope it can help to other developers and make the Yarn more great and popular.

And for now more details how does it work.

Enhanced workspace plugin details

First of all sorry for the code, if you’ll find any moments that can be improved, feel free to open issues or your PRs.
And for now about organisation.

When I started to design core principles of my plugin I marked the several key moments:

1. Tree manager - the tool helping to build a workspace dependency tree with dependencies chunks & breadcrumbs that should forecasts appropriate ordering and allow the group nodes by groups in the future.
2. Change detection manager - the tool which can distinguish necessary/unnecessary nodes by following specific config parameters, diffs from commits or branches or other rules.
3. Group manager - the tool allows to combine tree nodes by groups with configurable limitations and ordering, it also ranks them in order to solve problems with the following dependency ordering (parent should be utilised before children).

Following these concepts the plugin exposes commands allowing to get nodes in different variants and provide operations over them like original yarn workspaces foreach but with own logic and configuration.

Instead of copy-pasting README I’ll show how to start to work with this plugin.

How to work

To get started everything whatever you need is to execute the command from your project:

yarn plugin import https://raw.githubusercontent.com/RuBAN-GT/yarn-plugin-enhanced-workspaces/master/bundles/%40yarnpkg/plugin-enhanced-workspaces.js

It should import plugin into your project.

After it, for example, we have a common frontend monorepo where we have publishable packages as libraries and a one application.

Is better to have a convention for your project in order to keep consistent infrastructure. For example you decided to provide build script from each project and package. We decided to use root package.json and ask our pipeline to try to execute build script:

yarn workspaces changed foreach -p --ancestors --ignored-ancestors-markers="dist" run build

This command tells:
“Please, look on changed workspaces from previous master/main commit, perform for them build command in parallel, if some of them require parent workspace, build it as well BUT if it already has dist directory on package folder - skip it.”

Simple but efficient. Moreover we can setup incremental builds using default TypeScript if we use set appropriate paths for example.

You can play with already configured environment at https://github.com/RuBAN-GT/monorepo-demo/tree/master/yarn.

In our case, our pipeline after all builds and tests will try to publish public packages (workspaces) using yarn workspaces changed list --no-private by gathering all not private changed packages and using default yarn --cwd=workspace_path npm publish --tolerate-republish to publish them.

Summary

I hope this solution can help to organise your development and CI/CD environment and speed up your processes. It doesn’t solve issues with watchers or caching but from the other hand is better that nothing) Moreover it may be can help somehow to the yarn community.

If you have any questions, ideas, suggestions, feel free to ask!

Good code everyone!