The chronicles of semantic-release and monorepos

Since semantic-release and lerna appeared, the community has made several attempts to combine these technologies together. One does not simply run release in parallel.

There are several significant hurdles. The first one is the analyzeCommits step. semantic-release tries to resolve a new version by making a cut of commits from the previous release tag in repo, but has no idea how these changes are related to packages.
The second issue is publish. Or prepare. Or rather what happens between.

await plugins.prepare(context);

  if (options.dryRun) {
    logger.warn(`Skip ${nextRelease.gitTag} tag creation in dry-run mode`);
  } else {
    // Create the tag before calling the publish plugins as some require the tag to exists
    await tag(nextRelease.gitTag, nextRelease.gitHead, {cwd, env});
    await addNote({channels: [nextRelease.channel]}, nextRelease.gitHead, {cwd, env});
    await push(options.repositoryUrl, {cwd, env});
    await pushNotes(options.repositoryUrl, {cwd, env});
    logger.success(`Created tag ${nextRelease.gitTag}`);
  }

  const releases = await plugins.publish(context);

When these git add ., git commit -m and git push --tags origin queues from several async "threads" clash in runtime they produce unstashed changes and block each other.
The third problem is cross-dependencies. In most cases monorepository packages are interconnected, so it's necessary to update versions of these references in some way.

2015

atlassian/lerna-semantic-release
If I'm not mistaken, it seems this was the first working solution. The key idea of LSR is to disassemble semantic-release for parts and use them to construct several step-like pipelines and run them with Lerna

# Pre
lerna-semantic-release pre # Set up the versions, tags and commits
# Perform
lerna-semantic-release perform # Publishes to npm
# Post
lerna-semantic-release post # Generates a changelog in each package

Under the hood we see custom commits analyzer, custom changelog generator, custom npm publisher and so on. Unfortunately, this tool design does not provide standard semrel extending features and any other plugins support. Moreover, once semrel changes its inners, this dramatically affects LRS. Still, it was worth a try.

2017

pmowrer/semantic-release-monorepo
Another approach was proposed by Patrick Mowrer. He did not try to overcome the limitations of parallel running, and suggested that releases be performed sequentially.

lerna exec --concurrency 1 -- npx --no-install semantic-release -e semantic-release-monorepo

An important step forward was the use of semrel standard extension techniques. In essence, only one plugin required customization — analyzeCommits which was supplemented with commit-by-package filtering. Some implementation flaws are easily fixed with an additional hook, but... cross-dependency problem was not considered at all.

2019

dhoulb/multi-semantic-release
Dave Houlbrooke focused on implantation
of synchronization points inside the release flow. If we cannot should not change the semrel runner code, we can write a plug-in that will provide a coherence state of parallel threads. Each step of synthetic Inline plugin injects wait conditions in order to make sure all concurrent releases are in the same phase.

async function generateNotes(pluginOptions, context) {
    // Set nextRelease for package.
    pkg._nextRelease = context.nextRelease;

    // Wait until all todo packages are ready to generate notes.
    await wait(() => todo().every(p => p.hasOwnProperty("_nextRelease")));

But it’s not enough just to keep parallel releases at one step. Some actions, as we know, lie between plugin[step] calls. And with a moderately large number of packets the problem of unstashed changes appears again. However, it turned out this issue can be easily fixed by replacing internal asynchronous calls with synchronous ones.
The killer feature of this solution is cross-dependencies update. MSR shared milticontext knows about which packages will be updated as part of all releases, so its possible to update all the manifest files too.

// Loop through localDeps to update dependencies/devDependencies/peerDependencies in manifest.
    pkg._localDeps.forEach(d => {
    // Get version of dependency.
    const release = d._nextRelease || d._lastRelease;

    // Cannot establish version.
    if (!release || !release.version)
        throw Error(`Cannot release because dependency ${d.name} has not been released`);

    // Update version of dependency in manifest.
    if (manifest.dependencies.hasOwnProperty(d.name)) manifest.dependencies[d.name] = release.version;
    if (manifest.devDependencies.hasOwnProperty(d.name)) manifest.devDependencies[d.name] = release.version;
    if (manifest.peerDependencies.hasOwnProperty(d.name))
                    manifest.peerDependencies[d.name] = release.version;
            });

2020

qiwi/multi-semantic-release
Dave's solution works great for a small number of packages (<20). But the implemented mechanism that interlocks releases phases is extremely slow for huge "enterprise" monorepos. This fork of dhoulb/multi-semantic-release replaces setImmediate loops and mentioned execa.sync hook with event-driven flow and finally makes possible to run the most release operations in parallel.

// Shared signal bus.
    const ee = new EventEmitter();

    // Announcement of readiness for release.
    todo().forEach((p) => (p._readyForRelease = ee.once(p.name)));

    // Status sync point.
    const waitFor = (prop, filter = identity) => {
        const promise = ee.once(prop);
        if (
            todo()
                .filter(filter)
                .every((p) => p.hasOwnProperty(prop))
        ) {
            ee.emit(prop);
        }
        return promise;
    };
     ...
const publish = async (pluginOptions, context) => {
            pkg._prepared = true;
            const nextPkgToProcess = todo().find((p) => p._nextType && !p._prepared);

            if (nextPkgToProcess) {
                ee.emit(nextPkgToProcess.name);
            }

            // Wait for all packages to be `prepare`d and tagged by `semantic-release`
            await waitFor("_prepared", (p) => p._nextType);

202x

There're a lot of monorepo related issues in semrel repo. NPM's intention to standardize workspace notation brings moderate optimism about the fate of monoreps. As soon as monoreps become more common practice, semrel will probably add built-in support for them.

Comments

[Andy Edwards]

I've figured out a working config for using semantic-release with no custom plugins (though I had to use a fork of @semantic-release/npm that includes this fix).

The hardest part was separating versioning for each subpackage, and it mainly comes down to the difficulty of configuring conventional-commits tools to take the subpackage scope when parsing commits and generating the changelog; it's just barely possible to accomplish this with their limited config format, and not at all convenient. It would be much easier if they accepted customization functions in their config.

Here's how I build the full semantic-release config for a given subpackage.

function makeSubpackageReleaseConfig(packageName, allPackageNames) {
  const otherPackageNames = allPackageNames.filter(pkg => pkg !== packageName)
  return {
    branches: [
      '+([0-9])?(.{+([0-9]),x}).x',
      'main',
      'next',
      'next-major',
      { name: 'beta', prerelease: true },
      { name: 'alpha', prerelease: true },
    ],
    // use separate git tags for each subpackage
    tagFormat: `${packageName}-v\${version}`,
    plugins: [
      [
        require.resolve('@semantic-release/commit-analyzer'),
        {
          preset: 'conventionalcommits',
          releaseRules: [
            // prevent commits like fix(otherPackageName): ... from triggering a release in this package
            ...otherPackageNames.flatMap((scope) => ({
              scope,
              release: false,
            })),
            ...[
              // allow commits like fix(packageName): ... to trigger a release in this package
              packageName,
              // scope: undefined allows unscoped commits like fix: ... to trigger a release in all monorepo pkgs
              undefined
            ].flatMap((scope) => [
              { breaking: true, scope, release: 'major' },
              { revert: true, scope, release: 'patch' },
              { type: 'feat', scope, release: 'minor' },
              { type: 'fix', scope, release: 'patch' },
              { type: 'perf', scope, release: 'patch' },
            ]),
            // don't trigger a release for any other types of commits
            { scope: undefined, release: false },
          ],
        },
      ],
      [
        require.resolve('@semantic-release/release-notes-generator'),
        {
          preset: 'conventionalcommits',
          presetConfig: {
            types: [
              { type: 'build', section: 'Build System', hidden: true },
              { type: 'chore', section: 'Build System', hidden: true },
              {
                type: 'ci',
                section: 'Continuous Integration',
                hidden: true,
              },
              { type: 'style', section: 'Styles', hidden: true },
              { type: 'test', section: 'Tests', hidden: true },
              ...[
                { type: 'docs', section: 'Documentation' },
                { type: 'feat', section: 'Features' },
                { type: 'fix', section: 'Bug Fixes' },
                { type: 'perf', section: 'Performance Improvements' },
                { type: 'refactor', section: 'Code Refactoring' },
              ].flatMap((cfg) => [
                // include commits like fix(packageName): ... in the release notes for this package
                { ...cfg, scope: packageName, hidden: false },
                // exclude commits like fix(otherPackageName): ... from the release notes for this package
                ...otherPackageNames.map((otherPkg) => ({
                  ...cfg,
                  scope: otherPkg,
                  hidden: true,
                })),
                // include unscoped commits like fix: ... in the release notes for all packages
                { ...cfg, hidden: false },
              ]),
            ],
          },
        },
      ],
      require.resolve('@jcoreio/semantic-release-npm'),
      require.resolve('@semantic-release/github'),
    ]
  }
}

[Jucian0]

github.com/nrwl/nx with github.com/jscutlery/semver

[Anton Golub]

github.com/semrel-extra/zx-bulk-re...

[Billie Hilton]

Thank you for such a thorough review of the history + state-of-the-art!

[Renate Winther Ravnaas]

2021: Anyone who tried this: npmjs.com/package/auto ?

[Cameron Yick]

This is a great option, as long you're not in a monorepo using Yarn 2 workspaces that are connected using workspace:* references. However, this thread has examples with links to a few lerna-powered yarn-1 repos if people are looking for examples.

github.com/intuit/auto/discussions...

[Anton Golub]

@renatewr ,

Interesting tip, thanks. But there is a single versioning for all packages as I can see: github.com/intuit/auto/commit/146f... This is acceptable as long as there are just a few packages in repo. For hundreds, this is already becoming a problem)

[Renate Winther Ravnaas]

I have not tried it yet, but the docs says: ‘The npm plugin works out of the box with lerna in both independent and fixed mode. ’ If that would sove the problem?

[Renate Winther Ravnaas]

npmjs.com/package/@auto-it/npm