Since semantic-release and lerna appeared, the community has made several attempts to combine these technologies together. One does not simply run release in parallel.
There are several significant hurdles. The first one is the analyzeCommits step. semantic-release tries to resolve a new version by making a cut of commits from the previous release tag in repo, but has no idea how these changes are related to packages.
The second issue is publish. Or prepare. Or rather what happens between.
await plugins.prepare(context);
if (options.dryRun) {
logger.warn(`Skip ${nextRelease.gitTag} tag creation in dry-run mode`);
} else {
// Create the tag before calling the publish plugins as some require the tag to exists
await tag(nextRelease.gitTag, nextRelease.gitHead, {cwd, env});
await addNote({channels: [nextRelease.channel]}, nextRelease.gitHead, {cwd, env});
await push(options.repositoryUrl, {cwd, env});
await pushNotes(options.repositoryUrl, {cwd, env});
logger.success(`Created tag ${nextRelease.gitTag}`);
}
const releases = await plugins.publish(context);
When these git add ., git commit -m and git push --tags origin queues from several async "threads" clash in runtime they produce unstashed changes and block each other.
The third problem is cross-dependencies. In most cases monorepository packages are interconnected, so it's necessary to update versions of these references in some way.
2015
atlassian/lerna-semantic-release
If I'm not mistaken, it seems this was the first working solution. The key idea of LSR is to disassemble semantic-release for parts and use them to construct several step-like pipelines and run them with Lerna
# Pre
lerna-semantic-release pre # Set up the versions, tags and commits
# Perform
lerna-semantic-release perform # Publishes to npm
# Post
lerna-semantic-release post # Generates a changelog in each package
Under the hood we see custom commits analyzer, custom changelog generator, custom npm publisher and so on. Unfortunately, this tool design does not provide standard semrel extending features and any other plugins support. Moreover, once semrel changes its inners, this dramatically affects LRS. Still, it was worth a try.
2017
pmowrer/semantic-release-monorepo
Another approach was proposed by Patrick Mowrer. He did not try to overcome the limitations of parallel running, and suggested that releases be performed sequentially.
lerna exec --concurrency 1 -- npx --no-install semantic-release -e semantic-release-monorepo
An important step forward was the use of semrel standard extension techniques. In essence, only one plugin required customization — analyzeCommits which was supplemented with commit-by-package filtering. Some implementation flaws are easily fixed with an additional hook, but... cross-dependency problem was not considered at all.
2019
dhoulb/multi-semantic-release
Dave Houlbrooke focused on implantation
of synchronization points inside the release flow. If we cannot should not change the semrel runner code, we can write a plug-in that will provide a coherence state of parallel threads. Each step of synthetic Inline plugin injects wait conditions in order to make sure all concurrent releases are in the same phase.
async function generateNotes(pluginOptions, context) {
// Set nextRelease for package.
pkg._nextRelease = context.nextRelease;
// Wait until all todo packages are ready to generate notes.
await wait(() => todo().every(p => p.hasOwnProperty("_nextRelease")));
But it’s not enough just to keep parallel releases at one step. Some actions, as we know, lie between plugin[step] calls. And with a moderately large number of packets the problem of unstashed changes appears again. However, it turned out this issue can be easily fixed by replacing internal asynchronous calls with synchronous ones.
The killer feature of this solution is cross-dependencies update. MSR shared milticontext knows about which packages will be updated as part of all releases, so its possible to update all the manifest files too.
// Loop through localDeps to update dependencies/devDependencies/peerDependencies in manifest.
pkg._localDeps.forEach(d => {
// Get version of dependency.
const release = d._nextRelease || d._lastRelease;
// Cannot establish version.
if (!release || !release.version)
throw Error(`Cannot release because dependency ${d.name} has not been released`);
// Update version of dependency in manifest.
if (manifest.dependencies.hasOwnProperty(d.name)) manifest.dependencies[d.name] = release.version;
if (manifest.devDependencies.hasOwnProperty(d.name)) manifest.devDependencies[d.name] = release.version;
if (manifest.peerDependencies.hasOwnProperty(d.name))
manifest.peerDependencies[d.name] = release.version;
});
2020
qiwi/multi-semantic-release
Dave's solution works great for a small number of packages (<20). But the implemented mechanism that interlocks releases phases is extremely slow for huge "enterprise" monorepos. This fork of dhoulb/multi-semantic-release replaces setImmediate loops and mentioned execa.sync hook with event-driven flow and finally makes possible to run the most release operations in parallel.
// Shared signal bus.
const ee = new EventEmitter();
// Announcement of readiness for release.
todo().forEach((p) => (p._readyForRelease = ee.once(p.name)));
// Status sync point.
const waitFor = (prop, filter = identity) => {
const promise = ee.once(prop);
if (
todo()
.filter(filter)
.every((p) => p.hasOwnProperty(prop))
) {
ee.emit(prop);
}
return promise;
};
...
const publish = async (pluginOptions, context) => {
pkg._prepared = true;
const nextPkgToProcess = todo().find((p) => p._nextType && !p._prepared);
if (nextPkgToProcess) {
ee.emit(nextPkgToProcess.name);
}
// Wait for all packages to be `prepare`d and tagged by `semantic-release`
await waitFor("_prepared", (p) => p._nextType);
202x
There're a lot of monorepo related issues in semrel repo. NPM's intention to standardize workspace notation brings moderate optimism about the fate of monoreps. As soon as monoreps become more common practice, semrel will probably add built-in support for them.
Latest comments (8)
github.com/nrwl/nx with github.com/jscutlery/semver
github.com/semrel-extra/zx-bulk-re...
2021: Anyone who tried this: npmjs.com/package/auto ?
This is a great option, as long you're not in a monorepo using Yarn 2 workspaces that are connected using
workspace:*references. However, this thread has examples with links to a few lerna-powered yarn-1 repos if people are looking for examples.github.com/intuit/auto/discussions...
@renatewr ,
Interesting tip, thanks. But there is a single versioning for all packages as I can see: github.com/intuit/auto/commit/146f... This is acceptable as long as there are just a few packages in repo. For hundreds, this is already becoming a problem)
I have not tried it yet, but the docs says: ‘The npm plugin works out of the box with lerna in both independent and fixed mode. ’ If that would sove the problem?
npmjs.com/package/@auto-it/npm
Thank you for such a thorough review of the history + state-of-the-art!