DEV Community: Maël Nison

Yarn 3.2 🚢🔮 Libc, Yarn Explain, Next Major, ...

Maël Nison — Mon, 21 Feb 2022 13:21:32 +0000

Welcome to the release notes for Yarn 3.2! This release is a little smaller than the 3.0 and 3.1, as we've hold off on some changes in preparation for our next major ... but more on that later 😃

As always, keep in mind those are only the highlights, the full changelog is much more comprehensive. And if you just happen to love reading our release posts, here are the past entries 👇

Sponsoring

The Yarn org needs your help to make our work more sustainable! Please take a look at our OpenCollective and GitHub Sponsors pages for more details 😊

Libc Field

We implemented in 3.1 a feature we call "conditional dependencies". The idea is simple: if a package is listed in the optionalDependencies field and its os / cpu fields don't match the current machine, we don't install them at all. This pattern is today used by various tools like Esbuild or SWC to avoid overfetching dependencies that systems wouldn't needed.

One problem however is that while os and cpu are useful at differentiating systems, they aren't the only parameters at play. In particular, knowing the standard C library against which native modules are built is critical: using a module linked against the glibc with a Node release built against musl would promptly crash.

To avoid this, Yarn now supports a libc array field in the package.json that currently accepts any of two values: glibc and musl. Just like os and cpu, packages will be skipped if they don't match the host libc.

This isn't the final iteration; while libc is a good improvement, more parameters could be taken into account. Both Yarn and npm have open proposals to address this situation, and we'll see what we decide to implement.

New Command: `yarn explain`

It can be difficult to know how to react when facing errors. Our website tries to help with that by providing detailed explanations, but when you're in your terminal this might not be the first thing you have in mind.

The new yarn explain command will let you get all the details about an error, right from your terminal:

In the future we'll expand the documentation to cover more error messages, and may use yarn explain to aggregate some of the other similar mechanisms we already have (such as yarn explain peer-requirements).

UI Improvements

Every version we look for little UI annoyances to fix. This time is no exception with a couple of neat improvements:

The resolution step will now have a spinning wheel; we can't show a percent-based progress since we don't know how many packages we'll have to resolve until the end, but a spinner will at least let you know the process isn't stuck.
Errors thrown when cloning Git repositories were previously reported as regular stack traces. They will now have dedicated output.

Next Major

With 3.2 out of the door, we'll now start working on the next major release: Yarn 4! We have an issue highlighting the things we currently have in mind, but generally speaking expect us to decrease the friction when starting new projects. Some highlights:

We'll drop support for Node 12, as it will reach EOL in April
We'll be exploring a new resolution algorithm that will prevent most of the attacks similar to the recent color.js hijacking.
More commands will be integrated with Git; we used to refrain from doing so due to some related projects using Mercurial, but this isn't the case anymore. Projects not using Git will still be able to use Yarn, but some features may not be available there.
- To give you an idea of the kind of integration we have in mind, the yarn stage command (already available as a plugin) allows to automatically commit all dependency-related changes without impacting any other.

The official plugins will be shipped by default, to reduce the friction. In practice the Yarn binary is very small, so we have some leeway to bundle everything together so that you don't have to download more subparts.
- Even if bundled by default they'll technically remain plugins, so it doesn't change anything for third-party plugin authors: our plugin API will remain a focus for us, and will keep improve.

And more! 🙂 We have plenty of other ideas to improve Yarn, so expect to see a significant amount of improvements in our next major - including lower friction when starting new projects or migrating older ones.

Yarn 3.1 🎃👻 Corepack, ESM, pnpm, Optional Packages ...

Maël Nison — Mon, 25 Oct 2021 15:05:15 +0000

Welcome to the release notes for Yarn 3.1! We're quite excited by this release, as it brings various improvements that we've all been looking forward to. Let's dig into that!

As always, keep in mind those are only the highlights, the full changelog is much more comprehensive. And if you just happen to love reading our release posts, here are the past entries 👇

Sponsoring

The Yarn org needs your help to make our work more sustainable! Please take a look at our OpenCollective and GitHub Sponsors pages for more details 😊

Table of content

Node.js Corepack Integration
ESM Support
New Install Mode: pnpm
Conditional Packages
Smart Changeset Filters
New Workspace Syntax: workspace:^

Improvements

Node.js Corepack Integration

Did you know that Yarn now ships with Node? This is done via the Node.js Corepack project, which includes both the Yarn and pnpm binaries as shims. By adding the packageManager field to your package.json, you can enforce the use of a specific package manager & package manager version in a completely transparent way:

{
  "packageManager": "yarn@3.1.0"
}

Note that Corepack is available starting from Node.js 16.9, but is currently opt-in. Don't forget to run corepack enable a single time to make sure the shims are globally installed!

We also improved in 3.1 the init command to properly support Corepack: running yarn init -2 will now automatically setup a Yarn Modern project, setting its packageManager field as required 💫

ESM Support

ESM has always been supported when using the node_modules linker, since it's the same old install strategy that Node has always supported. However, with PnP taking ownership of the resolution pipeline, compatibility with ESM wasn't a given and had to be implemented using the Loader Hook API.

While the Loader Hook API isn't entirely stable yet, a large amount of work has been made lately and our team has been able to produce an initial experimental support for ESM modules. It should be enabled automatically if we detect that one of the packages in your dependency tree contains a "type": "module" field, but you can enable or disable it manually through your settings:

pnpEnableEsmLoader: true

Being experimental, it's possible that some bugs may arise or that new Node releases bring some breaking changes around the API. Be sure to report issues on our bug tracker!

New Install Mode: `pnpm`

The pnpm package manager was one of the first tools to advocate for using symlinks when installing packages within the node_modules folder. While we went another way with PnP, we decided that the implementation cost was low enough that it would be worth adding support for this symlink-based install strategy as well.

Starting from Yarn 3.1, you can try out symlink-based installs by adding the following setting to your .yarnrc.yml file:

nodeLinker: pnpm

Conditional Packages

Esbuild and swc are two native packages that gained a lot of attention lately thanks to their impressive performances over their competitors. They recently revamped how their packages are built to avoid complex postinstall scripts, but did so in a way that was less efficient than before for Yarn projects.

Yarn 3.1 features a new optimization that kicks in when a package is listed as optionalDependencies and lists os and/or cpu fields. When that happens, Yarn will skip fetching and installing those packages unless they match the current system parameters.

In case you need to manually configure a strict set of package architectures to support (for example like in a zero-install case, where you want to read from an immutable set of packages), you can use the supportedArchitectures setting:

supportedArchitectures:
  os: [linux, darwin]
  cpu: [x64, arm64]

Smart Changeset Filters

The yarn workspaces foreach and yarn workspaces list commands now ships with brand new --since flags. When set, those commands will only execute against the packages that changed when compared to the main branch (either main or master, depending on the branches in your repository).

This can come in handy if you wish to only run builds in some specific workspaces, or just get a list of the workspaces which changed for scripting purposes:

yarn workspaces foreach --since run eslint .
yarn workspaces list --since

The --since flag also accepts an optional argument (--since=${commit-ish}) to manually define a source from which the changes should be derived.

New Workspace Syntax: `workspace:^`

Workspaces supported a special syntax via workspace:*, with those ranges being replaced at publish-time by exact ranges corresponding to the real version of the target workspace. However, if you wanted to use a caret instead of an exact range, you had to use the verbose workspace:^x.y.z form, which Yarn updated repo-wide after each publish.

Yarn now supports workspace:^ and workspace:~ as well, making it much easier to cross-reference workspaces within a monorepo where most packages are intended to be published, by preventing a good amount of the merge conflicts that used to happen after Yarn updated the verbose ranges.

Additionally, as a special case, this syntax is now allowed in the peerDependencies field as well:

{
  "peerDependencies": {
    "@my/other-package": "workspace:^"
  }
}

Yarn 3.0 🚀🤖 Performances, ESBuild, Better Patches, ...

Maël Nison — Mon, 26 Jul 2021 16:06:24 +0000

Hello! Long time no see! Back in December, we decided to start working on our next major release, the 3.0. It took a bit of time to do everything we intended to do, but here we are! So let's talk a bit about what it changes, and what it brings. Note that these are only the highlights, the full changelog is much more comprehensive.

And if you just love to read our log posts, here are the past entries 👇

Governance

Back when the project was started in 2017, we didn't took the time to establish a formal governance document. This is now addressed, and our team composition can be found here. It doesn't change anything in practice (this is how we worked for more than two years now), but we hope it may give you a better understanding as to how we work and reach consensus.

OpenCollective

From 2017 to 2019 Yarn was mostly maintained by Facebook engineers. While it worked relatively well, the Yarn 1 -> 2 release also proved to be the right time to expand our active team to other horizons, and nowadays no two of our active contributors work at the same company - and none at Facebook.

Consequently, we've decided to setup an OpenCollective (or GitHub Sponsors) to give our supporters a way to both express their thanks to our team, and give us resources we can then inject back into the project.

Breaking Changes

While the migration from Yarn 1 to Yarn 2 brought some discomfort, the migration to Yarn 3 should prove easier - regardless of the version you come from. The user-facing breaking changes we made this season are mostly little details that may only affect you in very specific cases:

Node 10 isn't supported anymore
Plug'n'Play hooks are now called .pnp.cjs (vs .pnp.js)
Virtual folders are now called __virtual__ (vs $$virtual)
The editor SDKs have been moved to @yarnpkg/sdks
Etc; full list here

Even for Yarn 1 users, migrating from 1 to 3 should be easier: we made it so that Yarn will detect when this situation arises to then automatically enable the node-modules linker. That alone should address most of the problems you may have been hitting when attempting the upgrade - and for everything else, make sure to take a look at our Migration Guide which got significantly improved over the past year.

Support for the `exports` field

When using Yarn 3 w/ PnP, the exports field will be properly resolved regardless of your Node version. If you're not familiar with this field, you can see it as a way to:

Replace the main field
Soft-prevent accessing arbitrary files in the package
Conditionally remap files depending on the context (bundlers, ...)

Performances

Various tweaks have been made to address some of the largest resource consumptions in Yarn. Installs have been improved (turning us faster than pnpm in some scenarios, which is quite a feat!), but not only: script execution tends to have a natural overhead, but bugs in 2.4 and prior caused this overhead to grow relative to the size of the project itself. This is no longer the case, and the overhead should now be constant.

New `node_modules` linkers

As you may know, Yarn is built around a few interfaces. One of them is called a "linker", and tells Yarn how to install packages on disk. It's how we can support both PnP and node_modules installs without changing much code.

One advantage of this architecture is how it allows us to efficiently iterate on alternative install strategies. For this release, larixer implemented a new experimental nmMode setting that can be used to instruct the linker to use a specific copy scheme:

hardlinks-local will use hardlinks when the same package is found multiple times within the same project (but only if they have exactly the same version at the moment).
hardlinks-global will use hardlinks on identical files (even across different versions!), but will also make them point to a global content-addressable directory. This is similar to what pnpm does. Note that if the cache is corrupted (for example because you manually edited it), Yarn will automatically repair it on subsequent installs.

I myself have been playing with a pnpm-style linker. It hasn't shipped yet since I'm cautious about adding complexity that could end up unmaintained, but given how small it is there's a decent chance we could add it in a later release as an experimental install mode.

Improved Shell

As you may know, given that system shells are rarely portable across Windows and Posix, Yarn no longer uses them to run your scripts entries. Instead, we use our very own shell interpreter.

We're happy to report that this shell just got smarter, and now provides two additional syntaxes that you can reliably use on both Windows and Posix:

build-js & build-css &    # Background jobs
ls 2>/dev/null            # File descriptor redirections

Additionally, background jobs have their output color-coded, so you can clearly identify their output, even interlaced.

ESBuild support

We now use ESBuild to generate the Yarn bundles and as such worked to ensure good compatibility with Plug'n'Play installs. The result is the new @yarnpkg/esbuild-plugin-pnp package which lets you transparently build your code using the default Yarn installation mode. It's still relatively young, so feel free to drop us an issue if you notice something strange!

While it won't change much for most end-users, the move to ESBuild also provided decent build speed improvements (around 6x faster), making it less frustrating to build Yarn from sources ✨

New plugin APIs

Yarn supports writing plugins that can inject themselves into various places and leverage some of the builtin modules provided by the core. While we didn't get the chance to make all the improvements we hoped, we've still been able to upgrade the command line framework to Clipanion 3, which lets you write intuitive type-checked commands with a minimal syntactic overhead.

Conclusion

According to DEV.to, those change notes take about 5 minutes to read - by contrast, the 2.0 release post was a whopping 15 minutes! Of course, this time around we didn't need to fully rewrite Yarn, hence a lower amount of "critical information" we need you to be aware of 😉 We expect that to be the norm from now on: majors won't have a lot of super impactful changes, mostly just some architecture cleaning and modernization, as new features will tend to land in minors.

As for our team, we're very happy of the work we've been doing! Working on the codebase still feels like a treat, and features are often constrained to a few identifiable files - proving that our initial redesign bet was right. Our stats suggest that the result are visible to our users as well, and while I remain cautious about popularity metrics it's certainly nice to see.

Finally, remember that Yarn now has an OpenCollective / GitHub Sponsors! If your company benefits from our work, or would like to see particular fixes land, sponsoring the project is a good way to engage with us 🙂

What's to come?

A few features initially slated for 3.0 have been pushed back to the next minor so that we have more time to properly incubate them. Some of the things we have in motion:

Corepack integration
ESM support under PnP mode
Builtin CLI completion
Changelog generation
Improved performances
pnpm-style linker
And more...!

Of course that's only on the top of my head, so it's possible our objectives shift during the next weeks depending on our own priorities - and of course depending on whether you help us or not 😛

Yarn 💞 GitHub Sponsors

Maël Nison — Thu, 28 Jan 2021 12:14:14 +0000

The Yarn organization now accepts individual & corporate sponsorships via both GitHub Sponsors and OpenCollective! 🌟

Ever since its inception in 2016 the project operated without official structure allowing it to receive donations. While we made it work, developing expert tools like package managers revealed itself an extremely resource-intensive task. While our companies have been understanding, maintaining Yarn isn't part of our official job statements and many hours have to be spent maintaining the project and supporting our users during our spare time.

As we're working on Yarn 3, we hope donations will help our core team justify this work while also giving us the resources needed to sometimes delegate parts of the maintenance to professionals (be it for external security reviews, website redesign, documentation proofreading, etc), thus further improving the quality of our production and letting us focus on building the very tool and workflows we all use everyday.

If you're interested to know more, feel free to ping our team on our Discord server and we'll be happy to help!

Yarn 2.4 🎄🎁 Log Filters, Audits, Better Warnings, ...

Maël Nison — Mon, 30 Nov 2020 14:52:14 +0000

Hey everyone! It's this time of the year where everyone is slowly preparing for the holidays. This year will probably be slightly different, but I can't wait to at least take a well deserved time off. But before that, let's talk about our next minor Yarn release, and a little bit about the next-next release: Yarn 3!

Oh, and in case you missed our previous release notes, you can find them all here in all their glorious emojiness: 👇

Plugins

We'll try to reference external plugins made by our community in our release notes, so if you made one that you want to share, please ping us! We're also looking at adding a page on our website to list them all, improving discoverability 💫

For now, let me present those two:

yarn.build by ojkelly is a fast monorepo builder for Yarn. In a sense it's similar to yarn workspaces foreach but more opinionated, and thus easier to adapt to existing workflows. It parallelises builds, shows what's being executed, and generates zipped archives suitable for AWS and similar platforms.
prod-install by Larry1123 and NETSVS is a much more powerful version of yarn workspaces focus that copies the selected workspaces into a target location before transforming it to become self-sufficient - the final directory thus being ready to be efficiently cached and deployed via Docker layers.

Audits

Both Yarn 1 and npm had this handy little feature called audit. Originally developed by npm when they acquired Lift, this command lets you quickly check whether some of your dependencies have known vulnerabilities, which may come in handy in some types of application. Unfortunately, since the audit endpoint isn't documented, its implementation wasn't entirely obvious.

Thanks to our contributors, Yarn 2.4 now includes proper audit, available via the yarn npm audit command! And to make up for the delay, we've implemented various interesting ways to run it, under the form of the -A,--all and -R,--recursive options - check the examples for details!

We've also significantly improved the output to be more in line with the rest of the CLI, providing information in a more compact way:

This new output is compatible with the --json flag, meaning that you can leverage the information obtained from yarn npm audit --json from any script you want - even the command-line itself, using tools like jq!

Better Warnings

Peer dependencies have always been a difficult concept to grasp. They are not that hard per se (a peer dependency is always satisfied by the exact package instance used by the parent of the package listing it), but various other factors played into it and caused typical installs to produce many rarely actionable warnings.

No more!, do we say. Starting from 2.4, you can expect the warnings produced by Yarn to become vastly better than what we used to report. For this first release with warnings being a focus, we've implemented a new range merging algorithm that lets us drastically decrease the amount of warnings we emit. The idea is simple: imaging the following dependency tree:

.
└── your project/
    ├── @storybook/react/
    │   ├── (peer) react@^15
    │   ├── storybook-plugin-foo/
    │   │   └── (peer) react@^15
    │   └── storybook-plugin-bar/
    │       └── (peer) react@^15
    └── react@17

Before, these are the warnings you'd have had:

your project provides react@17 to @storybook/react, which isn't compatible with react@^15
your project provides react@17 to storybook-plugin-foo, which isn't compatible with react@^15
your project provides react@17 to storybook-plugin-bar, which isn't compatible with react@^15

From all those warnings, only one was truly actionable: the @storybook/react one. The two others were mere byproducts from the first, and were just making the output harder to read. This is now fixed, and Yarn will instead report:

your project provides react@17 (pXYZ), which doesn't satisfy what @storybook-react and its dependents request

The pXYZ is a hash that you can use with a new command, yarn explain peer-requirements <hash>, to get the exact list of packages that contribute to the final peer dependency requirement, and whether they are met or not. For instance, this is what I get in one of my projects:

Log Filters

Even if warnings will get smarter, there's always this one case where you really don't care about a specific message. For instance the message saying that a package wasn't in the cache is sometimes controversial, with half of our users liking it, and the other half wanting to hide it.

While you could use preferAggregateCacheInfo to tweak that, it's only about one message. What about others? Well, starting from 2.4 we introduce a new setting called logFilters. It has the following syntax:

logFilters:
  - code: YN0005
    level: discard

With this configuration, all messages matching the specified code (which would be builds being disabled, per our documentation) will be removed from the output. And if you need to only remove a single line, it's possible as well:

logFilters:
  - text: "core-js@npm:2.6.11 lists build scripts, but its build has been explicitly disabled through configuration."
    level: discard

We hope this feature will let you tune your package managers to watch what you truly care about, which can be different from one person to the other.

And also

As always, these release notes focus exclusively on the big-picture stuff - as always, there's a lot more things that have been improved under the hood. Check our changelog for a comprehensive list, but we can mention:

Updated our patches to account for TS 4.1 and FSEvents 2.1.2
Improved usability when using the global cache
Improved usability in the VSCode ZipFS extension
Improved performances on recurrent installs
Improved Windows compatibility when running binaries
Improved the display for yarn upgrade-interactive
Fixed the postinstall scripts run by yarn workspaces focus
Fixed some edge cases with || and interpolation errors
Added support for proxy settings (caFilePath, ...)
... and more!

What about Yarn 3?

This is a big news for us! Yarn 2.4 is expected to be the last minor release for the 2.x line! After a year of development, we now have put aside enough items to be worth addressing in a new major.

While the 3.x branch will be much less disruptive than the jump from 1.x to 2.x (after all we won't need to rewrite the whole codebase this time! 😁), it'll include a few breaking changes. Most of those are listed here but, as you'll see, they are mostly about old workflows being deprecated, and are unlikely to affect most codebases.

One important note though: given that Node 10 will reach its end-of-life in April, it's likely that Yarn 3 will be Node 12+ only. So if you want to prepare for it, start considering upgrading to either Node 12 or, better yet, 14!

Yarn 2.3 🍦✨ Info Command, Detailed Options, Nohoist, ...

Maël Nison — Fri, 02 Oct 2020 10:56:46 +0000

Howdy! Another big month just went by, 2020 confirming being a very weird year for everyone. I hope things will be ok for you, wherever you are.

As for Yarn itself, we're happy to meet you again to talk a bit about the highlights for the work we've done in the third minor of the Yarn 2 release line! Remember that we try to limit these blog posts to about three core items, and that the exhaustive list will always be in our repository. Check it out sometime, there's some very good stuff there too! 📝

And in case you missed our previous release notes, don't forget to take a look at them for even more awesome stuff: 2.1 | 2.2 😃

Don't know how to upgrade? It's easy: just run yarn set version berry in your project, and you'll get the latest build. Want to skip the upgrade? Just revert the changes!

Info command

Every now and then, we have this dependency that we want to know more about. We want to know its authors, we want to know its license, we want to know its size ... there's a lot of thing we want to know! And sometimes, we want to retrieve those information from many different packages at once.

Yarn already provides the yarn npm info command, but this command is a bit special in that it prints by default the latest info from the npm registry ... and that's not necessarily what we are using!

To address some of the problem with this command, we're now introducing a new top-level command, yarn info. It looks like this:

First, wow, that's pretty! But there are a few interesting things we can note about this display:

It prints by default the information based on what's currently used by the active workspace. I could use the -A,--all or -R,--recursive flags to change that, though!
It prints less information than yarn npm info. For instance, there's little point in printing the README content as a raw single-line string, so we removed it. By default, yarn info will only print the most relevant information.
But it prints more information than yarn npm info as well! For instance, because we passed the --cache flag, it also reported the size of the package in the cache, and its exact location.

There are many other gems in the command. By passing the --manifest flag you also get additional fields like the license or the homepage. By passing the --json flag you generate a stream of data that can be easily transformed using jq. You can even add your own data sections if you want, by using our plugin system! Ever wanted a place to show the number of downloads for your dependencies? Their CVEs? Their maintainers? Just use the provided hook, and all those information are yours to give!

Option Documentation

You might not be aware of it, but Yarn uses a pretty unique CLI framework: Clipanion. Very few tools have as much requirements as we do, and it was very important for us to be able to fix bugs and implement features without decreasing our velocity.

In the latest Clipanion update, our contributors implemented a syntax to individually document options. Another one took this new feature, and went over every command, documenting each option one by one. The result looks absolutely great:

Because our CLI is the source of our website's documentation, you can find the exact same information online. We hope this effort will prove useful to you, as you discover new features you weren't even aware of until now!

Nohoist

As package manager authors, we try to do our best to support the ecosystem, sometimes going as far as building features just to help one single large project migrating to better practices. In 2017, in order to let React Native users use our newly released workspaces, we implemented a feature called nohoist.

Nohoist was a bit weird. It accepted glob patterns, and presumably the paths matching this glob pattern couldn't be hoisted. But what if their ancestors were hoisted? Was it meant to support targeting deep packages? After all, it was really only meant to help React Native users in one specific case. Because the feature itself wasn't entirely clear, it suffered from many bugs over the years, where noone really knew what to do of it. In Yarn 2.0, we decided to completely remove it.

Now, the problem is, React Native still doesn't support workspaces without help. And we like React Native users. So we've been looking for a way to reintroduce something similar to nohoist, but in a way that actually made sense to us. That's where we introduce you to hoisting limits:

nodeLinker: node-modules
nmHoistingLimits: workspaces

By configuring the nmHoistingLimits setting to workspaces when using the node_modules linker, Yarn will prevent packages from being hoisted past the workspaces that transitively depends on them. In practice, it means that you don't need to care about the specific hoisting glob patterns: just declare where the hoisting limit is, and Yarn will take care of the rest.

This design is interesting, because it allows us to support one additional feature: "safe hoisting". See, one problem with the classic hoisting is that it makes it very likely that you're going to eventually start referring to dependencies without explicitly listing them. Then your users install your packages, and all hell breaks loose.

By configuring nmHoistingLimits on dependencies, Yarn will prevent packages from being hoist past their transitive top-level dependent. It may seem a bit arcane, said like this, but it's actually quite simple! Imagine the following project:

With the default hoisting, it would turn into the following, mistakenly letting you access all dependencies as your own:

With nmHoistingLimits set on dependencies, Yarn will instead generate the following, ensuring that you won't ever be able to mistakenly require dependencies you don't list as your own:

Of course it has its own drawbacks, since the imperfect deduplication also means an heavier disk footprint and slower installs, but it may provide a good safety valve until you can migrate to standard PnP installs.

What's to come?

With Hacktoberfest coming, now is as good a time as ever to let you know about our issues labelled Good First Issues! In fact, we wrote a whole article about it a few days ago:

Yarn 🎊 Hacktoberfest

Maël Nison ・ Sep 28 '20

#hacktoberfest #node #webdev #npm

As for the features we're planning for Yarn 2.4, our focus are currently on:

Adding back yarn audit with revamped output
New changelog generation capabilities
PnP support for the exports field, and ESM in general
And more...!

Of course that's only on the top of my head, so it's possible our objectives shift during the next weeks depending on our own priorities - and of course depending on whether you help us or not 😛

One very long-term topic we're starting to explore are package support for non-
JavaScript languages (think C++, Python, Rust, PHP, ...). We already have a few ideas (we have an experimental branch generating CMake files, and another contributor played with Python), and we'll keep evaluating the work needed to get there during the next few months. If you're familiar with any of those ecosystems and are interested in helping Yarn become the universal package manager, please contact us on Discord!

Yarn 🎊 Hacktoberfest

Maël Nison — Mon, 28 Sep 2020 11:36:40 +0000

As you may know, every October the dev community at large starts an event called Hacktoberfest. The idea is simple: a whole month specially designed to help onboard people with little to no prior experience contributing to open-source codebases. In return, you get t-shirts, a valuable experience, and maybe will find a community you'll want to be a part of. Win-win!

October is almost there, so it's a good time to think: what are some interesting tasks you could tackle? Not too hard but still impactful? I happen to lead the work on Yarn, the famous JavaScript package manager, and I can tell you: we have a bit of everything! Whether you prefer frontend work or brain teasers, I'm sure we'll find you great project to hone your skills 😃

I've listed some tasks I have in mind, but please feel free to join our little community on Discord! My fellow maintainers and me will be happy to give you more pointers, ideas, help, and feedback. Until then, happy Hacktoberfest! 🚀

If you're fluent in React, we're looking to revamp the package search interface on our website. We're displaying a lot of information, and we'd like to highlight the most important ones in as few clicks as possible.

If you're a great technical writer, we could really use your help refining our documentation. We're often not native english speakers, and I personally make typos, grammatical errors, or sometimes even forget words in the of my sentences.

If you'd like working on the CLI, we have various tasks tagged Good First Issue that we've identified as being useful without requiring more than a minimal amount of context.

If you want to take a look at the infra side, talk with us! There are various places where we'd benefit from better typings, automated workflows, extra tests, or many other small things that let us focus on designing new features.

Yarn Tips, part 1

Maël Nison — Sun, 13 Sep 2020 16:04:26 +0000

After leading Yarn's development for a bit more than three years now, I noticed that many of our users weren't aware of some very useful features and settings. While often documented on our website, most of the time it was a case of "I didn't even know I could solve my problem this way". To help you discover them, I started the following initiative:

Maël

@arcanis

In this thread: a bunch of tips and “Did you know” about Yarn (and Yarn 2 in particular), updated daily ⬇

14:37 PM - 31 Aug 2020

32 108

This series of article will go over this thread, 5 tweets each time, to dive into them and provide more information than tweets would allow. And if you find these tips interesting, don't forget to follow me on Twitter to learn more about Yarn and its behind-the-scene development 😃

1. The node_modules linker

Maël

@arcanis

1 ✌️ Yarn 2 handles node_modules projects just fine. In fact, our test suites proved our n_m linker to be more stable than in Yarn 1! Only use PnP if you want too 🙂
yarnpkg.com/configuration/…

14:37 PM - 31 Aug 2020

4 19

Click to learn more!

Funnily, we've had a few people opening issues on our bug tracker to report that "Yarn doesn't generate a node_modules when I run an install, something is wrong!". Then we ask them to still try to run their programs and, lo and behold, it works! This is due to Yarn 2 using a new install strategy by default, that we call Plug'n'Play.

Still, Plug'n'Play isn't for absolutely everyone. Some older projects aren't compatible with it, and some others see preexisting bugs being exposed, usually under the form of missing dependency reports.

As regular users we don't necessarily want to have to deal with all that, so Yarn still features the good old node_modules strategy. Just one setting to set, an install to run, and you'll find your node_modules back! Without the Plug'n'Play stability and performance improvements, of course, but at least you'll be in known territory and ready to pursue your work.

2. The `packageExtensions` settings

Maël

@arcanis

2 🧩 But if you use PnP, and if a project doesn’t list its dependencies, just use `packageExtensions` to fix it in a few seconds!
yarnpkg.com/configuration/…

14:37 PM - 31 Aug 2020

0 11

Click to learn more!

That being said, enabling the node_modules linker should probably be seen as a last resort option. After all, there are (many) reasons we decided to make it the exception rather than the norm. But how to proceed when working with problematic packages that don't list all their dependencies? Do we need to wait for them to fix that? Fortunately not.

The packageExtensions setting is the recommended way to fix such problems. Just declare yourself the dependencies that the relevant packages are missing, and let Yarn deal with that. I myself use it quite a lot in my projects, with for example:

packageExtensions:
  "@apollo/client@*":
    dependencies:
      "@wry/context": "*"
    peerDependenciesMeta:
      "react":
        optional: true
      react-dom:
        optional: true

And from time to time, don't forget you can send PRs upstream to solve this problem durably both for you and your community!

3. Yarn's E2E battery

Maël

@arcanis

3 🔭 Yarn runs E2E tests against various projects from the community every four hours. If something looks off (new release missing a dependency?), we’re typically first aware, and often contribute fixes ourselves
github.com/yarnpkg/berry#…

14:37 PM - 31 Aug 2020

2 21

Click to learn more!

While a large part of Yarn 2 has been to, well, implement Yarn, we also wanted to help fix the ecosystem. After all, what good is it to build a package manager that enforces strict rules if nobody is following them? So we submitted many, many PRs, adding missing dependencies to projects that forgot them, explaining the danger, and making sure that everyone had time to address those problems before our release in January.

Still, even projects that actively took steps to fix those issues sometimes had regression. For example, large OSS projects like Next.js, because of their very high velocity, sometimes merged PRs that didn't list new dependencies. In the worst cases, we only noticed it when someone opened an issue on our bug tracker. Not great!

And so, after some time, we decided to start implementing our own set of E2E tests. Baked by a bunch of GitHub workflows, they run every four hours and let us know should anything awry happen in the ecosystem. Thanks to this, we help ensure that the JavaScript ecosystem not only trends towards sanity, but doesn't deviate from the goal even when we're not looking 😉

4. Cache integrity

Maël

@arcanis

4 🔥 Yarn 2 checks the cache integrity between each install. It’s a tiny bit slower, but you get the certainty that your project is in a good state if `yarn install` exits without error code 🙂

14:37 PM - 31 Aug 2020

0 12

Click to learn more!

Back in v1 our cache could, sometimes, be corrupted. It was a fairly rare occurrence, but in theory it could happen. And, should this happen, it was impossible for Yarn to detect it, which would eventually lead to very difficult debug sessions.

In the v2, Yarn is much more careful, and runs a full validation pass before attempting to install the packages on the disk. The validation is really fast (I think it takes less than 2 seconds on our own repository), and it protects against various problems. For example, since Plug'n'Play can share the same cache between all projects on your machine, it may happen that you want to modify a dependency's sources during a debug session, and forget to revert it. With the integrity check, Yarn will now let you know the problem the next time you run yarn install!

5. File cloning

Maël

@arcanis

5 👥 Yarn will use native file cloning when available, so you’ll only pay the size of your cache once, not per project
nodejs.org/api/fs.html#fs…

14:37 PM - 31 Aug 2020

0 10

Click to learn more!

One interesting thing that made node_modules installs very slow was way the installs and the cache interacted together. Back in Yarn 1, our cache contained the unpacked archives, and we copied each file into each project during installs. We used file cloning as well (using fs.copyFile), but because of the sheer amount of files, a lot of I/O still had to be performed.

In Yarn 2, the pipeline is quite different. Plug'n'Play allows us to keep the dependencies compressed, meaning that there isn't any difference between the cache and the install artifacts! Because of this, even in the absolute worst case, we only have to perform the clone operation exactly one time for each package - compared to the hundreds of times it previously was.

Yarn 2.2 🚅🌟 Dedupe, Faster, Lighter, ...

Maël Nison — Fri, 28 Aug 2020 12:16:40 +0000

I hope you enjoyed the summer! As for us, we've been hard at work, and this update comes with its good chunk of improvements in various aspects. As usual we keep a detailed list in our repository, but let's go over the highlights!

Don't know how to upgrade? It's easy: just run yarn set version berry in your project, and you'll get the latest build. Want to skip the upgrade? Just revert the changes!

Dedupe command

One of Yarn's core values is predictability. We want you to be confident that your project won't suddenly change in unexpected ways. The lockfile is a large part of this, ensuring that you always get the same dependencies during install, now or in the future.

To explain what the dedupe command is, I first need to explain a bit the lockfile format. In Yarn, we have descriptors (a combination of package name and range), and we associate them with references (versions). A lockfile essentially stores which reference is linked to a specific range.

So what happens when you add new ranges? For example if you already have lodash@^4.0.0 in your lockfile, resolved to 4.0.0, and suddenly add lodash@^4.1.0? Since this new range isn't compatible with the old one, Yarn will need to resolve it on its own - let's say to 4.1.0. And now is the interesting part - remember when I said that Yarn tries to be predictable, and thus avoid to update things unless ordered to do so? In this case, it means that lodash@^4.0.0 will not be updated to use 4.1.0, even if they'd be compatible. Instead, it will keep using whatever else it was using before, meaning that you'll end up with both 4.0.0 and 4.1.0 in your tree.

Functionally this isn't a problem, because both ranges will use versions compatible with what they advertise. In practice however, it may cause your lockfile to grow needlessly over time as it starts referencing multiple copies of packages, despite the fact that they would have been compatible if the lockfile had been allowed to make wider changes.

The new yarn dedupe command is our solution to that. By default, it will apply a resolution pass that will go over each range and use the highest compatible version that's already in the lockfile. This has various advantages:

It doesn't require the network, so very fast
In the end, most duplicates will be removed
It's very predictable: the highest version wins

Of course, if you have incompatible ranges (for example ^1 and ^2), they won't be deduped together, since that would lead to invalid trees. In this case, you'll have to fix your dependencies to remove references to the older range.

Finally, if you want this kind of check to happen on your CI, the -c,--check option will cause the dedupe algorithm to report an error if optimizations would be possible.

Performances

Better performances lead to better UX, and Yarn is a lot about a good UX. To this end, we've done various improvements in the 2.2 to improve the performances on real-world projects. For instance, Gatsby on cold cache went 92s → 83s, and 17s → 13s on hot cache.

And because we think we should do better than flaunt about perf increases without live numbers to back them up and publicly track regressions, we've setup a live dashboard with our friends at Datadog that shows the results of the daily benchmarks we run against most common package managers. We're pretty happy about the results!

Note that Yarn currently does a bit more work than its siblings on cold cache installs because we need to convert the registry archives in zip format, more suitable for the usage we have. As registries get better at this, we expect cold cache performances to drastically improve 🚅

Size

Since we're recommending checking-in the Yarn binary in your repository, we better be careful about how large we are. Our team made various improvements in this regard, and Yarn 2.2 is now exactly 1.8MB large. To give you an idea:

Yarn Classic is ~5MB large
pnpm is 35MB
npm is 61MB

So, yeah. 1.8MB is nice, isn't it? 🙂

Telemetry

One interesting change in the v2 is that we're going to enable basic opt-out telemetry. The full details are here, but the gist is that we hope this will allow us to spend more time working on Yarn itself, and with a better understanding of how it's used in our community at large - which will then help inform the tradeoffs we make.

The telemetry payload is easily opt-out, and we're committed to send as little information as possible. As soon as the data starts flowing we plan to build public dashboards (similar to our benchmarks) that will help everyone get a better picture of the project.

Other works

Smaller Improvements

This is only a very short list, as always please look at our official changelog for a comprehensive list, but the 2.2 also ships with:

The shell script language now supports more syntaxes (shell groups { echo foo; echo bar } > bar, basic arithmetic $(($RANDOM + 10)))
The --immutable flag now accepts an immutablePatterns settings that you can use to define additional paths that aren't allowed to change during an install - useful to prevent changes to .pnp.js or other artifacts
Packages referenced via the file: protocol will now update when running yarn add again (they're still stored in the cache - prefer portal: if you want a symlink-like behavior).
The new publishConfig.executableFiles field lets you define paths in your package that should be flagged as executable. By default, since Windows has no way to express the executable flag, only files referenced in the bin field will be marked as such, but sometimes you might need others.
Error messages have been clarified in various contexts, such as when accessing Node builtin within Webpack's browser context, when running yarn add on unknown packages, or when a lingering package.json exists in a parent directory.

Website

Multiple improvements were made on the website. In particular:

The migration guide now features a step-by-step section that should help migrate without having to read the entire documentation beforehand.
The search engine now covers both the manifest and yarnrc pages, making it easier to find information about specific fields.

"Package manager manager"

We are starting discussions with the Node TSC to bundle Yarn with Node in some capacity (the current plan is to ship a shim that would, in turn, install Yarn transparently the first time you call it). The full proposal can be found on the following repository: arcanis/pmm. We strongly advise that you play with it and let us know what you think!

As often, this kind of change benefits from wide support, so if you use Yarn (or pnpm), please feel free to follow the discussion and contribute when relevant. If you don't use either, remember that others do, and shutting the proposal down purely because you wouldn't directly benefit from it may not be representative of an inclusive community.

What's to come?

We'll try to make more regular minor releases from now on, shipping exactly one minor per month (eventually leading up to the release of Yarn 3 in January 2021). Some topics we have in mind for the next one (come help us! we have a lot of Good First Issues!):

All-new yarn info
New changelog generation capabilities
PnP support for the exports field, and ESM in general
And more...!

Of course that's only on the top of my head, so it's possible our objectives shift during the next weeks depending on our own priorities - and of course depending on whether you help us or not 😛

Until then stay safe, wear a mask, and see you next month 😉

Yarn 2.1 🐱‍🏍 Git Workspaces, Focused Installs, Loose mode, Live Playground, ...

Maël Nison — Thu, 09 Jul 2020 14:05:27 +0000

How are you doing since January? So many things happened since then. I hope you're all safe, wherever you are.

As for today, we'll be here to talk about Yarn. And as far as Yarn goes I'm happy to report that our work continued at a very good pace! So good in fact that it's now time to release the next minor build, the 2.1 🎉

Still, don't let this little number trick you: more than 350 pull requests were merged since the previous release! This is an incredible pace for our project, only made possible by the dedicated community that gathered around our favorite tool 🌟

So what's in the 2.1? Many, many things! We'll go over the main items, but a more detailed list can be found in our repository. You should check it out too, there's a lot of interesting tidbits!

Don't know how to upgrade? It's easy: just run yarn set version berry in your project, and you'll get the latest build. Want to skip the upgrade? Just revert the changes!

Linker improvements

Node-modules linker

Some people can't migrate to Plug'n'Play installs just yet. That's fine! Some of our contributors don't use it! Yarn supports node_modules installs too! And thanks to Larixer's impressive work, we're happy to report that even large and complex repositories have successfully upgraded to Yarn 2. And when I say large, I mean freaking massive ones 🤓

In fact, our position is now that the node_modules linker in Yarn 2 is a strict improvement over the v1. Multiple hoisting issues have been identified and fixed, and the workspace support has also been improved significantly.

To give you an idea, back in the v1, Babel had never been able to use the stock Yarn workspaces. We all wanted it to happen, but because of the very tricky nature of self-hosted compiler repositories, it proved very challenging. Until now! The Babel and Jest repositories are now powered by Yarn 2, and that's frankly the best seal of quality we could hope for.

So if you're still on the fence about Plug'n'Play ... don't use it for now! Just migrate for all the other speed and stability and UX improvements 😃

Loose mode

The Plug'n'Play linker also improved, with the introduction of the Loose Mode. In Loose Mode, Yarn will simply warn should the runtime make an unsafe module access, avoiding to throw hard exceptions. This works because we generate at install-time the hoisting map that would have been generated by the node-modules linker, then we use that as a fallback pool for any unspecified dependency. It's still unsafe, but now you can quickly get a bird's eye view of all the potential problems without having to fix them all immediately.

Note that the loose mode isn't enabled by default because, somewhat ironically, it may lead to more verbose executions than the strict mode depending on various factors. In particular, packages that wrapped optional require calls between try/catch blocks won't be able to prevent the warnings from being emitted, thus causing false positive.

Major improvements to the `git:` protocol

Workspace cloning

For the past years, most projects have typically followed an "edit, commit, push, release" workflow - the first three parts happening on GitHub while the fourth one was being delegated to the npm registry. Downloading dependencies from Git was always an option, of course, but it didn't always received the attention it deserved. In particular, cloning specific packages from monorepos was still an unsolved problem.

With Yarn 2.1, this situation changes. Yarn is now able to clone any workspace from any Yarn project. Note that this only works with Yarn projects at the moment due to the lack of yarn workspace <name> run build counterparts on current npm and pnpm releases.

Respectful builds

See, there's a very important misconception that we (as in, the package manager authors, collectively speaking) have failed to address during the past years. They are not interchangeable. You cannot use X instead of Y and expect a reproducible build. Regardless of what the advertisement says, each package manager has its own feature set, and to expect them all to be in sync is fruitless. We sometimes implement features we like from other package managers, of course, but when all is said, each project still has its own characteristics that others will never truly replicate. And that's fine!

So what does that mean for Git builds in particular? Imagine, you want to use a project that's maintained by someone using pnpm. That's fair. Well, until now, if you were referencing this project with a git: dependency, Yarn would clone it, then run yarn install, then yarn pack. All good! But wait ... did it run yarn install? Why not pnpm install? Turns out, there were no good reasons. Package managers aren't interchangeable, as I was saying. If a project is configured with a pnpm-lock.yaml, then using Yarn to install it is wrong, and would lead to unpredictable builds. Clearly, that's not an acceptable behaviour.

And so we fixed it! Yarn will now properly detect which packages managers are meant to be used by projects cloned from git dependencies. If there's a yarn.lock, it'll be Yarn. If there's a pnpm-lock.yaml, it'll be pnpm. And if there's a package-lock.json, npm it is.

CLI Improvements

Readability

The output was very verbose, sometimes hiding important information (especially on CI, where the cache is either always there or never there). Various changes were made to streamline the output and make it easier to digest.

On terminals, only five fetch notifications will be displayed at a time. The sixth one will cause the removal of the oldest one, and so forth.
On CI, Yarn will now print a one-line summary instead of the whole definition (unless configured otherwise).
A new optional setting, preferTruncatedLines, will ensure that infos and warnings only take a single line each, keeping your output clean and tidy.
Most CI systems will offer fold groups on each Yarn step. We're still tweaking a bit this behaviour, and we encourage CI maintainers to reach out to us if you wish to discuss better integrations in this area.

Focused workspaces

The yarn workspaces focus command is a new addition inspired by a 1.x feature of the same name. It allows you to only install the dependencies from one specific workspace (plus its own workspace dependencies), thereby decreasing the install size by a significant factor. Coupled to the --production flag, it's a great tool for developers looking to integrate monorepos with Docker images.

By the way, the focus implementation takes exactly 99 lines of code. If you're curious what a plugin looks like, it's a prime example to keep in mind 😉

Deep accesses from `yarn config get/set`

The yarn config get/set commands now accept deep paths (ie foo.bar), allowing you to access settings with different levels of granularity.

Additionally, the configuration will now always be redacted before being printed (unless requested otherwise), thereby preventing secrets from accidental leaks.

Meta improvements

Cache filenames

Our cache filenames used to be versioned using a global cache key. As a result, each time we had to bump the cache key (for example because we fixed an issue in the tarball conversion algorithm), all file names changed and were causing a fairly large noise in the Git history for people using zero-installs.

This isn't the case anymore, as we made the cache content-indexed. Each file will only ever change if the archive content actually changes! 💫

Playground

One of our contributors put CodeSandbox and Yarn together in an impressive playground. Through it, you can easily build reproduction cases for bugs you encounter, decreasing the time needed for us to understand and fix them.

Documentation index

Thanks to Algolia, the Yarn website is now indexed and can be searched from the status bar. We hope this will allow you to quickly find any information you're looking for - whether it's authentication configuration, gitignore examples, or lexicon entries.

VSCode Zip Filesystem

We've published the Zip FS extension on the VSCode Marketplace. Thanks to the work from Matt Penrice, using the Jump to Definition feature with the extension installed will properly send you to the right files, opened straight from the zip archive.

Note that VSCode has an internal limitation preventing the TypeScript server to cover the files located within zip archives (ie you can Jump to Definition from your sources to zip files, but TypeScript won't show its types once you get there). Please upvote the following issue to raise the ticket's priority (we already made a PR, but it unfortunately got rejected).

Other improvements

Performances

The Plug'n'Play runtime has been further optimized, which may yield significant boost in some cases (in particular ESLint when using the eslint-plugin-import package).
The binary size also received a lot of attention, and the 2.1 Yarn binary now takes 2.35MB, vs 2.91MB for the 2.0.

Ecosystem

Packages can now declare they need to be unpacked in order to be functional using the new "preferUnplugged": true field in the manifest. This will hurt the experience of your users (your project will require hard installs, meaning a heavier footprint and slower installs), so please refrain using this field unless there's absolutely no other choice.

What's to come?

Add a new dedupe command to optimize dependency trees
Add changelog support to the builtin release workflow
Add support for the exports field
Add yarn list & yarn fund
Add the telemetry support (RFC)
And more...!

Of course that's only on the top of my head, so it's possible our objectives shift during the next weeks depending on our own priorities - and of course depending on whether you help us or not 😛

Until then stay safe, wear a mask, and see you next month!

Introducing Yarn 2 ! 🧶🌟

Maël Nison — Fri, 24 Jan 2020 17:26:55 +0000

Hi everyone! After exactly 365 days of very intensive development, I'm extremely happy to unveil the first stable release of Yarn 2. In this post I will explain what this release will mean for our community. Buckle up!

If you're interested to know more about what will happen to Yarn 1, keep reading as we detail our plans later down this post: Future Plans. If you just want to start right now with Yarn 2, check out the Getting Started or Migration guides.

Release Overview

Describing this release is particularly difficult - it contains core, fundamental changes, shipped together with new features born from our own usage.

Highlights

The output got redesigned for improved readability
designed for improved readability
Our CLI commands (yarn add, ...) are now aware of workspaces
Running yarn install can be made optional on per-repo basis
A safer npx counterpart called yarn dlx to run one-shot tools
Run commands on all workspaces with yarn workspaces foreach
Packages can be modified in-place through the patch: protocol
Local packages can be referenced through the new portal: protocol
A new workflow has been designed to efficiently release workspaces
Workspaces can now be declaratively linted and autofixed

But also...

Package builds are now only triggered when absolutely needed
Package builds can now be enabled or disabled on a per-package basis
Scripts now execute within a normalized shell
Peer dependencies now work even through yarn link
The lockfile is now proper YAML
The codebase is now full TypeScript
Yarn can now be extended through plugins

Breaking changes...

Configuration settings have been normalized
Packages must respect their boundaries
Bundle dependencies aren't supported anymore
Packages are stored in read-only archives

Those highlights are only a subset of all the changes and improvements; a more detailed changelog can be found here, and the upgrade instructions are available here.

Frequently Asked Questions

Who should we thank for this release?

A significant amount of work has been done by larixer from SysGears, who crawled deep into the engine with the mission to make the transition to Yarn 2 as easy as possible. In particular he wrote the whole node_modules compatibility layer, which I can tell you is no easy feat!

My thanks also go to everyone who spontaneously joined us for a week or a month during the development. In particular embraser01 for the initial Windows support, bgotink for typing our filesystem API, deini for his contributions to the CLI, and Daniel for his help on the infrastructure migration.

This work couldn't have been possible without the support from many people from the open-source community - I think in particular to Nicolò from Babel and Jordan from Browserify, but they're far from being the only ones: the teams of Gatsby, Next, Vue, Webpack, Parcel, Husky, ... your support truly made all the difference in the world.

And finally, the project lead and design architect for Yarn 2 has been yours truly, Maël Nison. My time was sponsored in large part by Datadog, which is a super dope place to develop JS (which is hiring 😜), and by my fiancé and our cats. Never forget that behind all open-source projects are maintainers and their families.

How easy will it be to migrate to Yarn 2?

Thanks to our beta testers and the general support of the ecosystem we've been able to soften a lot the pain associated with such a major upgrade. A Migration Guide is available that goes into more detail, but generally speaking as long as you use the latest versions of your tools (ESLint, Babel, TypeScript, Gatsby, etc), things should be fine.

One particular caveat however: Flow and React-Native cannot be used at the moment under Plug’n’Play (PnP) environments. We're looking forward to working with their respective teams to figure out how to make our technologies compatible. In the meantime you can choose to remain on Yarn 1 for as long as you need, or to use the node_modules plugin, which aims to provide a graceful degradation path for smoother upgrade (note that it's still a work in progress - expect dragons). More details here.

If you don't want to upgrade all of your projects, just run yarn policies set-version ^1 in the repositories that need to stay on Yarn 1, and commit the result. Yarn will always prefer the checked-in binaries over the global ones, making it the best way to ensure that everyone in your team shares the exact same release!

What will happen to the legacy codebase?

Yarn 1.22 will be released next week. Once done, the 1.x branch will officially enter maintenance mode - meaning that it won't receive further releases from me except when absolutely required to patch vulnerabilities. New features will be developed exclusively against Yarn 2. In practical terms:

The classic repository (yarnpkg/yarn) will move over to yarnpkg/classic to reflect its maintenance status. It will be kept open for the time being, but we'll likely archive it in a year or two.
The modern repository will not be renamed into yarnpkg/yarn, as that would break a significant amount of backlink history. It will remain yarnpkg/berry for the foreseeable future.
The old website will move over to classic.yarnpkg.com, and the new website (currently next.yarnpkg.com) will be migrated to the main domain name.
The yarn package on npm will not change; we will distribute further version using the new yarn set version command.

We expect most of those changes to be completed by February 1, 2020.

In Depth

CLI Output

Back when Yarn was released its CLI output was a good step forward compared to other solutions (plus it had emojis! 🧶), but some issues remained. In particular lots of messages were rather cryptic, and the colours were fighting against the content rather than working with it. Strong from this experience, we decided to try something different for Yarn 2:

Almost all messages now have their own error codes that can be searched within our documentation. Here you'll find comprehensive explanations of the in-and-outs of each message - including suggested fixes. The colours are now used to support the important parts of each message, usually the package names and versions, rather than on a per-line basis.

We expect some adjustments to be made during the following months (in particular with regard to colour blindness accessibility), but over time I think you'll come to love this new display!

Workspace-aware CLI

Working with workspaces can sometimes be overwhelming. You need to keep the state of your whole project in mind when adding a new dependency to one of your workspaces. "Which version should I use? What’s already used by my other workspaces?", etc.

Yarn now facilitates the maintenance of such setups through various means:

yarn up <name> will upgrade a package in all workspaces at once
yarn add -i <name> will offer to reuse the same version as the ones used by your other workspaces (and some other choices)
The version plugin will give you a way to check that all the relevant workspaces are bumped when one of them is released again.

Those changes highlight the new experience that we want to bring to Yarn: the tool becomes an ally rather than a burden.

Zero-Installs

While not a feature in itself, the term "Zero Install" encompasses a lot of Yarn features tailored around one specific goal - to make your projects as stable and fast as possible by removing the main source of entropy from the equation: Yarn itself.

To make it short, because Yarn now reads the vendor files directly from the cache, if the cache becomes part of your repository then you never need to run yarn install again. It has a repository size impact, of course, but on par with the offline mirror feature from Yarn 1 - very reasonable.

For more details (such as "why is it different from checking in the node_modules directory"), refer to this documentation page.

New Command: `yarn dlx`

Yarn 2 introduces a new command called yarn dlx (dlx stands for download and execute) which basically does the same thing as npx in a slightly less dangerous way. Since npx is meant to be used for both local and remote scripts, there is a decent risk that a typo could open the door to an attacker:



$ npx serv # Oops, should have been "serve"

This isn't a problem with dlx, which exclusively downloads and executes remote scripts - never local ones. Local scripts are always runnable through yarn run or directly by their name:



$ yarn dlx terser my-file.js
$ yarn run serve
$ yarn serve

New Command: `yarn workspaces foreach`

Running a command over multiple repositories is a relatively common use case, and until now you needed an external tool in order to do it. This isn't the case anymore as the workspace-tools plugin extends Yarn, allowing you to do just that:



$ yarn workspaces foreach run build

The command also supports options to control the execution which allow you to tell Yarn to follow dependencies, to execute the commands in parallel, to skip workspaces, and more. Check out the full list of options here.

New Protocol: `patch:`

Yarn 2 features a new protocol called patch:. This protocol can be used whenever you need to apply changes to a specific package in your dependency tree. Its format is similar to the following:



{
  "dependencies": {
    "left-pad": "patch:left-pad@1.3.0#./my-patch.patch"
  }
}

Together with the resolutions field, you can even patch a package located deep within your dependency tree. And since the patch: protocol is just another data source, it benefits from the same mechanisms as all other protocols - including caching and checksums!

New Protocol: `portal:`

Yarn 2 features a new protocol called portal:. You can see portal: as a package counterpart of the existing link: protocol. Where the link: protocol is used to tell Yarn to create a symlink to any folder on your local disk, the portal: protocol is used to create a symlink to any package folder.



{
  "dependencies": {
    "@my/app": "link:./src",
    "eslint-plugin-foo": "portal:./pkgs/eslint-plugin-foo"
  }
}

So what's the difference you say? Simple: portals follow transitive dependencies, whereas links don't. Even better, portals properly follow peer dependencies, regardless of the location of the symlinked package.

Workspace Releases

Working with workspaces brings its own bag of problems, and scalable releases may be one of the largest one. Most of large open-source projects around here use Lerna or a similar tool in order to automatically keep track of changes applied to the workspaces.

When we started releasing the beta builds for Yarn 2, we quickly noticed we would be hitting the same walls. We looked around, but existing solutions seemed to have significant requirements - for example, using Lerna you would have to either release all your packages every time, or to keep track yourself of which packages need to be released. Some of that work can be automated, but it becomes even more complex when you consider that a workspace being released may require unrelated packages to be released again too (for example because they use it in their prepack steps)!

To solve this problem, we've designed a whole new workflow available through a plugin called version. This workflow, documented here, allows you to delegate part of the release responsibility to your contributors. And to make things even better, it also ships with a visual interface that makes managing releases a walk in the park!

This workflow is sill experimental, but it works well enough for us that we think it'll quickly prove an indispensable part of your toolkit when building large projects using workspaces.

Workspace Constraints

Workspaces quickly proved themselves being one of our most valuable features. Countless projects and applications switched to them during the years. Still, they are not flawless. In particular, it takes a lot of care to keep the workspace dependencies synchronized.

Yarn 2 ships with a new concept called Constraints. Constraints offer a way to specify generic rules (using Prolog, a declarative programming language) that must be met in all of your workspaces for the validation to pass. For example, the following will prevent your workspaces from ever depending on underscore - and will be autofixable!



gen_enforced_dependency(WorkspaceCwd, 'underscore', null, DependencyType) :-
  workspace_has_dependency(WorkspaceCwd, 'underscore', _, DependencyType).

This other constraint will require that all your workspaces properly describe the repository field in their manifests:



gen_enforced_field(WorkspaceCwd, 'repository.type', 'git') :-
  workspace(WorkspacedCwd).

gen_enforced_field(WorkspaceCwd, 'repository.url', 'ssh://git@github.com/yarnpkg/berry.git') :-
  workspace(WorkspacedCwd).

Constraints are definitely one of our most advanced and powerful features, so don't fret yourself if you need time to wrap your head around it. We'll follow up with blog posts to explore them into details - watch this space!

Build Dependency Tracking

A recurrent problem in Yarn 1, native packages used to be rebuilt much more than they should have. For example, running yarn remove used to completely rebuild all packages in your dependency tree.

Starting from Yarn 2 we now keep track of the individual dependency trees for each package that lists postinstall scripts, and only run them when those dependency trees changed in some way:



➤ YN0000: ┌ Link step
➤ YN0007: │ sharp@npm:0.23.0 must be rebuilt because its dependency tree changed
➤ YN0000: └ Completed in 16.92s
➤ YN0000: Done with warnings in 21.07s

Per-Package Build Configuration

Yarn 2 now allows you to specify whether a build script should run or not on a per-package basis. At the moment the default is to run everything, so by default you can choose to disable the build for a specific package:



{
  "dependenciesMeta": {
    "core-js": {
      "built": false
    }
  }
}

If you instead prefer to disable everything by default, just toggle off enableScripts in your settings then explicitly enable the built flag in dependenciesMeta.

Normalized Shell

Back when Yarn 2 was still young, the very first external PR we received was about Windows support. As it turns out Windows users are fairly numerous, and compatibility is important to them. In particular they often face problems with the scripts field which is typically only tested on Bash.

Yarn 2 ships with a rudimentary shell interpreter that knows just enough to give you 90% of the language structures typically used in the scripts field. Thanks to this interpreter, your scripts will run just the same regardless of whether they're executed on OSX or Windows:



{
  "scripts": {
    "redirect": "node ./something.js > hello.md",
    "no-cross-env": "NODE_ENV=prod webpack"
  }
}

Even better, this shell allows us to build tighter integrations, such as exposing the command line arguments to the user scripts:



{
  "scripts": {
    "lint-and-build": "yarn lint \"$@\" && yarn build \"$@\""
  }
}

Improved Peer Dependency Links

Because Node calls realpath on all required paths (unless --preserve-symlinks is on, which is rarely the case), peer dependencies couldn't work through yarn link as they were loaded from the perspective of the true location of the linked package on the disk rather than from its dependent.

Thanks to Plug’n’Play which can force Node to instantiate packages as many times as needed to satisfy all of their dependency sets, Yarn is now able to properly support this case.

New Lockfile Format

Back when Yarn was created, it was decided that the lockfile would use a format very similar to YAML but with a few key differences (for example without colons between keys and their values). It proved fairly annoying for third-party tools authors, as the parser was custom-made and the grammar was anything but standard.

Starting from Yarn 2, the format for both lockfile and configuration files changed to pure YAML:



"@yarnpkg/parsers@workspace:^2.0.0-rc.6, @yarnpkg/parsers@workspace:packages/yarnpkg-parsers":
  version: 0.0.0-use.local
  resolution: "@yarnpkg/parsers@workspace:packages/yarnpkg-parsers"
  dependencies:
    js-yaml: ^3.10.0
    pegjs: ^0.10.0
  languageName: unknown
  linkType: soft

TypeScript Codebase

While it might not directly impact you as a user, we've fully migrated from Flow to TypeScript. One huge advantage is that our tooling and contribution workflow is now easier than ever. And since we now allow building Yarn plugins, you'll be able to directly consume our types to make sure your plugins are safe between updates.



export interface Package extends Locator {
  version: string | null,
  languageName: string,
  linkType: LinkType,
  dependencies: Map<IdentHash, Descriptor>,
  peerDependencies: Map<IdentHash, Descriptor>,
  dependenciesMeta: Map<string, Map<string | null, DependencyMeta>>,
  peerDependenciesMeta: Map<string, PeerDependencyMeta>,
};

Modular Architecture

I recently wrote a whole blog post on the subject so I won't delve too much into it, but Yarn now follows a very modular architecture.

In particular, this means two interesting things:

You can write plugins that Yarn will load at runtime, and that will be able to access the true dependency tree as Yarn sees it; this allows you to easily build tools such as Lerna, Femto, Patch-Package, ...
You can have a dependency on the Yarn core itself and instantiate the classes yourself (note that this part is still a bit experimental as we figure out the best way to include the builtin plugins when operating under this mode).

To give you an idea, we've built a typescript plugin which will automatically add the relevant @types/ packages each time you run yarn add. Plugins are easy to write - we even have a tutorial -, so give it a shot sometime!

Normalized Configuration

One very common piece of feedback we got regarding Yarn 1 was about our configuration pipeline. When Yarn was released we tried to be as compatible with npm as possible, which prompted us to for example try to read the npm configuration files etc. This made it fairly difficult for our users to understand where settings should be configured.



initScope: yarnpkg
npmPublishAccess: public
yarnPath: scripts/run-yarn.js

In Yarn 2, the whole configuration has been revamped and everything is now kept within a single source of truth named .yarnrc.yml. The settings names have changed too in order to become uniform (no more experimental-pack-script-packages-in-mirror vs workspaces-experimental), so be sure to take a look at our shiny new documentation.

Strict Package Boundaries

Packages aren't allowed to require other packages unless they actually list them in their dependencies. This is in line with the changes we made back when we introduced Plug'n'Play more than a year ago, and we're happy to say that the work we've been doing with the top maintainers of the ecosystem have been fruitful. Nowadays, very few packages still have compatibility issues with this rule.



// Error: Something that got detected as your top-level application
// (because it doesn't seem to belong to any package) tried to access
// a package that is not declared in your dependencies
// 
// Required package: not-a-dependency (via "not-a-dependency")
// Required by: /Users/mael/my-app/
require(`not-a-dependency`);

Deprecating Bundle Dependencies

Bundle dependencies are an artefact of another time, and all support for them has been dropped. The installs will gracefully degrade and download the packages as originally listed in the dependencies field.



{
  "bundleDependencies": [
    "not-supported-anymore"
  ]
}

Should you use bundle dependencies, please check the Migration Guide for suggested alternatives.

Read-Only Packages

Packages are now kept within their cache archives. For safety and to prevent cache corruptions, those archives are mounted as read-only drives and cannot be modified under normal circumstances:



const {writeFileSync} = require(`fs`);
const lodash = require.resolve(`lodash`);

// Error: EROFS: read-only filesystem, open '/node_modules/lodash/lodash.js'
writeFileSync(lodash, `module.exports = 42;`);

If a package needs to modify its own source code, it will need to be unplugged - either explicitly in the dependenciesMeta field, or implicitly by listing a postinstall script.

Conclusion

Wow. That's a lot of material, isn't it? I hope you enjoy this update, it's the culmination of literally years of preparation and obstinacy.

Everything I believe package management should be, you'll find it here. The result is for sure more opinionated than it used to be, but I believe this is the way going forward - a careful planning of the long term user experience we want to provide, rather than a toolbox without directions.

As for me, working on Yarn has been an incredible experience. I'm simultaneously project manager, staff engineer, lead designer, developer relations, and user support. There are ups and downs, but every time I hear someone sharing their Yarn success story my heart is internally cheering a little bit. So do this: tell me what you like, and help fix what you don't.

Happy 2020! 🎄

Plugin systems - when & why? 🧩

Maël Nison — Mon, 30 Dec 2019 15:01:38 +0000

I've recently heard some rants against plugin systems and modular architectures. One particular criticism argued that they're merely marketing keywords, adding significant complexity to a software's architecture for little end value. This criticism makes sense to some degree, and there's a trap to be aware of when designing such systems, but we need to be careful. There are reasons why project health might benefit from a plugin architecture, and they might not be the ones you even had in mind.

Given that plugins hold a central place in the new architecture we've built for Yarn 2, I figured it could be interesting to put my thoughts on paper for future reference. Grab your hat and let's dive into the depth of the Plugin Temple 🤠

Plugins are boundaries

Because plugins make it possible to implement new behaviors in pre-existing software it's easy to see them as a way to open up a project to the outside world. But it's also very easy to forget that they're the exact opposite: a way to add constraints to an architecture.

Imagine the same application implemented twice - the first time as a monolith, and the second time with a typical core + plugins architecture. Now you need to build a new feature:

With the monolithic application, you'll likely be able to do your assignment by tweaking a few modules here and there, adding a few new branches, and possibly adding new fields to the data structures. You may not even need to create new files!
With a well-designed plugin system, it'll be more difficult - you'll need to make sure that your changes go through the predefined core hooks. You won't be able to just change the core logic to fit your new need, so you think hard about the implementation before even starting to code.

The monolithic application sounds better, right? Easier to work with, faster iterations. And that's true, given the few parameters I've exposed! But now consider those additional ones:

Multiple people will work on the codebase. There's even a non-zero chance that no one from the current maintainer team will be there in a year. Worse: it's also quite likely that no one from the current maintainer team was here even a year ago.
Most contributors only ever make a single commit - to fix the one bug they experience. They won't ever come back, and probably don't have any context regarding why things work the way they do.
This software will be used for years, and its userbase will keep growing.

Under those new parameters, the monolith will quickly start to spiral out of control. New features get developed and injected into the core. When something isn't quite possible yet, a few small hacks are used. And it works! Time flows, contributors come and go, and suddenly you start to notice a weird pattern: each feature you develop introduces new bugs. People send PRs to help you fix those bugs, but introduce new ones in the process. Long-forgotten hacks trigger edge cases more and more often. Technical debt creeps in and, eventually, we come to a point where no one dares to make a change.

The plugin architecture, however, survives. Bugs still happen, but because the broken features are typically scoped to a single plugin people who aim to fix them only have to understand the context of the one affected module instead of the whole codebase. Same things for reviews, which can be done by people familiar with the individual plugins rather than the whole application. Core maintainers can focus on the core work, and delegate the plugin implementation to new contributors.

The monolith application is Yarn 1 and its hardcoded code paths. The plugin architecture is Yarn 2 and its specialized hooks.

It's still too early to call it a definitive victory, but after working almost a year on this approach I've already been able to see its first payoffs. Freshly onboarded contributors have been able to focus their efforts on specific subparts of the codebase without having to be aware of all the subtle details of the core implementation. Finding how a feature is implemented is mostly a matter of finding the right file.

Plugins give focus

Working on an open-source project the size of Yarn is challenging for various reasons, but the one we'll focus on in this article is pretty simple: what are the features worth implementing?

Lots of people use Yarn every day and, as a result, we get a lot of pull requests to add new features into our tool. Each time, when we're about to merge them, the same questions pop into our minds: will it be useful? Is it worth the complexity? Will I feel comfortable having to maintain this myself in a year?

Back in the v1, our typical answers were along the line of "well, let's move forward and see what happens". But I can already tell you what happens: some of those features became cornerstones of our offering (like workspaces, or resolution overrides) while others ended up cluttering our codebase (like Bower support, or multilingual support). In almost every case even though the implementations worked in isolation, they happened to hit some weird edge case when used together with other features.

Plugin systems offer a very simple solution to this problem by stating that not everything has to belong to the core. It becomes perfectly fine if a lot of features are first implemented through community plugins, the time we can assess their cost/value ratio.

Even better, if we someday decide that a feature shouldn't be shipped anymore, it's just a matter of removing the plugin from the codebase. Of course, such actions sometimes make parts of the core irrelevant and subject to changes. Thankfully, the resources freed by outsourcing part of the feature development can then be reassigned to allow maintainers to spend more timing on keeping up to date the most critical part of their software: the core itself.

Conclusion

Plugins aren't good in every scenario. In particular, they can only be designed once you already have a perfect knowledge of the design space - or at least good enough to know exactly what are the parts you're still missing.

In my case, for example, it took almost two years before I finally felt confident enough about package managers to bootstrap the project. Before that, I spent my time writing various package manager implementations, various tentative APIs, all to grasp the extent of what we would need to cover. It's only after failing a few times that I decided we were ready to go.

So plugins are dangerous. They might put you off the trail, looking for the mysterious cities of gold rather than building your fort. Still, in the case of popular open-source projects, I believe using a modular architecture offers some very strong advantages that go far beyond the idea that people may have in mind when thinking about plugins. More than just a way to open your project to new features, they also provide crucial structure and support, helping those projects to stand the test of time.

DEV Community: Maël Nison

Yarn 3.2 🚢🔮 Libc, Yarn Explain, Next Major, ...

Sponsoring

Libc Field

New Command: yarn explain

UI Improvements

Next Major

Yarn 3.1 🎃👻 Corepack, ESM, pnpm, Optional Packages ...

Sponsoring

Table of content

Improvements

Node.js Corepack Integration

ESM Support

New Install Mode: pnpm

Conditional Packages

Smart Changeset Filters

New Workspace Syntax: workspace:^

Yarn 3.0 🚀🤖 Performances, ESBuild, Better Patches, ...

Governance

OpenCollective

Breaking Changes

Support for the exports field

Performances

New node_modules linkers

Improved Shell

ESBuild support

New plugin APIs

Conclusion

What's to come?

Yarn 💞 GitHub Sponsors

Yarn 2.4 🎄🎁 Log Filters, Audits, Better Warnings, ...

Plugins

Audits

Better Warnings

Log Filters

And also

What about Yarn 3?

Yarn 2.3 🍦✨ Info Command, Detailed Options, Nohoist, ...

Info command

Option Documentation

Nohoist

What's to come?

Yarn 🎊 Hacktoberfest

Maël Nison ・ Sep 28 '20

Yarn 🎊 Hacktoberfest

Yarn Tips, part 1

1. The node_modules linker

2. The packageExtensions settings

3. Yarn's E2E battery

4. Cache integrity

5. File cloning

Yarn 2.2 🚅🌟 Dedupe, Faster, Lighter, ...

Dedupe command

Performances

Size

Telemetry

Other works

Smaller Improvements

Website

"Package manager manager"

What's to come?

Yarn 2.1 🐱‍🏍 Git Workspaces, Focused Installs, Loose mode, Live Playground, ...

Linker improvements

Node-modules linker

Loose mode

Major improvements to the git: protocol

Workspace cloning

Respectful builds

CLI Improvements

Readability

Focused workspaces

Deep accesses from yarn config get/set

Meta improvements

Cache filenames

Playground

Documentation index

VSCode Zip Filesystem

Other improvements

Performances

Ecosystem

New Command: `yarn explain`

New Install Mode: `pnpm`

New Workspace Syntax: `workspace:^`

Support for the `exports` field

New `node_modules` linkers

2. The `packageExtensions` settings

Major improvements to the `git:` protocol

Deep accesses from `yarn config get/set`

New Command: `yarn dlx`

New Command: `yarn workspaces foreach`

New Protocol: `patch:`

New Protocol: `portal:`