DEV Community

Cover image for Yarn 3.0 🚀🤖 Performances, ESBuild, Better Patches, ...
Maël Nison
Maël Nison

Posted on

Yarn 3.0 🚀🤖 Performances, ESBuild, Better Patches, ...

Hello! Long time no see! Back in December, we decided to start working on our next major release, the 3.0. It took a bit of time to do everything we intended to do, but here we are! So let's talk a bit about what it changes, and what it brings. Note that these are only the highlights, the full changelog is much more comprehensive.

And if you just love to read our log posts, here are the past entries 👇

Governance

Back when the project was started in 2017, we didn't took the time to establish a formal governance document. This is now addressed, and our team composition can be found here. It doesn't change anything in practice (this is how we worked for more than two years now), but we hope it may give you a better understanding as to how we work and reach consensus.

OpenCollective

From 2017 to 2019 Yarn was mostly maintained by Facebook engineers. While it worked relatively well, the Yarn 1 -> 2 release also proved to be the right time to expand our active team to other horizons, and nowadays no two of our active contributors work at the same company - and none at Facebook.

Consequently, we've decided to setup an OpenCollective (or GitHub Sponsors) to give our supporters a way to both express their thanks to our team, and give us resources we can then inject back into the project.

Breaking Changes

While the migration from Yarn 1 to Yarn 2 brought some discomfort, the migration to Yarn 3 should prove easier - regardless of the version you come from. The user-facing breaking changes we made this season are mostly little details that may only affect you in very specific cases:

  • Node 10 isn't supported anymore
  • Plug'n'Play hooks are now called .pnp.cjs (vs .pnp.js)
  • Virtual folders are now called __virtual__ (vs $$virtual)
  • The editor SDKs have been moved to @yarnpkg/sdks
  • Etc; full list here

Even for Yarn 1 users, migrating from 1 to 3 should be easier: we made it so that Yarn will detect when this situation arises to then automatically enable the node-modules linker. That alone should address most of the problems you may have been hitting when attempting the upgrade - and for everything else, make sure to take a look at our Migration Guide which got significantly improved over the past year.

Support for the exports field

When using Yarn 3 w/ PnP, the exports field will be properly resolved regardless of your Node version. If you're not familiar with this field, you can see it as a way to:

  • Replace the main field
  • Soft-prevent accessing arbitrary files in the package
  • Conditionally remap files depending on the context (bundlers, ...)

Performances

Various tweaks have been made to address some of the largest resource consumptions in Yarn. Installs have been improved (turning us faster than pnpm in some scenarios, which is quite a feat!), but not only: script execution tends to have a natural overhead, but bugs in 2.4 and prior caused this overhead to grow relative to the size of the project itself. This is no longer the case, and the overhead should now be constant.

New node_modules linkers

As you may know, Yarn is built around a few interfaces. One of them is called a "linker", and tells Yarn how to install packages on disk. It's how we can support both PnP and node_modules installs without changing much code.

One advantage of this architecture is how it allows us to efficiently iterate on alternative install strategies. For this release, larixer implemented a new experimental nmMode setting that can be used to instruct the linker to use a specific copy scheme:

  • hardlinks-local will use hardlinks when the same package is found multiple times within the same project (but only if they have exactly the same version at the moment).

  • hardlinks-global will use hardlinks on identical files (even across different versions!), but will also make them point to a global content-addressable directory. This is similar to what pnpm does. Note that if the cache is corrupted (for example because you manually edited it), Yarn will automatically repair it on subsequent installs.

I myself have been playing with a pnpm-style linker. It hasn't shipped yet since I'm cautious about adding complexity that could end up unmaintained, but given how small it is there's a decent chance we could add it in a later release as an experimental install mode.

Improved Shell

As you may know, given that system shells are rarely portable across Windows and Posix, Yarn no longer uses them to run your scripts entries. Instead, we use our very own shell interpreter.

We're happy to report that this shell just got smarter, and now provides two additional syntaxes that you can reliably use on both Windows and Posix:

build-js & build-css &    # Background jobs
ls 2>/dev/null            # File descriptor redirections
Enter fullscreen mode Exit fullscreen mode

Additionally, background jobs have their output color-coded, so you can clearly identify their output, even interlaced.

ESBuild support

We now use ESBuild to generate the Yarn bundles and as such worked to ensure good compatibility with Plug'n'Play installs. The result is the new @yarnpkg/esbuild-plugin-pnp package which lets you transparently build your code using the default Yarn installation mode. It's still relatively young, so feel free to drop us an issue if you notice something strange!

While it won't change much for most end-users, the move to ESBuild also provided decent build speed improvements (around 6x faster), making it less frustrating to build Yarn from sources

New plugin APIs

Yarn supports writing plugins that can inject themselves into various places and leverage some of the builtin modules provided by the core. While we didn't get the chance to make all the improvements we hoped, we've still been able to upgrade the command line framework to Clipanion 3, which lets you write intuitive type-checked commands with a minimal syntactic overhead.

Conclusion

According to DEV.to, those change notes take about 5 minutes to read - by contrast, the 2.0 release post was a whopping 15 minutes! Of course, this time around we didn't need to fully rewrite Yarn, hence a lower amount of "critical information" we need you to be aware of 😉 We expect that to be the norm from now on: majors won't have a lot of super impactful changes, mostly just some architecture cleaning and modernization, as new features will tend to land in minors.

As for our team, we're very happy of the work we've been doing! Working on the codebase still feels like a treat, and features are often constrained to a few identifiable files - proving that our initial redesign bet was right. Our stats suggest that the result are visible to our users as well, and while I remain cautious about popularity metrics it's certainly nice to see.

Finally, remember that Yarn now has an OpenCollective / GitHub Sponsors! If your company benefits from our work, or would like to see particular fixes land, sponsoring the project is a good way to engage with us 🙂

What's to come?

A few features initially slated for 3.0 have been pushed back to the next minor so that we have more time to properly incubate them. Some of the things we have in motion:

  • Corepack integration
  • ESM support under PnP mode
  • Builtin CLI completion
  • Changelog generation
  • Improved performances
  • pnpm-style linker
  • And more...!

Of course that's only on the top of my head, so it's possible our objectives shift during the next weeks depending on our own priorities - and of course depending on whether you help us or not 😛

Discussion (18)

Collapse
iskin profile image
Andrey Sitnik

yarn up -R! It was my last blocker from migration from Yarn 1.

Collapse
blikblum profile image
Luiz Américo

Thanks for the update. The new shell concurrent feature saved my day.

How do i upgrade from berry v2 to berry v3.

I used yarn set version latest but it used the development branch something i don t want

Collapse
hydrosquall profile image
Cameron Yick

I just tested this out upgrading from v2, dev.to/hydrosquall/comment/1gop5, hope this helps!

Collapse
liuliangsir profile image
流浪大法师

how to upgrade global yarn from berry v2 to berry v3 when using npm

Collapse
corysimmons profile image
Cory Simmons

Been using Yarn 2 for several months now. PnP is fine except when 3rd party projects have stuff like references to node_modules hard-coded in their distributable. I don't see a workaround to this, and a HUGE chunk of 3rd party libraries do this. I'm going back to Yarn 1 or even NPM (if work on Yarn 1 halts).

Yarn 1 was cool, and moved NPM to get their act together a bit, but I hate when maintainers just assume the rest of the ecosystem will migrate with them. This is why Python 3 took a decade to be adopted, and that adoption only happened because people were absolutely forced to use Python whereas switching back to NPM is a breeze.

Almost every single day, for months, I've ran into problems with PnP...

Collapse
arcanis profile image
Maël Nison Author

You can enable the node_modules just fine if that's what you prefer.

From my experience, very few 3rd-party libraries actually hardcode the n_m folder anymore, and I found them relatively easy to ignore (RN being the main contender). If large projects like Next or Gatsby manage to make things work without trouble, there's really no reason in 2021 for projects to still misslist their dependencies.

Collapse
corysimmons profile image
Cory Simmons • Edited on

You can enable the node_modules just fine if that's what you prefer.

You're referring to nodeLinker? Doesn't that just disable PnP project-wide? In which case, why use Yarn 2+ at all?

I agree it'd be nice if every maintainer fixed their projects to not reference n_m, but that's just not the reality of the Node ecosystem... github.com/search?q=%22node_module...

Yesterday my use case was trying to use Playwright (30k Stars, hundreds of thousands of downloads every week) with Yarn PnP. Nope.

The maintainer literally said "Wait what version of Yarn?" I said "2.4.2", he said "Ah yarn 2 my old enemy" github.com/microsoft/playwright/se...

I dunno man. I don't want to dog the project and I really appreciated what Yarn 1 did, and the goals of Yarn 2, but someone needs to say "Look, I've used this thing for several months on a decent sized project and it's painful. Every few days I run into something. I wish I would've just used Yarn 1" so others know what they're getting themselves into.

I'm not the only one with this opinion news.ycombinator.com/item?id=28122318

Until there is a really straightforward dev-friendly way to say "Okay, this package is fine if it's a zip and this package is probably never going to work with Yarn 2 and would just be easier to plop into node_modules" people are just going to keep jumping ship back to npm.

What dev wants to have to study a ton of docs and do a bunch of workarounds to be able to use a package manager that used to be as simple as yarn add foo?

It's just hard, whereas Yarn 1 wasn't, and npm isn't, and npm has workspaces now. 🤷‍♂️

Collapse
malloc007 profile image
Ryota Murakami

I know ESM as well, but what does it mean ESM support as a package manager?

Collapse
borekb profile image
Borek Bernard

Congrats, the release sounds really awesome! ❤️

Collapse
maracujajuice profile image
Maurice

Uhm, probably a dumb question but how do we update to yarn 3?

Collapse
hydrosquall profile image
Cameron Yick

I used these two commands in sequence:

# Now upgrades you to version 3.0.0
yarn set version berry
# Upgrades to latest version of 3
yarn set version 3.x
Enter fullscreen mode Exit fullscreen mode
Collapse
boscheliblack profile image
Bosch-Eli-Black

Thanks! 🙂 For me, just running yarn set version berry was good enough.

If I also ran yarn set version 3.x, it upgraded me to a release candidate version.

Collapse
uptonking profile image
uptonking

yarn is good.
BUT I prefer npm 7 workspaces.
Upgrading yarn is a burden now.
npm is bundled with node, so easy.

Collapse
milo123459 profile image
Milo

Not sure what's a burdern about yarn up -R but I respect your opinion :)

Collapse
xenoterracide profile image
Caleb Cushing

can you still use && or is there a new yarn scripts block? just wondering given the prolifity of && and I guess ||. not expecting full shell eval ;)

Collapse
boscheliblack profile image
Bosch-Eli-Black

Yay! Happy to hear that there are fewer breaking changes this time around, and nice work! 🙂

Collapse
dmail profile image
Damien Maillard

The day yarn support ESM I'll do the switch! I hope it will happen soon 🤞

Collapse
larixer profile image
Victor Vlasenko

Yarn supports ESM with node_modules install scheme. The PnP install scheme will support ESM when Node API let this happen