DEV Community: Zoltan Kochan

2021 pnpm recap

Zoltan Kochan — Sat, 01 Jan 2022 21:18:19 +0000

It is the end of the year and it was a good year for pnpm, so let's see how it went.

Usage

Download stats

My goal this year was to beat Bower by the number of downloads. We were able to achieve this goal in November:

pnpm was downloaded about 3 times more in 2021 than in 2020:

These stats don't even measure all the different ways that pnpm may be installed!
They only measure the downloads of the pnpm npm package. This year we also added compiled binary versions of pnpm, which are shipped differently.

Docs visits

We collect some unpersonalized stats from our docs using Google Analytics.
In 2021, sometimes we had more than 2,000 unique visitors a week.

Most of our users are from the United States and China.

GitHub stars

Our main GitHub repository received +5,000 stars this year.

New users

Our biggest new user this year is Bytedance (the company behind TikTok).

Also, many great open-source projects started to use pnpm. Some switched to pnpm because of its great support of monorepos:

Some switched because they like how efficient, fast, and beautiful pnpm is:

Feature highlights

New lockfile format (since v6.0.0)

One of the first and most important changes this year was the new pnpm-lock.yaml format. This was a breaking change, so we had to release v6. But it was a success. The old lockfile was causing Git conflicts frequently. Since the new format was introduced, we did not receive any complaints about Git conflicts.

Managing Node.js versions (since v6.12.0)

We shipped a new command (pnpm env) that allows to manage Node.js versions. So you may use pnpm instead of Node.js version managers like nvm or Volta.

Also, pnpm is shipped as a standalone executable, so you can run it even with no Node.js preinstalled on the system.

Injecting local dependencies (since v6.20.0)

You may "inject" a local dependency. By default, local dependencies are symlinked to node_modules but with this new feature you may instruct pnpm to hard link the files of the package instead.

Improved reporting of peer dependency issues (since v6.24.0)

Peer dependency issues used to be printed as plain text and it was hard to understand them. They are now all grouped and printed in a nice hierarchy structure.

The competition

Yarn

Yarn added a pnpm linker in v3.1. So Yarn can create a similar node-modules directory structure to the one that pnpm creates.

Also, the Yarn team plans to implement a content-addressable storage to be more disk space efficient.

npm

The npm team decided to also adopt the symlinked node-modules directory structure that pnpm uses (related RFC).

Others

Bun written in Zig and Volt written in Rust both claim to be faster than npm/Yarn/pnpm. I did not benchmark these new package managers yet.

Future Plans

Faster, better, best.

pnpm v4.9 comes with command completion!

Zoltan Kochan — Fri, 31 Jan 2020 20:56:55 +0000

Command completion in bash, zsh, fish is awesome! Unfortunately, even though there are thousands of great command-line tools in the npm registry, I cannot recall any (except the npm CLI) that comes with command completion.

A few weeks ago nikoladev suggested to implement autompletion in pnpm. After a brief investigation, I have found a brilliant tool that helps with autocompletion of CLI apps written in Node.js. This tool is called tabtab and is currently not maintained, so I forked it and added autocompletions to pnpm👌.

To set up autocompletion, just update pnpm to v4.9 and run pnpm install-completion.

Let's see what you can do with it.

Type pnpm <tab-tab> and see all the available commands:

Type pnpm remove <tab-tab> and see the list of dependencies currently installed. This also works with pnpm update and pnpm outdated.

Type pnpm --filter <tab-tab> and see the list of projects in the workspace:

Type pnpm run <tab-tab> and see the list of available scripts:

Type any command, type tab-tab and see the list of supported options:

Type an option and see the possible values:

And these are just a few examples! Happy tabbing😃

In a future version of pnpm we will also add descriptions to completions😍

pnpm vs Yarn: monorepo node_modules

Zoltan Kochan — Sun, 04 Nov 2018 20:12:04 +0000

Both pnpm (as of v2.17) and Yarn (as of v1.12) support fast, concurrent installations in monorepos. However, there is a big difference between how they store dependencies in monorepos. Yarn tries to hoist all dependencies from all workspace packages into the root node_modules of the monorepo, which means that packages have access to dependencies of other packages in the workspace. This is described in the Yarn docs:

Be careful when publishing packages in a workspace. If you are preparing your next release and you decided to use a new dependency but forgot to declare it in the package.json file, your tests might still pass locally if another package already downloaded that dependency into the workspace root. However, it will be broken for consumers that pull it from a registry, since the dependency list is now incomplete so they have no way to download the new dependency. Currently, there is no way to throw a warning in this scenario.

Like Yarn, pnpm hoists all packages to the root. However, pnpm stores all dependencies in a hidden folder and only links the direct dependencies of every package into their node_modules.

Let's see how Yarn and pnpm work on a simple example:

there is a small monorepo with two packages: foo and bar
foo has is-negative@1.0.0 as a dependency
bar has is-positive@1.0.0 as a dependency

Yarn in action

To make this repo work with Yarn, there should be a package.json in the root of the repo with the following content:

{
 "private": true,
 "workspaces": ["foo", "bar"]
}

To install all dependencies of all workspace packages, you should run yarn install.

See results here

In this case, Yarn will create a single node_modules in the root of the repo with both is-negative and is-positive. It means that both foo and bar will have access to the dependencies of each other.

pnpm in action

To make pnpm install dependencies in this repo, you'll need to create a pnpm-workspace.yaml in the root of the repo (it can be empty) and a .npmrc file with the following content:

shared-workspace-shrinkwrap = true
link-workspace-packages = true

To install all dependencies of all workspace packages with pnpm, you should run pnpm multi install (or just pnpm m i).

See results here

Like Yarn, pnpm creates a node_modules in the root of the repo. However, if you open that folder, you'll see that all directories and files are hidden there:

node_modules
+ .registry.npmjs.org
+ .modules.yaml
+ .shrinkwrap.yaml

This node_modules stores all the dependencies but they are hidden in .registry.npmjs.org. Node's resolution algorithm will not find them.

Now if you check the directory of bar, you'll see a node_modules there as well, with a single symlink called is-negative. This symlink is pointing at ../../node_modules/.registry.npmjs.org/is-negative/1.0.0/node_modules/is-negative. Likewise, there is a symlink to is-positive in the node_modules directory of foo. So foo is able to resolve only is-positive and bar is only able to resolve is-negative 🎉😊

As you can see, pnpm is strict not only when used with a single package.json but also in a multi-package repository.

It is OK to keep random things in a single monorepo

Zoltan Kochan — Mon, 22 Oct 2018 22:21:34 +0000

For a long time, I was an opponent of monorepos. There are many popular open source contributors that have hundreds of packages on npm and each of those packages have a dedicated GitHub repository. I thought everyone does it this way, so it should be the right way! No?

I started to publish some things to npm as well and after a few years, I have now almost 300 packages in the registry. It took me a long time but I realized, the majority of npm packages don't need dedicated repositories.

Most of the npm packages are almost never updated

Once a package is ready, you will probably never update it again. The only time you will need to update a package is when Renovate or Greenkeeper will open a PR to update dependencies that had major version changes.

So why create a dedicated repository for a package that will have less than 10 useful commits?

Most of the npm packages never get any contributions

Even popular packages get few contributions. Certainly, you will be the only contributor of your non-popular packages.

So why keeping a separate GitHub repository? There will be no other developers that will need admin permissions to a given npm package.

It is OK, use one repo!

You might think: "but those packages are completely unrelated". And that is true. But that is the only downside: keeping random packages in a single repository. Think about all the advantages:

fewer notifications from Greenkeeper/Renovate
less CI setup
less boilerplate

Additional advantages:

you can use services that limit the number of repositories you use.
you can easily migrate all your code to other git servers because there is only one repository to migrate

How to

My recipe is using the recursive commands of pnpm to install all dependencies of your packages and run their tests:

pnpm recursive install
pnpm recursive test --workspace-concurrency 1

But you may also use Rush, Lerna, or other monorepo managing tools.

To see how I moved some of my packages to a single repo, see zkochan/packages.

You can always create a dedicated repository later

If one of your packages will receive a lot of attention, you can always move it to a dedicated repository later.

Photo by Olav Ahrens Røtne on Unsplash

pnpm vs Lerna: filtering in a multi-package repository

Zoltan Kochan — Sun, 02 Sep 2018 21:53:00 +0000

Everyone heard about Lerna, which is "A tool for managing JavaScript projects with multiple packages." A lot fewer devs heard about pnpm, which is a fast, disk space efficient package manager for JavaScript. Both Lerna and pnpm try to improve tooling for multi-package repositories (MPR). For Lerna it was the reason for creation. For pnpm, MPR support is a nice bonus feature that is implemented via a set of commands called recursive. Of course, there are many differences between how Lerna manages a multi-package repo vs pnpm. In this article, I want to compare one seemingly simple aspect: filtering.

Filtering in an MPR is important because, during development, changes are mainly made inside one or two packages. It wouldn't make sense to run commands on the whole repository if only a few packages were modified.

Filtering in Lerna

Filtering in Lerna (as of v3.2.1) is achieved by the following flags:

scope - Include only packages with names matching the given glob.
include-filtered-dependents - Include all transitive dependents when running a command regardless of --scope, --ignore, or --since.
include-filtered-dependencies - Include all transitive dependencies when running a command regardless of --scope, --ignore, or --since.
ignore - Exclude packages with names matching the given glob.
private - Include private packages. Pass --no-private to exclude private packages.
since - Only include packages that have been updated since the specified [ref]. If no ref is passed, it defaults to the most recent tag.

These flags make filtering in Lerna quiet powerful. However, they are pretty hard to type. Let's say you downloaded a repository and want to work only on login-page component. You'd want to run installation for login-page and any of its dependencies:

lerna bootstrap --scope login-page --include-filtered-dependencies

Or maybe you changed a component called site-header and would like to run tests on all the dependent packages:

lerna run test --scope site-header --include-filtered-dependents

These flags are not only hard to type but also hard to remember and easy to mix up.

Filtering in pnpm

Unlike Lerna, pnpm uses a special package selector syntax to restrict its commands. So instead of memorizing a set of long flag names, you should only remember a very easy-to-remember selector syntax.

As of version 2.15.0, these are the selectors that pnpm supports:

<pattern> - Restricts the scope to package names matching the given pattern. E.g.: foo, @bar/*
<pattern>... - Includes all direct and indirect dependencies of the matched packages. E.g.: foo...
...<pattern> - Includes all direct and indirect dependents of the matched packages. E.g.: ...foo, ...@bar/*
./<directory> - Includes all packages that are inside a given subdirectory. E.g.: ./components
. - Includes all packages that are under the current working directory.

These filters may be specified either via the --filter flag or after a --at the end of the command.

Note: as of v2.15.0, filters after -- are not supported by run, exec, test

So if you want to bootstrap login-page and all of its dependencies, this is the way you do it with pnpm:

pnpm recursive install -- login-page...

And if you want to run tests of site-header and all of its dependents, use the ...<pattern> selector:

pnpm recursive test --filter ...site-header

Of course, you can combine as many selectors as you'd like:

pnpm recursive update -- ...site-header login-page... ./utils @style/*

The command above updates dependencies in:

site-header
dependents of site-header
login-page
dependencies of login-page
all packages that are located in the utils directory
all packages from the @style scope

pnpm might not have yet all the features that Lerna provides but for many users it might be enough already.

If you haven't heard about pnpm yet, I recommend reading also Flat node_modules is not the only way which explains the unique node_modules structure created by pnpm.

Cheatsheet

Lerna v3.2	pnpm v2.15
--scope my-component	-- my-component
--scope toolbar-*	-- toolbar-*
--scope my-component --include-filtered-dependencies	-- my-component...
--scope my-component --include-filtered-dependents	-- ...my-component

originally posted in pnpm blog

The not fancy CLI output of pnpm

Zoltan Kochan — Sun, 26 Aug 2018 21:36:57 +0000

pnpm is a JavaScript package manager that differs from npm and Yarn in many ways. If you haven't heard about it yet, I recommend checking it out. In this article, I would love to write about the design system that we use to report during installation.

When I first started contributing to pnpm (around v0.15), this is how an installation was reported:

It wasn't really useful but some users of pnpm liked it. They thought it was beautiful. But then as we started adding more features, we realized that it is very important to print the right amount of information in a nice readable format.

So let's see how pnpm has evolved and how it reports in different scenarios as of v2.13.6.

Reporting installation in a single project

When you first install pnpm and you run pnpm install in a project, you'll see an output like this:

Unlike the old output, this one is very static and minimalistic but it contains a lot more useful information.

We see that:

one of the installed packages is deprecated
117 new packages were added to node_modules
Installation slowed down a bit because the huge typescript tarball was being downloaded
0 packages were available in the store, so all 117 packages were downloaded (pnpm saves one version of a package only ever once on a disk, so when a package is available in the store, it is just hard linked to the node_modules)
express@4.16.0 was added as a production dependency
a newer version of express is available in the registry
babel-preset-es2015@6.24.1 and typescript@3.0.1 were added as dev dependencies

Now lets update express to the latest version and see what we get:

5 packages were removed from node_modules
5 packages were added to node_modules
all 5 packages were downloaded from the registry
the newest express was added to the project

Reporting installation in a multi-package repository

pnpm has a set of commands for working with multi-package repositories (MPR). When installing dependencies in an MPR, the amount of information that is being processed is so big that printing all of it would just make an unreadable mess. In order to provide some basic information anyway, we came out with the concept of zoomed-out reporting. A zoomed-out reporting contains just the most important pieces of information.

Every package in the MPR is printed with the number of added/removed packages (inspired by Git):

Zoomed-out reporting also prints warnings (only warnings, no info messages):

When we came out with the concept of zoomed-out reporting for the recursive commands, we realized that there are other scenarios in which they are useful. For instance, when packages are linked in, it should be a mixture of zoomed-out and zoomed-in reporting. Packages that are linked in should be reported briefly and the package in the current working directory should be in focus:

Implementation details

Although the output seems minimalistic and simple, it is produced by a very complex system. pnpm consists of many components and many operations may happen in random order (this is one of the reasons pnpm is so fast). That is why reporting is performed by a specialized part of pnpm called "reporter" (code).

The reporter is a package that listens for logs, filters them, combines and forms an output from them. pnpm uses bole to pass the logs from the loggers to the reporter. This modularization is great because we can mock the logs and cover reporting with tests!

For printing the output to the console, we use ansi-diff. ansi-diff is great because it accepts "frames" of output and it updates only those parts of the console that are changed (and it is fast). Before we switched to ansi-diff, we used another popular library for updating the console output but it was doing the update with noticeable flickering.

P.S.

It is very hard to implement good CLI reporting. But good reporting allows developers to focus on the important things and possibly notice issues earlier.

Of course, pnpm's reporting can improved a lot and we have many open issues in that area. Give pnpm a try and don't hesitate to let us know if there are things that can be improved further.

originally posted in the pnpm blog

Flat node_modules is not the only way

Zoltan Kochan — Sun, 24 Jun 2018 18:24:17 +0000

This article covers an old version of pnpm. For an updated version of the article go here.

New users of pnpm frequently ask me about the weird structure of node_modules that pnpm creates. Why is it not flat? Where are all the sub-dependencies?

I am going to assume that readers of the article are already familiar with flat node_modules created by npm and Yarn. If you don't understand why npm 3 had to start using flat node_modules in v3, you can find some prehistory in Why should we use pnpm?.

So why is pnpm's node_modules unusual? Let's create two directories and run npm install express in one of them and pnpm install express in the other one. Here's the top of what you get in the first directory's node_modules:

You can see the whole directory here.

And this is what you get in the node_modules created by pnpm:

You can check it here.

So where are all the dependencies? There is only one folder in the node_modules called .registry.npmjs.org and a symlink called express. Well, we installed only express, so that is the only package that your application has to have access to

Read more about why pnpm's strictness is a good thing here

Let's see what is inside express:

express has no node_modules? Where are all the dependencies of express?

The trick is that express is just a symlink. When Node.js resolves dependencies, it uses their real locations, so it does not preserve symlinks. But where is the real location of express, you might ask?

Here: node_modules/.registry.npmjs.org/express/4.16.3/node_modules/express.

OK, so now we know the purpose of the .registry.npmjs.org/ folder. .registry.npmjs.org/ stores all the packages in a flat folder structure, so every package can be found in a folder named by this pattern:

.registry.npmjs.org/<name>/<version>/node_modules/<name>

This flat structure avoids the long path issues that were caused by the nested node_modules created by npm v2 but keeps packages isolated unlike the flat node_modules created by npm v3,4,5,6.

Now let's look into the real location of express:

Is it a scam? It still lacks node_modules! The second trick of pnpm's node_modules structure is that the dependencies of packages are on the same directory level on which the real location of the dependent package. So dependencies of express are not in /express/4.16.4/node_modules/express/node_modules/ but in /express/4.16.4/node_modules/:

All the dependencies of express are symlinks to appropriate directories in node_modules/.registry.npmjs.org/. Placing dependencies of express one level up allows avoiding circular symlinks.

So as you can see, even though pnpm's node_modules structure seems unusual at first

it is completely Node.js compatible
packages are nicely grouped with their dependencies

The structure is a little bit more complex for packages with peer dependencies but the idea is the same: using symlinks to create a nesting with a flat directory structure.

If you'd like to try out pnpm, you can easily install it with npm: npm i -g pnpm. Then just run it instead of npm when you need to install something: pnpm install foo bar.

Why package managers need hook systems

Zoltan Kochan — Sat, 02 Sep 2017 22:20:07 +0000

Installation hooks were introduced to pnpm in version 1.12. In this article, I want to write about why I think package managers (PMs) need hooks.

Why would we want to hook into `node_modules`?

When installing dependencies of a Node-project, the node_modules structure is 100% controlled by the package.json files of the dependencies (a.k.a. manifests). So if your project depends on foo@1.0.0 which depends on bar@2.0.0 then you’ll be going to have two dependencies installed in your node_modules. While you can change your project’s dependency set, you don’t have control over the manifests of your dependencies.

A typical dependency tree is huge and you have no ownership over the majority of your dependencies. Just analyze any of your dependencies at npm.anvaka.com. For instance, this is the dependency graph of browserify:

What to do if there is a bug in one of the packages inside your node_modules? If the issue is in a root dependency, then you have 3 options:

find an alternative, more reliable package
submit a pull request (PR) that fixes the issue
create your own package and use it instead of the buggy one

The 2nd option seems to be the correct one and I encourage everyone to contribute as frequently as possible. However, submitting a PR doesn’t mean that your changes will be merged/published. Even if they will be merged and published, it won’t happen immediately, you might have to wait for years. Here are some frequent issues:

the project is unmaintained/poorly maintained
you did breaking changes and the author doesn’t like bumping major versions. The author will wait for several breaking changes to come in, before publishing a major version
the author thinks the bug is not a bug

If the problematic package is a root dependency, then you can easily switch to another package or to your fork. The problem is match harder to solve if the package is a sub-dependency. In that case, your options are:

submit a PR that fixes the issue
submit many PRs to packages that depend on the buggy package

With hooks though, you can have a third option: create a fork and install it instead of the problematic package. So if you have foo@1.0.0 that depends on bar@2.0.0 you can have a hook that will override the dependency.

Let’s say there is a bug in bar@2.0.0 and you submitted a PR with a fix. However, the maintainer of bar is on vacation. You can use a hook to make pnpm install bar from your PR branch instead of bar@2.0.0 from the npm registry.

Why hooks are important for pnpm’s survival

Hooks are nice to have in any PM but for pnpm they are especially important. As you may know already, pnpm creates a strict symlinked node_modules structure. You can read more about it in: pnpm’s strictness helps to avoid silly bugs

Although the node_modules structure created by pnpm is Node.js-compatible, many packages have bugs that show up only when installed via pnpm. As a consequence, pnpm has issues with some popular frameworks and tools.

Of course, we try to solve these issues via PRs (I want to give a shout out to aecz who managed to fix many such issues in Angular). But besides the usual issues, some of the maintainers are hostile and refuse to accept PRs just because they don’t like pnpm or believe that the flat node_modules created by npm/Yarn is a feature (it is not).

In the end, we have two options to fix the ecosystem for the strict pnpm:

make pnpm popular enough. In that case, authors of frameworks/toolings will be testing their product with pnpm as well as npm and Yarn.
make a hook system to temporarily substitute the buggy packages that don’t work with pnpm.

IMHO, the 1st scenario is pretty much impossible. pnpm can’t become popular without being a drop-in replacement for npm.

Real-life example

There is a popular package called resolve for resolving dependencies from node_modules (1,3K dependents, 765K downloads a day). Unfortunately for pnpm, resolve preserves symlinks when resolving modules. This is an issue on resolve’s end as Node.js does not preserve symlinks during resolution. I did a PR to fix this issue and now resolve from version 1.4 has an option to not preserve symlinks.

This does not solve the problem for pnpm though. We can’t submit PRs to the 1.3K dependent packages to update resolve and pass preserverSymlink: false to it. The lead maintainer of resolve agreed to switch the option’s default value in the next major version. So I hoped Greenkeeper would create the PRs for us and most of the packages would update resolve to version 2.

I created another PR with the breaking change but resolve’s maintainer wants to wait for more breaking changes before changing resolve to version 2.

I realized that we cannot change the world but we can change pnpm, so I released the readPackage hook. My changes to resolve are available via my fork on GitHub, so all we have to do is to tell pnpm to install resolve from the fork. This can be done by declaring the hook in a file called pnpmfile.js:

During installation, pnpm will pass every package.json to this hook first and use the version of the package.json returned by the hook. So it won’t matter on which version of resolve the package depends, my fork will be installed instead and the project will work fine with pnpm.

Summary

I did not describe all the use cases when the readPackage can be useful. It is a really powerful tool and I think we’ll learn how to use it smartly.

Also, I want to say thanks to Andrei Neculau, who convinced me that this hook-system was a good idea.

To make it a little bit interactive, check how many unresolved PRs you have on GitHub and post the number in the comments section. You can use this link to see all your open PRs.

Do you want to give pnpm a try?

Just install pnpm via npm: npm install -g pnpm. And use it instead of npm whenever you want to install something: pnpm i foo.

Also, you can read more info at the pnpm GitHub repo or pnpm.js.org. You can follow pnpm on Twitter or ask for help at the pnpm Gitter Chat Room.

This post was originally published on kochan.io