We announced at ViteConf that our WebContainers now support pnpm. It was a major achievement in our commitment to support the Vite ecosystem as many projects running on Vite also use pnpm (examples include Vue, Astro, SvelteKit).
In this post, Iâd like to talk about what sets pnpm apart from other package managers - and why we need package managers at all. As a person who best operates through analogy, I will also talk about: đŁ sushi, đȘŠ cemetery, đ birthdays, and đŒ memes.
What are package managers?
I've also written an âExplain like Iâm 5â version of this section!
Letâs start with the terminology.
package, dependency, node module
A âpackageâ (or âdependencyâ) is one or a few files bundled neatly together that can be downloaded from a package registry. Once itâs downloaded and installed, it resides in the node_modules folder in a projectâs directory. In this post, I will use the words a Node module, a package, and a dependency interchangeably as many folks in the tech community do. That being said, the npm docs offer a differing opinion so check it out if youâre interested in trivia and precision. đ
package.json
Each package has a file called package.json which contains information on the package author, its version, its dependencies, and so on. A package can have many dependencies, which in turn can also have many dependencies (called "sub-dependencies"). This sounds like a lot of files, doesnât it! It really is, especially if your project requires bigger packages.
In the olden times, youâd manually download, install, and configure the dependencies of your app. Keep in mind that most packages are frequently updated, which means that youâd have to always make sure that your project's dependencies (and their sub-dependencies!) are up to date - or that in a given project you use a correct version of a given package. This sounds not only stressful but also unnecessarily repetitive! This is where a package manager comes in.
package managers
A package manager is a tool that helps you keep track of all dependencies of your app in a consistent way. It automates the tasks of dependency installation, upgrading, configuration, and removal. It makes sure that all the packages that your app needs are installed, of the correct version, and up-to-date. This frees so much time for coding⊠and procrastination đ€Ș
The first package manager for the Node runtime was introduced only in 2010 and it was npm, (which, contrary to popular belief, doesnât stand for âNode package managerâ and, in fact, is an empty acronym). Twelve years later, npm is now managed by Microsoft and there are two other contenders to the iron throne of the package universe: yarn and pnpm.
What do package managers do?
A package manager grabs the published source files from the online registry and installs them into a directory (a folder) called node_modules. This is the folder where your app will look for the packages when you call the require() method.
You can think about the node_modules folder as a box with helpful tools that you can plug into your app so it works in a desired way. Except that all the tools are made of iron (or osmium!) and the box is massive. What I mean to say is that, sadly, prior to pnpm the node_modules folder would usually be bulky and take a lot of disk space. This resulted in a collection of memes such as:
Imagine that you have numerous projects on your local machine, and each project comes with its own node_modules folder. Most likely, many of these projects use a similar stack. In such a case, you end up with the same packages in numerous projects. Duplication and redundancy! Think about all those inspired side projects that never left the boilerplate stage, all the past attempts to check out all the hot new frameworks - all of them are chilling on your precious disk space.
Who cursed the node_modules folder?
To understand these jokes, we need to take a stroll down memory lane and look at how package management has traditionally been handled.
To the best of its intentions and abilities, npm handled the dependencies by splitting the installation (which is triggered by, for example, npm install ) process into three phases:
-
Resolving, when the package manager is checking all the projectâs dependencies (and their sub-dependencies) listed in the
package.jsonfile, finds a version that satisfies the version specifier (for instance,^1.0.0would install any future minor/patch versions). Afterwards, it creates a file likepackage-lock.jsonfor npm andpnpm-lock.yamlfor pnpm. you can think about it like a projectâs equivalent of a birthday gift wishlist. - Fetch, when the package manager takes the list of resolved dependencies and fetches all the packages from the package registry, which is an equivalent of the feverish shopping in a crowded mall two hours before the party for the gifts all of your friends will give to the birthday person.
-
Linking, when the package manager writes all the dependencies into the projectâs
node_modulesfolder, which, finally, is an equivalent of placing all the gifts in the corner of the party room for when they are needed.
In this scenario, each phase needs to end for the next one to begin. This means that if one dependency has twenty dependencies itself or if one package takes forever to be downloaded, you may have to wait for a long time. If we follow the birthday analogy, imagine that the wishlist included a new version of a popular phone and a box of chocolates. You end up queuing for a few days in front of the phone store to only then buy the chocolates. In such a system, you canât âsaveâ your spot in the queue, go buy the chocolates and join the party, even if that looked like a better use of your time. You see that t*he way npm manages its tasks is just not efficient.*
As we said earlier, it is also the matter of the disk space. You will end up with numerous duplications of the same packages. And even if there are different versions of the same package, it is rarely the case that all of the files have been changed from one version to another. You still end up with some number of copies. We can agree that the way npm manages the disc space is not efficient either.
How is pnpm a solution to all our worries and troubles?
On its docs page you can read that pnpm is a âfast, disk space efficient package manager.â It really is fast - locally, up to three times faster than the alternatives - and space-efficient. But - how?
pnpm is fast
Do you remember the birthday wishlist analogy? As a gofer, npm first collected all the wishes, then bought all the gifts, then placed them in the corner of the party room. Each phase needed to end for another to begin. This is how the npm gofer organizes its way through life.
Well, if pnpm were the gofer, theyâd buy the gift as soon as theyâve read the wish, and designated the spot in the room as soon as they placed the order. They may not even have the physical gift yet to already be able to mark the space in the room where the gift will stand. This happens for each gift separately and independently from others. By design, pnpm doesnât have the blocking stages of installation - the processes run for each of the packages independently.
pnpm is disc space efficient
Letâs talk about food. I love veggie sushi. I order a lot of veggie sushi. I also eat a lot of wasabi and I donât like waste. Even though it would be fair to assume that such portions surely are to be shared, I usually order just for myself. My order always arrives with two pairs of chopsticks even though I say on the phone that I donât need them - I have metal ones at home. I wouldnât just throw unused chopsticks away and now I have a drawer full of them, and you can also see them laying around in the most surprising places in the kitchen. Similarly, I have my own wasabi tube but would I throw away those little sachets? Never. They chill on the shelf in my fridge in an ever-expanding fashion.
My chopsticks and wasabi could work as an analogy for how npm manages disc space. âOh, youâve already installed React two hundred times? Iâm SURE you NEED the 201st copy!â
To my relief, pnpm checks what is already available on your disk and only adds what you additionally need for your project to run. All the dependencies are located in one global location (for instance, ~/.pnpm-store/ - you can check it by running pnpm store path in the terminal) called âa content-addressable storeâ. In your projectâs node_modules folder there is a .pnpm file that contains the âvirtual storeâ with many so-called âhard linksâ. it creates one hard link for each file of each package. I like this explanation that pnpm docs offer:
So, for example, if you have
fooin your project as a dependency and it occupies 1MB of space, then it will look like it occupies 1MB of space in the project'snode_modulesfolder and the same amount of space in the global store. However, that 1MB is the same space on the disk addressed from two different locations. So in totalfoooccupies 1MB, not 2MB.
This makes the node_modules folder more like a portal (like the wardrobe in Narnia) to files located in different nooks of the global store, and not like a bloated storage unit. This diagram illustrates the strategy employed by pnpm:
But, does pnpm âknowâ if there are duplicate files between two versions of the same package? Yes, because, just like git, it identifies the files by a hash id (called also âcontent integrityâ or âchecksumâ) and not by the filename. This means that two same files will have identical hash id and pnpm will determine that thereâs no reason for duplication.
pnpm has got your back
There are so many ways that pnpm looks out for you. One of them is that it is impossible to invite silly bugs by trying to use modules that are not directly specified in the project's package.json but, for instance, are required by your projectâs dependencies. While itâs not the end of the world if everything works well, what happens if your projectâs dependency no longer requires one of the packages and as a result it is no longer available in the the node_modules folder? Well, everything crashes and you donât even know why.
Such bugs may happen in npm and Yarn Classic because of the flat node_modules directory because during installation, they hoist (elevate) all of the packages to the node_modules , regardless of whether they are directly required by your app or not. As a result, your project gets access to distant ârelativesâ of its dependencies.
âŠbut what if my project doesnât support symbolic links?
And even if your project presents an edge case and doesnât work well with symbolic linking, worry not! You can still use pnpm but set it to an npm-like mode. While it will not be space-efficient, it will still be faster.
But⊠what about Yarn?
There are two editions of Yarn:
- âYarn Classicâ, which comprises versions below v2 and is no longer maintained,
- âYarn Berryâ, which is v2 and higher.
Yarn Classic operates similarly to npm with regards to managing the node_modules folder. Yarn Berry, on the other hand, offers three solutions:
- the npm-like mode
- the âPlug'n'Playâ mode
- the pnpm-like mode, which uses hard links to reduce the disc space (not available by default and not thoroughly documented)
The PlugânâPlay seems like an intriguing option but it has been met with the communityâs apprehension as the .zip files are not so easily accessible during the dev workflow. The earlier-mentioned post by TakeShape explains some of the other challenges.
Closing remarks
Considering how space-efficient and fast pnpm is, it is not a surprise that more and more projects make a move towards it and that its popularity is rapidly growing. This year, the weekly download rate for pnpm was seven times as high as last year!
If youâre interested in learning about why projects choose to switch to pnpm, check out these posts by Gestalt and TakeShape.
Further reading
Here are further readings that may give you a better understanding of the current landscape of package managers:
- feature comparison between the three package managers
- an in-depth comparison of the three package managers by LogRocket
- package managers speed benchmarks on the pnpm page and yarn page
Come work with us on package managers!
While youâre here, an announcement: we are looking for a new team member who will work on package managers including pnpm, npm, yarn, and our own Turbo. Interested? Apply today or reach out to me on Twitter!
Top comments (22)
Very interesting, thanks!
How does pnpm manages cleanup though? With npm, you delete the project folder, and node_modules disappear. Is pnpm able to detect some deps in the cache are now dangling and useless?
If not, does it mean cleanup requires deleting the pnpm folder, and re-run pnpm install on all projects? (which no one will do and thus the pnpm folder may grow indefinitely?)
Thank you for your questions, @derlin!
Yes, the pnpm cache grows indefinitely basically but there is usually a lot of overlap of dependencies between projects. Pruning the store every once in a while is a good idea. You can do it via
pnpm store prune, which removes unreferenced packages that are not used by any project.great question, I use pnpm daily and I see it has
pnpm prunecommand, but I never tested it. If it works as same asdocker volume prune, it's exactly what we need.Unfortunately, even the document:
It doesn't seems easy to understand how it actually works.
May Sylwia help us to clarify?
Thank you, @ndaidong! I think we posted at the same time - yes, you're right about
pnpm prune!Thank you for your questions, @derlin!
Yes, the pnpm cache grows indefinitely basically but there is usually a lot of overlap of dependencies between projects. Pruning the store every once in a while is a good idea. You can do it via
pnpm store prune, which removes unreferenced packages that are not used by any project.You're rock! We need
pnpm store prune. Just cleaned :)This really open my eyes about pnpm.
Ive been only using npm, and not looking at others as i felt it unnecessary. But its tempting to test out and try using pnpm because why not right? hahaha
Would definitely try it! Thanks @sylwiavargas!
Ah thank you for sharing this! I'm happy this post brought some clarity and curiosity đ
Insightful and a great alternative đâš
Thank you!
Could you say more about that ?
Would you provide links where this is documented?
I've set my VPN to Belarus and I'm able to run it.
Ok I see, I think he is mentioning the pnpm decision on twitter : https://twitter.com/pnpmjs/status/1498306992577957890?s=46&t=0bwOqnztoi2cUIkmGvGBow
Yes, I see that on my end as well. Given that the author of pnpm is from Ukraine and still in Ukraine, I donât find this surprising.
(EDIT: I see that he was also open about this decision so nothing sneaky there)
However, you were talking about CLI - could you provide links?
Blocking a website based on the location is not a security threat, especially if the website and docs are open sourced and accessible on GitHub.
I hear you and what youâre saying is not overlapping with the experience of my friend in Belarus nor with my experience on VPN. Would you provide links to where this issue is documented?
You also mentioned this as âchildishnessâ and âserious projectâ but this is a common practice in tech - whether itâs good or bad, thatâs a subject for opening a discussion. To give you an example, GitHub blocked Devs from Iran, Syria, and Crimea two years ago and hereâs a whole list of serious business blocking Iran. Hereâs a Wikipedia entry on GitHubâs track record in this field. It is a common practice by the protect authors or whole businesses.
A discussion about political decisions of a project or a business and its merits is one thing. Youâre throwing an accusation without documentation. Iâm not saying that what youâre saying is not true but so far I havenât managed to see evidence of that and youâve avoided providing me with one, even if itâs a link to an issue or a tweet which engaged Zoltan. Moreover, Zoltan was asked if the CLI will be affected:
to which he responded:
I understand that a decision like this is bound to trigger responses and emotions - and itâs fair to express them.
As Sylwia already stated, only the website with the docs is blocked. The CLI works. Also, the standalone install script doesn't work because it is a script from the website. Other install methods work, like
corepack enableandnpm i -g pnpm.However, if companies in Russia and Belarus decide not to use pnpm, my goal is achieved. I don't want my work to help such companies and people. I live in Ukraine, my life is in constant danger because of Russia and Belarus.
Slava Ukraini
So, if you don't need the 3x speed increase for the dependencies installation and don't want to reduce some disc space consumption then you can continue to be happy with npm đ€·ââïž
There is a feature parity between pnpm and npm, but the later wins because it is shipped with Node and it has much broader adoption. I think that all the innovation from the pnpm eventually will come to the npm, it is just a matter of time.
Yes! It's always good continue to be happy with whatever makes you happy đ
I appreciate your optimism about npm roadmap - let's hope it will be that way.