Background
As a front-end veteran, I have to point out that the increasing complexity of front-end projects nowadays makes the dependenc...
For further actions, you may consider blocking this person and/or reporting abuse
Of course, with optimizations on the server side it is possible to do a lot of interesting stuff. I think these only scratch the surface of what is possible.
I agree that in the past there were many issues with the symlinks approach but by now there are almost none. So I don't agree with the claims in the article regarding the disadvantages of symlinks. Even npm is working on an "isolated mode" that will create a symlinked node_modules.
Regarding FUSE. I am also experimenting with FUSE. I created this package that mounts a node_modules directory. I think FUSE is better than Yarn PnP but it might be not the silver bullet. On Linux it works really well but I am not sure about other systems. Especially about Windows.
I am glad to see that the author is actually trying out new ideas instead of just rewriting it to Rust/Zig/Go etc.
We're use the nydus as the fuse implemention. nydus is based linux, works well with our build/ci system. Now I'm working on port nydus to macos with macfuse. (We can see the progress with fuse-backend-rs).I'm sure our plan can run macos.
By the way, wsl is good choice on the windows, fuse can be used with wsl2.
cnpm isn’t trying to be the replacement to the existing package managers. We have always been an enterprise-ready solution for building an on-premise private registry. We don’t recommend developers to use cnpm cli without that specific need. pnpm and yarn are good enough, and I really like pnpm, thanks for your work.
Last week, we were told pnpm had experimented utilizing FUSE. We believed the experiment was still in an early stage as it didn’t support tree command; JS binding could be improved by other solutions, such as nybus native implementation.
npmfs is a package manager agnostic tool by design. We hope it benefits not only tnpm but also all the package managers that the community loves.
If the community recognizes the solution we propose, we are happy to welcome other widely accepted package managers to work together to improve and use npmfs module alone. Please stay tuned for npmfs to be open source!
merge it to pnpm :-)
@atian is npmfs open source or not yet?
@zkochan I believe github.com/cnpm/rapid is the repository, licensed under
MIT.I've already read it in Chinese via google translate :-)
It's a quite interesting experiment. I thought also about the idea of pre-computing dependency trees (or at least part of them) on the server, but the problem with that is that it relies on cooperation from a third-party we know we won't be able to influence. Similarly, a part of Yarn's overhead is caused by the tgz->zip conversion we run¹. It'd be much faster if npm could serve such archives natively, but we can't bet on that.
¹ We need that because, quite similar to what you do with Fuse, Yarn PnP loads files directly from zip archives, without decompression. However tgz doesn't support random file access, whereas zip does. Hence the conversion.
We are inspired by stargz introduced by crfs. It append the tar toc to the tail to make tar random access.
In our practice, we download the tgz file with stream, uncompress to tar, and read the tar entries to generate toc file.
Yes, we are an enterprise scenario, so it is relatively manageable.
Regarding the package format, we had considered converting the format to stargz + lz4 when syncing packages to cnpm registry at server-side, but we have found that the ROI is not high in our current practice.
To expand a little on the subject of collaboration: the npm cli team at GitHub hosts a weekly meeting that everyone is welcomed to join. It's one of the most effective ways to get ideas brought up to the core npm team and we had in multiple occasions in the past registry-side ideas that came from these meetings being bubbled up to the internal registry team.
For reference, here is the Youtube channel with past meetings: youtube.com/c/npminc and the RFC repo in which meetings get announced and it's possible to propose new ideas that can be added to the agenda: github.com/npm/rfcs
That said, I understand the idea of a package manager spec may sound tempting but I believe that each one of them focus on different problem spaces, ecosystems and user bases. At the end of the day that is great for the larger JavaScript community that benefits from all these different innovations and I'm not sure to what extent adding conformance tests and specs will benefit end users.
This was a fun article to read, thanks TZ for documenting and publishing this!
@atian25 Regarding the question in the article why Yarn is slower, its because Yarn with node_modules linker needs extra config to match Yarn 1.x or npm behaviour. For the best performance the following config settings should be added:
I tried to reproduce the large number of system calls in yarn@3.1.1 with a vue vite-ts project, but the benchmarks show yarn has least number of syscalls:
Details in GitHub discussions: github.com/yarnpkg/berry/discussio...
@atian25 Can you share the steps you used for counting number of system calls using
strace?Update: The benchmark numbers were not accurate as the default modes for package managers are different. Also
straceneeds-foption to trace system calls from child processes.I added new numbers in github.com/yarnpkg/berry/discussio...
As per discussions in discord thread npm is not included in this benchmarks as:
yarn can be compared to npm by disabling central store.
yarn can be compared to pnpm by adding central store support to yarn's pnpm linker.
In my current experience, pnpm should be the fastest (much faster than yarn3, npm hasn't been used in a long time, but generally seems to be considered the slowest?), if tnpm has a part to see, I think To try the real performance for yourself.
pardon? from the table you can see that pnpm is faster than both npm and yarn. For Example, Test1, pnpm(14s), npm(48s), yarn(61s).
Unfortunately, As mentioned in the post, tnpm is still in the internal testing phase. There's currently no way for the community to experience it.
What we use is the official pnpm benchmark tests. As for why we are relatively faster, I believe this has been explained in detail in the post.
npmfs is a package manager agnostic tool by design. We hope it benefits not only cnpm/tnpm but also all the package managers that the community loves. Please stay tuned for npmfs to be open source. Nevertheless, the idea behind it are also worth sharing and discussing with the community.
This sentence I refer to the official npm
if you use pnpm, tell me, did you find it faster than yarn and npm?
Because to me it feels like everytime when I have to install new package it takes so long.
It was relativley better with npm. and even better with yarn.
This post is pure gold and among recently created posts, diamond. I’m saving it for tomorrow to read in the details.
Great innovation!
That's one small step for antgroup, one giant leap for npm community!
for me it seems slow even for 2nd time and also when I try to add new modules