DEV Community

Cover image for In-depth of tnpm rapid mode - how we managed to be 10 second faster than pnpm

In-depth of tnpm rapid mode - how we managed to be 10 second faster than pnpm

TZ | 天猪 on January 25, 2022

Background As a front-end veteran, I have to point out that the increasing complexity of front-end projects nowadays makes the dependenc...
Collapse
 
zkochan profile image
Zoltan Kochan

Of course, with optimizations on the server side it is possible to do a lot of interesting stuff. I think these only scratch the surface of what is possible.

I agree that in the past there were many issues with the symlinks approach but by now there are almost none. So I don't agree with the claims in the article regarding the disadvantages of symlinks. Even npm is working on an "isolated mode" that will create a symlinked node_modules.

Regarding FUSE. I am also experimenting with FUSE. I created this package that mounts a node_modules directory. I think FUSE is better than Yarn PnP but it might be not the silver bullet. On Linux it works really well but I am not sure about other systems. Especially about Windows.

I am glad to see that the author is actually trying out new ideas instead of just rewriting it to Rust/Zig/Go etc.

Collapse
 
killagu profile image
killa

We're use the nydus as the fuse implemention. nydus is based linux, works well with our build/ci system. Now I'm working on port nydus to macos with macfuse. (We can see the progress with fuse-backend-rs).I'm sure our plan can run macos.

By the way, wsl is good choice on the windows, fuse can be used with wsl2.

Collapse
 
atian25 profile image
TZ | 天猪 • Edited

cnpm isn’t trying to be the replacement to the existing package managers. We have always been an enterprise-ready solution for building an on-premise private registry. We don’t recommend developers to use cnpm cli without that specific need. pnpm and yarn are good enough, and I really like pnpm, thanks for your work.

Last week, we were told pnpm had experimented utilizing FUSE. We believed the experiment was still in an early stage as it didn’t support tree command; JS binding could be improved by other solutions, such as nybus native implementation.

npmfs is a package manager agnostic tool by design. We hope it benefits not only tnpm but also all the package managers that the community loves.

If the community recognizes the solution we propose, we are happy to welcome other widely accepted package managers to work together to improve and use npmfs module alone. Please stay tuned for npmfs to be open source!

Thread Thread
 
zkochan profile image
Zoltan Kochan

merge it to pnpm :-)

Thread Thread
 
zkochan profile image
Zoltan Kochan

@atian is npmfs open source or not yet?

Thread Thread
 
bb010g profile image
bb010g

@zkochan I believe github.com/cnpm/rapid is the repository, licensed under MIT.

 
zkochan profile image
Zoltan Kochan

I've already read it in Chinese via google translate :-)

Collapse
 
arcanis profile image
Maël Nison

It's a quite interesting experiment. I thought also about the idea of pre-computing dependency trees (or at least part of them) on the server, but the problem with that is that it relies on cooperation from a third-party we know we won't be able to influence. Similarly, a part of Yarn's overhead is caused by the tgz->zip conversion we run¹. It'd be much faster if npm could serve such archives natively, but we can't bet on that.

¹ We need that because, quite similar to what you do with Fuse, Yarn PnP loads files directly from zip archives, without decompression. However tgz doesn't support random file access, whereas zip does. Hence the conversion.

Collapse
 
killagu profile image
killa

We are inspired by stargz introduced by crfs. It append the tar toc to the tail to make tar random access.

In our practice, we download the tgz file with stream, uncompress to tar, and read the tar entries to generate toc file.

  • Store the package tar file, gzip is too expensive.
  • Toc file store independently for the random access.
Collapse
 
atian25 profile image
TZ | 天猪 • Edited

Yes, we are an enterprise scenario, so it is relatively manageable.

Regarding the package format, we had considered converting the format to stargz + lz4 when syncing packages to cnpm registry at server-side, but we have found that the ROI is not high in our current practice.

Collapse
 
ruyadorno profile image
Ruy Adorno

To expand a little on the subject of collaboration: the npm cli team at GitHub hosts a weekly meeting that everyone is welcomed to join. It's one of the most effective ways to get ideas brought up to the core npm team and we had in multiple occasions in the past registry-side ideas that came from these meetings being bubbled up to the internal registry team.

For reference, here is the Youtube channel with past meetings: youtube.com/c/npminc and the RFC repo in which meetings get announced and it's possible to propose new ideas that can be added to the agenda: github.com/npm/rfcs

That said, I understand the idea of a package manager spec may sound tempting but I believe that each one of them focus on different problem spaces, ecosystems and user bases. At the end of the day that is great for the larger JavaScript community that benefits from all these different innovations and I'm not sure to what extent adding conformance tests and specs will benefit end users.

This was a fun article to read, thanks TZ for documenting and publishing this!

Collapse
 
larixer profile image
Victor Vlasenko

@atian25 Regarding the question in the article why Yarn is slower, its because Yarn with node_modules linker needs extra config to match Yarn 1.x or npm behaviour. For the best performance the following config settings should be added:

enableGlobalCache: true
compressionLevel: 0
nmMode: hardlinks-local
Enter fullscreen mode Exit fullscreen mode
Collapse
 
trivikr profile image
Trivikram Kamat

I tried to reproduce the large number of system calls in yarn@3.1.1 with a vue vite-ts project, but the benchmarks show yarn has least number of syscalls:

  • npm v8.1.2: cold - 66912, warm - 9116
  • yarn v3.1.1: cold - 1386, warm - 864
  • pnpm v6.29.1: cold - 32890, warm - 5798

Details in GitHub discussions: github.com/yarnpkg/berry/discussio...

@atian25 Can you share the steps you used for counting number of system calls using strace?

Collapse
 
trivikr profile image
Trivikram Kamat

Update: The benchmark numbers were not accurate as the default modes for package managers are different. Also strace needs -f option to trace system calls from child processes.

I added new numbers in github.com/yarnpkg/berry/discussio...

  • yarn v3.1.1: cold - 221068, warm - 37702
  • pnpm v6.29.1: cold - 74734, warm - 28430

As per discussions in discord thread npm is not included in this benchmarks as:

  • npm cannot use central store.
  • it's not possible to turn off central store in pnpm.

yarn can be compared to npm by disabling central store.
yarn can be compared to pnpm by adding central store support to yarn's pnpm linker.

Collapse
 
rxliuli profile image
rxliuli

In my current experience, pnpm should be the fastest (much faster than yarn3, npm hasn't been used in a long time, but generally seems to be considered the slowest?), if tnpm has a part to see, I think To try the real performance for yourself.

Collapse
 
atian25 profile image
TZ | 天猪 • Edited

but generally seems to be considered the slowest?

pardon? from the table you can see that pnpm is faster than both npm and yarn. For Example, Test1, pnpm(14s), npm(48s), yarn(61s).

Unfortunately, As mentioned in the post, tnpm is still in the internal testing phase. There's currently no way for the community to experience it.

What we use is the official pnpm benchmark tests. As for why we are relatively faster, I believe this has been explained in detail in the post.

npmfs is a package manager agnostic tool by design. We hope it benefits not only cnpm/tnpm but also all the package managers that the community loves. Please stay tuned for npmfs to be open source. Nevertheless, the idea behind it are also worth sharing and discussing with the community.

Collapse
 
rxliuli profile image
rxliuli

This sentence I refer to the official npm

Collapse
 
picwellwisher12pk profile image
Amir Hameed

if you use pnpm, tell me, did you find it faster than yarn and npm?
Because to me it feels like everytime when I have to install new package it takes so long.
It was relativley better with npm. and even better with yarn.

Collapse
 
meatboy profile image
Meat Boy

This post is pure gold and among recently created posts, diamond. I’m saving it for tomorrow to read in the details.

Collapse
 
412799755 profile image
Luo Yu

Great innovation!
That's one small step for antgroup, one giant leap for npm community!

 
picwellwisher12pk profile image
Amir Hameed

for me it seems slow even for 2nd time and also when I try to add new modules