I think I'm not the first one to talk about this problem even here in dev.to. I quick research trying to found any solution concluded with the image that is the head of this text. The node_modules folder is where your project dependencies are stored, common knowledge. Its weight is also common knowledge.
Why I decided to vent my frustration now
Black Friday is here! It means discounts and the opportunity to update your computer. Therefore I decided to buy a SSD to boost the performance of my laptop, from 1 TB HDD to 500 GB SSD. All my files right now sums 299 GB, so I will not lose much space, but I decided to do the housekeeping work even so, this includes making backups of my projects. Not all projects I make I put on GitHub, sometimes I'm just experimenting and it is not worth the trouble, but I keep them anyway.
When I started the copy & paste process I remembered how heavy node_modules are...
Some comparisons
One example that shows clearly the problem is the node_modules folder of my ToRead CLI project as you can see in the imagem below.
The size of the folder is not really the problem although I will get to that later, but 15.000 files and more than 1800 folders!? Are you kidding me?! It is a simple CLI project with 5 files! Just for a comparison, let's see how many files and folders there is in the Windows folder:
While the system was counting I really thought node_modules would win this, but no. In any case, the folder has almost half of the amount of files an entire operating system has!
As I've said, the problem when copying node_modules folder from one place to another is not the size, it is the amount of files and folders, the complexity of the tree. It is a nightmare for a HDD. It takes many minutes to discover all files let alone copy them. In the end, it also impacts npm performance and there are memes for that also.
Other comparisons come from my passion for serverless. It is not rare for me to implement the same function in both Java and Javascript and since you have to bundle the function along with its dependencies it is a good way to compare which one is more efficient in dependency management. In one of my projects I wtote the function in both languages with virtually the same features and Java bundle size is 11.1 MB and NodeJS bundle size was 29.0 MB. Therefore, NodeJS can do a better job at the size of dependencies as well.
What other languages do
Besides NodeJS I have experience dealing with dependencies in two more languages: Java and C#. They have, in my opinion, a very similar way of handling dependencies and a much more efficient way than NodeJS.
Java has Maven, Gradle and other dependency management applications that works basically the same. There is a remote repository of the dependencies, generally Maven Central and a local repository. Maven always looks for the dependency in the local repository first and if not found it downloads from the remote repository. The dependencies are not within the project, like node_modules folder, it is more global, it is downloaded once and can be used by many projects, just add to your pom.xml.
C# follows the same idea, you list your dependencies in a .csproj file and Nuget takes care of the dependencies having also a remote and a local repository. It is much more efficient to handle dependencies this way, download once use in any project locally.
I think there is also a difference in culture and the way the languages were structured and what people see as libraries. Java has a very mature core of libraries that can deal with almost anything, common scenarios or not. Therefore, libraries in Java generally are meant to be an abstraction of what Java already has, making it easier to use. Therefore, the libraries have a more shallow dependency tree, reaching the Java core libraries much quicker.
What I see in NodeJS on the other hand is the opposite, everything can become a library, even a library that sums two numbers (hypothetical example, I hope) and libraries are heavily dependent of one another generating deep dependency trees, many files and folders.
Conclusion & Discussion
I am certainly not qualified to criticize NodeJs structure and engineering, but as a user I clearly see a problem and some lessons from other languages that can be used to improve the dependency management, which is paramount nowadays for almost every application. What do you think this problem came to be and what has been done to solve it? It would be very interesting to hear from more experienced developers what you do to remedy this.
Latest comments (90)
Wow life is only on earth or somewhere else where we live. We need to save every assets every signal and evolve.
I still cant figure out how to fix my can't find module problem
The memes in this article were great 👍
As for a solution, it's containerization with docker. If you always build your apps into docker containers you never have loose node_modules folders cluttering up your filesystem. When it comes time to move your data, you're only moving your own source code and (optionally) any composed docker images you've built.
As an added bonus, your entire runtime environment gets shipped so your app can be spun up on any server with all your dependencies available regardless of the host OS's globally installed libraries. Personally, I use node:lts-alpine as my base image which weighs in at < 40mb.
I will cite Oskar Lindgren's response just for convenience:
This is straight to the point! Look, people even found a solution for housekeeping dependencies in global path: PNPM. Btw, having
node_modulesunder my nose is heck of the case: I don't need to unpack *.jar to access source code of each library that I wanted to investigate. I have confidence in which version of library I use and observe in project, plus I get deprecation notices in timely manner after any update.I just searched my Mac's HDD. 6,617 node_module folders!! 6.2GB. 898,647 files!!!!!!!
I think such ridiculous situation about npm modules is about to reach its apogee. And after it it will become trending to create projects with minimum dependencies.
I hope to start this trend with Dominion framework for NodeJs, dominion.js.org.
I figured out a way to exclude node_modules from Dropbox/Google Drive and it works really well. Might help ease people's frustrations with node_modules. I wrote an article about it: arshaw.com/exclude-node-modules-dr...
I never could understand why npm doesn't have a single local repo approach with simply linking.
Me neither. It would be easier and faster for everybody.
LMAO! I have to double check your cover image cause' i thought I was the only one with this perception (I'm newbe wuth node)
Copying lots of files on Windows has always been painfully slow. On my Mac, copying a node_modules folder, 625MB and 63,130 files. Took 30 sec.