DEV Community

Cover image for Why is node_modules heavier than the universe? (No seriously, why is it so big?)

Why is node_modules heavier than the universe? (No seriously, why is it so big?)

Faiz Bachoo Shah on October 28, 2023

The problem Have you ever faced the issue of waiting what seems to be eternity while you are downloading dependencies after hitting that...
Collapse
 
ngfizzy profile image
Olufisayo Bamidele • Edited

Although it is true other languages are compiled and JS is not, that's not the major reason behind the heaviness of node_modules.

Python, for example, is not a compiled programming language, but it comes with a mature standard library that solves most of your daily programming needs, so you need fewer third-party dependencies. That said, if you choose to install a Python library, you're getting the library's source code, not a binary.

Unlike JS and Python, Golang is a compiled language, but when you install a dependency, you also get the source code of that dependency

If you run ls $GO_PATH/pkg/mod/github.com, you can see the source file of all go dependencies you're using in your projects, Not the binaries but the source code.

I think the main reason for the heaviness of node_modules comes in two folds.

  1. Javascript's lack of a standard library has forced the community to depend solely on third-party solutions. Nodejs has a standard library, but it's not as mature as what you get from, say, a language like Python.

  2. I think the Js community has generally adopted a culture of preferring someone else's code to theirs (I'm sometimes guilty). For example, [npmjs.com/package/is-string] package has 20+m weekly downloads. A package less than 30 lines of code (counting white spaces), and I'm almost certain that most people downloading this package could have just done return typeof value === 'string' and then we have is-string-and-not-blank with 90+k weekly download. If you ask me I'd say that's ridiculous

Collapse
 
faizbshah profile image
Faiz Bachoo Shah • Edited

You are correct, and if you read the blog, I did mention it that the reason why Golang projects are small even after downloading the source code, its because it has a rich standard, due to which most go dependencies will have less external dependencies, and your own project will also have less external dependencies.

Another thing you might have missed is that in languages like go and java, the dependency manager has very nice ability to not re-download a package which is already there. This is one of the faults of npm as a monorepo with multiple submodules might have the same dependency installed separately in each of the submodules, instead of doing it only once. In a language like Go, the dependency is managed centrally, which saves the storage of dependency.

Also, I slightly disagree on the part of python. I have definitely seen many python projects which uses multiple dependencies, and python dependencies are usually equal in size with node dependencies

Collapse
 
ngfizzy profile image
Olufisayo Bamidele

@faizbshah I must have scrolled past the paragraph where you talked about go dependencies.

Great post

And I'm excited about what new runtimes like bun and deno are bringing to the game.

Thread Thread
 
faizbshah profile image
Faiz Bachoo Shah

Same here!! Im very excited about the future of Bun. Its probably the most significant thing which happened in the ecosystem in the past 3-4 years

Collapse
 
zach_sents profile image
Zach Sents

totally. i bet there's tons of redundant dependencies in even the simplest tree

Collapse
 
faizbshah profile image
Faiz Bachoo Shah

Exactly. The redundancy is a big part why people have started to dislike npm as a package manager in recent times

Collapse
 
manchicken profile image
Mike Stemle

If you want to avoid node_modules, use esbuild or some other similar tool to only grab what you need.

Additionally, abandon TypeScript, as type definition modules and the TypeScript facade itself are pretty chonky. TypeScript may add value for the developer, but it adds zero value to the end-user of the program.

There are a number of compilers for JavaScript out there, and they all have their trade-offs.

This is one of those problems where it feels like developer complain about the unintended consequences of their own convenience, but it is possible to write great JavaScript apps without massive bundles and dependency payloads.

Collapse
 
andrewtrefethen profile image
AndrewTrefethen

Reduced bugs is a benefit to the user. Developer productivity and speed of delivery for features are benefits to the user. TS has multiple benefits for the end user. Things that make developers more efficient and more accurate are pretty much defacto good for end users unless the tools have tradeoffs that negatively impact users to a greater degree. Arguments about the impact on bundle size and time to interactive can certainly be had, but the assertion that TS has no benefit to users is just false.

Collapse
 
manchicken profile image
Mike Stemle

Naw, nothing you said requires typescript to solve it. There are a number of approaches and languages which produce the same outcomes you’re describing, and when you combine that with the reality that typescript is a tedious layer on top of JavaScript, and that the majority of its ecosystem is still JavaScript, I stand my my correct assertion.

I do wish that the typescript zealotry would stop, it’s not constructive.

Thread Thread
 
andrzej_kowal profile image
Andrzej Kowal

I think you are wrong here. Typescript is the best thing which was created for JS world from last 10 years. TypeScript will allow developers to improve their efficiency, improve confidence in the codebase and solution on which they are working. The codebase will become “understandable”, even through time, because of rich type system which JS misses. You are right in only one moment here - everything what was done in TS can be done in JS. But in large codebase pure JS become a pain in the butt and in most cases will take a lot more time and mental health to maintain. It’s always a point into TS money-box, benefits are outweigh a lot more, than possible drowback in performance (in correct setup and product architecture, with some development conventions the performance footprint can be can be very small)

Thread Thread
 
manchicken profile image
Mike Stemle

I'm really excited for the enthusiasm you're bringing to the table, but please be careful not to confuse your enthusiasm and preferences for objective fact. I don't think JavaScript is lacking a type system because I've been programming long enough to know that a type system—especially one that isn't present at runtime—isn't necessary.

Some people prefer a type system, and some don't. I have my own preferences, and as long as we all remember that our preferences are only that—preferences—we can all be friends.

Collapse
 
pengeszikra profile image
Peter Vivo • Edited

pnpm give a link instead copy same node_module library, so if you use same dependency multiple then it will be save your disk space

pnpm organized node_modules

Keep your package.json simple as you can. In my experience with vite + pnpm is the best for speed a storage too.

Also the npm module creator responsibility is keep the module dependencies keep at minimal level.
For example, my really minimalist npm module: react-state-factory still using peer-dependency for avoide React version collision:

  "peerDependencies": {
    "react": ">=16.0.0 <19.0.0",
    "redux-saga": "^1.2.3",
    "use-saga-reducer": "^3.0.0"
  },
Enter fullscreen mode Exit fullscreen mode

Even If someone just use useStateFactory hook, then do not need to install redux-saga and use-saga-reducer.

So I think node_modules is also heavy if someone is check how many different React version is included ( after few React module used)

Collapse
 
faizbshah profile image
Faiz Bachoo Shah

Yes, I mentioned pnpm in my blog post as well as in one of my comments. pnpm is far amazing than what npm has to offer!!

Collapse
 
johannesvollmer profile image
Johannes Vollmer • Edited

The next question then would be: Why, if any real project requires a JS compiler of some sort, why not use a real compiler instead? If we have to use a compiler either way, why not output a binary?

For any real project you need to use babel/TS/swc and a module system, then you can just use a real compiler instead.

And that's where WebAssembly comes in. The size of the node_modules folder is just one of the many problems that wasm will solve. Stop using JavaScript

Collapse
 
faizbshah profile image
Faiz Bachoo Shah • Edited

To answer your question, I think the reason is simple: No one in the ecosystem is too interested in taking up that job. It took years before somebody actually become frustrated enough to create a runtime env better than Node (talking about Bun). A reason why the ecosystem don't want to deal with binaries is because that will introduce the dreaded problem of platform-level machine code dependency. And, unless there's a dedicated team behind that goal, no one wants to deal with that issue even from a distance. Using JS as a source code intentionally or unintentionally helps keeping JS platform independent.

And as a fan of WASM, I agree with you xDD

Collapse
 
slobodan4nista profile image
Slobi

There is a lot of great stuff writen in JS you can't write in TS or it feels wrong. Personaly I using WASM as ascelerator, JS for libraries that are used a cross multiple projects, some up to 7 years old and TS for new apps. All that to meet you nead sort of glue.

Collapse
 
dannyengelman profile image
Danny Engelman

Node Modules is the Universe

It was originally created by Zaphod Beeblebrox in 1988, but since there was no WWW yet, he programmed it to eventually truly answer The Question.

So Node Modules has to contain every F&@^ Byte all programmers in the world can come up with

Some say Node Modules is the ultimate practical joke.

Collapse
 
jfftck profile image
jfftck

There are too many dependencies for the dependencies that you are downloading, if these dependencies were updated to use modern JavaScript, you might find that JavaScript has a built-in feature that removes a lot of extra code that is being downloaded as dependent packages. It is the issue of libraries not using the full capabilities of a language as that language evolves -- this is true of Java, C#, etc... It would be a lot of work to rewrite to use those features and that is why old libraries are continually used. Python and Go have large standard libraries, that is why it is less of an issue in these languages. This is technical debt and it is everywhere in software development.

Collapse
 
lucianobustos profile image
Luciano Bustos • Edited

You can take a look at pnpm.io
I’ve got really interesting results in installation times and storage

Collapse
 
faizbshah profile image
Faiz Bachoo Shah

Yes, I mentioned pnpm in my blog too. Its amazing!!

Collapse
 
pavelee profile image
Paweł Ciosek

Recommending to try bun.sh as package manager. It is not solving problem of amount of space but speed up download process.

Collapse
 
faizbshah profile image
Faiz Bachoo Shah

Yes, Bun has been a breath of fresh air in the JS ecosystem!!

Collapse
 
shricodev profile image
Shrijal Acharya

Great! learned something new today. 🍵

Collapse
 
faizbshah profile image
Faiz Bachoo Shah

That's great to hear!! :D

Collapse
 
livioribeiro profile image
Livio Ribeiro • Edited

Dependencies in go and rust are source files, compiled with your application, and python or ruby dependencies are source files just like nodejs. Java dependencies are compiled and it is very heavy.

Collapse
 
faizbshah profile image
Faiz Bachoo Shah

Thanks for the feedback!! I added some extra stuffs to explain it too. About Python you are correct, so removed it. Actually I didn't even noticed that I put it in the first place, but yeah the thing with large dependencies is not just limited to JS. About Java, the total dependencies size is usually smaller than the Node because maven and gradle is better with managing duplicate dependencies in the dependency tree, and they don't even need to download the source code. The duplication management is now finally starting to get handled by nice package managers such as pnpm

Collapse
 
tahadev profile image
taha.dev

Thanks, It was very usefull

Collapse
 
faizbshah profile image
Faiz Bachoo Shah

Thank You!!

Collapse
 
shafa45 profile image
Md Shafaullah

Insightful!

Collapse
 
faizbshah profile image
Faiz Bachoo Shah

Thank You!!