DEV Community: Hiroki Osame

Introducing pkg-size.dev

Hiroki Osame — Tue, 07 Nov 2023 15:00:00 +0000

"npm dependencies are too large"

If you've worked with JavaScript and npm, you're likely familiar with the notorious node_modules directory. This folder houses all your project's dependencies, and is widely known for its hefty size. It's not uncommon for a small project to exceed 100MB, and a quick Google search reveals many frustrated devs.

A popular meme about `node_modules`

While this problem can be improved by npm, the root problem lies within the individual packages installed. Most package authors don't intend to create bulky packages, but they may also not be cognizant of how a few unassuming files in their package can contribute an unwieldy node_modules directory.

This problem is also not limited to developers. If you're making a website, these packages are typically bundled together. Inefficiently designed packages will bloat your application and result in slower load times for your users.

To address this situation, the first step is to enter the node_modules directory. However, navigating through countless directories with obscure names, each containing numerous files, can be a daunting task.

Questions start swirling in your mind:

"Which files are essential and which can be removed?"
"Which packages are the largest?"
"Who can I report these issues to?"

It's understandable to feel overwhelmed.

Your new favorite tool: pkg-size.dev

pkg-size.dev

pkg-size.dev is a powerful online tool designed to answer these questions. It empowers JavaScript developers to examine npm packages effortlessly. It provides valuable insights into the installation size and bundle size of the packages, helping developers make informed decisions when choosing dependencies for their projects.

Want to see for yourself? Try it out!

Using pkg-size.dev should feel simple and familiar. Just as you would on your computer, start by typing the name of the npm package you want to examine into the terminal. Hit the Return key, and watch the magic happen. The tool will run npm install on the specified package and analyze the node_modules directory to get you the size of each package and the total size. It will then bundle the package with esbuild, and analyze that too.

To get started, explore some popular packages:

How does it work?

The most incredible part about this tool is all of this happens right within your browser, courtesy of the amazing WebContainers API!

WebContainers is a new technology developed by StackBlitz that allows you run Node.js in the browser. No need to set up a new project environment or deal with complex setups. Just a seamless experience right where you're most comfortable.

Because of this, the website can be completely static and doesn't need a backend service. As the author, removing server & maintenance costs was one of the most compelling reasons for creating this tool. It's exciting that many new ideas are now possible with this technology.

Features to explore

1. Install size insights

The "Install size" section for vue

Once the installation is complete, pkg-size.dev presents you with an informative Install Size section. Here, you'll find a detailed breakdown of every package that was installed during the process. The insights include:

The largest dependency installed.
The reasons behind why each package was installed.
Links to license, repository, homepage, and npm page.
Whether they have CommonJS/ESM/type distributions.
The different file types that make up the package.

With this comprehensive information at your fingertips, you can easily evaluate the impact each dependency has and make informed decisions.

Here are some questions you might want to consider for your evaluation:

Is there a package that's abnormally large?
Is there a package that's installed unnecessarily?
Is there a package that can benefit from ESM distribution?
Is there a package unnecessarily publishing files that can be ignored?
Is there a CLI package installing TypeScript types?

If you come across any issues or areas for improvement, don't hesitate to report them to the project repository. Your feedback could benefit the entire community.

2. Bundle size exploration

The "Bundle size" section for vue

In addition to the Install Size insights, pkg-size.dev also offers a Bundle Size section. This feature is especially useful when you're dealing with frontend packages that you expect to be bundled together. Unlike the installation size, the bundle size is usually much smaller because it only includes code that is actually referenced in your project, and it's also minified to reduce its footprint. As a tip, the bundle size can also give you an idea of how much bloat is present in your node_modules directory.

If the bundled package has an ESM distribution (JavaScript/ECMAScript Modules), you can experiment with an exciting optimization called "tree shaking" to reduce the bundle size even further. Try excluding a named import to see how many bytes it could save in the final bundle. This feature gives you the flexibility to play around with different scenarios and optimize your bundle effectively.

Ready to experiment? Try these ESM packages:

If you find a package that's including a lot of unnecessary code, and could benefit from tree shaking, you can report it to the project repository and help improve their package.

To dive deeper into the bundle size, you can also download the bundle and analyze it locally using your favorite tools.

3. Interaction between package dependencies

Another fascinating use-case of pkg-size.dev is observing how multiple packages with shared dependencies interact with each other. By providing multiple package names and specifying their semver ranges, you can gain insights into their compatibility and spot any potential issues.

Pass in multiple packages and semver ranges to see how they interact

For example, installing both esbuild and esbuild-loader@3.0.1 reveals that two different versions of esbuild are installed due to a mismatch in the semver ranges. This discovery could prompt maintainers to update their dependencies and release an update.

Two different versions of esbuild installed

Have feedback or ideas?

I'm committed to making pkg-size.dev a tool that gives you super powers.

If you have any feedback, feature requests, or bug reports, don't hesitate to share them in the GitHub repository. Your contributions will help shape the future of this tool.

Share your insights

As you dive into the npm ecosystem using pkg-size.dev, you're bound to discover interesting insights and valuable learnings. Feel free to share your findings in GitHub Discussions. These discussions could spark more ideas for features and ways to gain insight.

Show your support 💞

If you found pkg-size.dev to be useful and save you a lot of time, supporting the project is a fantastic way to show appreciation. If you find the tool useful, please consider sponsoring it on GitHub.

Happy analyzing and exploring the npm world with pkg-size.dev! ✌️

4 reasons to avoid using `npm link`

Hiroki Osame — Mon, 25 Apr 2022 18:15:54 +0000

TL; DR

Instead of using npm link, use npm install or npx link to symlink a local package as a dependency:

$ npx link <package-path>

npx link is a tool I developed as a safer and more predictable alternative to npm link.

Avoid using npm link because of the following footguns:

Error-prone with multiple Node.js versions
No fail-case and unexpected fallback to npm registry
Unexpected binary installation
Unexpected link removal

What's `npm link`?

npm link is a command-line tool for symlinking a local package as a dependency during development. It is commonly used for testing packages before publishing them.

Read more about it in the official documentation.

Usage

Given the following packages:

my-library: an npm package that you want to test in another package as a dependency.

The name property in my-library/package.json should be my-library.
my-application: the package/project you want to test in

Here's how you would link them:

Registration (Global installation)

Run npm link in my-library to install it globally, making it possible to link my-library to any local project. Note: this is the same thing as running npm install --global.

$ cd ./my-library
$ npm link

Installation

Run npm link my-library in my-application to link it:

$ cd ./my-application
$ npm link my-library

Shortcut

npm link <package-path> is a shortcut to automate the two steps by simply passing in the package path.

Using the example above:

$ cd ./my-application
$ npm link ../my-library

The shortcut approach is much easier to use and is less error-prone because it's a single command that requires an explicit path to the package to link.

4 footguns of `npm link`

1. Multiple Node.js versions

If your environment has multiple Node.js versions using a manager like nvm, both npm link commands must be run using the same version.

As explained above, the first step of npm link is installing the package globally. Since each version of Node.js has its own global package registry, lookups will fail if different versions are used.

You can check if the global package registry is scoped to the Node.js version with the following command. If the Node.js version is in the path, the global package registry is scoped:

$ npm root -g
~/.nvm/versions/node/v14.16.1/lib/node_modules

When working on multiple packages in separate terminal sessions, it's very easy to overlook the Node.js version. The version discrepancy can be especially hard to notice since npm link doesn't error when it's unable to find the local package to link, which is discussed in the next section.

Pro tip: Add the recommended shell integration to automatically use the appropriate Node.js version when entering a directory with a .nvmrc file.

2. Non-existent fail-case

Try running npm link a in a package.

It will succeed despite never registering package a to be linkable before:

$ npm link a
~/my-package/node_modules/a -> ~/.nvm/versions/node/v14.16.1/lib/node_modules/a

This is because when npm link can't find package a as a global package, it installs it globally from the npm registry and creates a symlink to it.

It only fails when the package is also not found on the remote registry:

$ npm link non-existent-package
npm ERR! code E404
npm ERR! 404 Not Found - GET https://registry.npmjs.org/non-existent-package - Not found
npm ERR! 404 
npm ERR! 404  'non-existent-package@*' is not in this registry.
npm ERR! 404 You should bug the author to publish it (or use the name yourself!)
npm ERR! 404 
npm ERR! 404 Note that you can also install from a
npm ERR! 404 tarball, folder, http url, or git url.

To tell if the link actually succeeded, you can check if the output has two arrows (->). (Notice how the false-positive above only has one arrow.) Two arrows means it created a symlink to the global package, which then points to the local package:

$ npm link my-linked-package
~/my-package/node_modules/my-linked-package -> ~/.nvm/versions/node/v14.16.1/lib/node_modules/my-linked-package -> ~/my-linked-package

This check only works in npm v6. Unfortunately, starting in npm v7, the symlink paths are no longer logged. Looking at the output, it's impossible to determine if linking the local package succeeded, or if an unintended package was accidentally installed and linked:

$ npm link a

up to date, audited 3 packages in 671ms

found 0 vulnerabilities

To confirm the package was successfully linked, you can use realpath to verify the symlink path:

$ realpath node_modules/package-name
~/my-linked-package

The lack of a proper fail case makes using npm link a confusing and frail process. Especially when compounded with having multiple Node.js versions.

3. Unexpected binary installation

The first step of npm link installs the package globally. This happens in the shortcut as well, because it just automates the two steps.

Global package installation (npm install --global ...) is a type of package installation used to make binaries available as a system-wide CLI command. So, if your package has a bin field, npm linking it will make it available as a CLI command.

Considering npm link is a tool for testing a package in development, global binary installation can be an unexpected and undesired side-effect. The implications of this unexpected behavior can be quite serious given packages can declare binaries with arbitrary names.

In this example package, an arbitrary binary name random-command is specified in the package.json file:

{
    "name": "my-package",
    "bin": {
        "random-command": "bin.js"
    }
}

Running npm link installs binary random-command:

$ random-command
zsh: command not found: random-command

$ cd my-package && npm link
added 1 package, and audited 3 packages in 548ms

found 0 vulnerabilities

$ random-command
Suddenly works!

Global install can also override existing binaries depending on your PATH configuration—the variable of paths the shell uses to lookup commands from. If you're using nvm, your configuration is likely susceptible to this.

In this example, I override the binary cat, a standard Unix utility:

$ type cat
cat is /bin/cat

$ cd my-package && npm link
added 1 package, and audited 3 packages in 230ms

found 0 vulnerabilities

$ hash cash
$ type cat
cat is ~/.nvm/versions/node/v16.14.0/bin/cat

In regards to software installation, these risks are prevalent in every software manager and aren't considered too dangerous from a security perspective.

However, npm link is not a package installer. It's supposed to be a simple tool to setup symlinks for development. It's worth pausing to reflect on how unexpected this behavior is, and what mistakes it could lead to.

By the way, if you ran npm link a in the prior section, a binary a has been installed to your system. You would think npm unlink a will uninstall it, but it only removes the local link and not the globally installed binaries.

Uninstall a global package and its binaries with:

$ npm uninstall --global a

4. Unexpected link removal

When linking multiple packages, previously linked packages are removed. This behavior is a regression introduced in npm v7.

In this example, pkg-a is linked and confirmed to be in node_modules. However, after linking a second package pkg-b, pkg-a is no longer in node_modules:

$ npm link ../pkg-a
added 1 package, and audited 5 packages in 684ms
found 0 vulnerabilities

$ ls node_modules 
pkg-a

$ npm link ../pkg-b
added 1 package, removed 1 package, and audited 5 packages in 703ms
found 0 vulnerabilities

$ ls node_modules  
pkg-b

Removing previous links can be unexpected and confusing when working with multiple packages. Often times, after linking the second package, we'd continue to run code expecting the links to persist.

To link multiple packages, you must pass in all package paths into one command:

$ npm link ../pkg-a ../pkg-b
added 1 package, and audited 6 packages in 645ms
found 0 vulnerabilities

$ ls node_modules 
pkg-a pkg-b

While this works, it's not a great developer experience. In development, we don't always know ahead of time all the packages that need to be linked. Or keep track of the previously linked packages.

This confusing behavior compounds to the poor usability and predictability of npm link.

Potential for accidents

As with any popular package registry, npm has a diverse collection with no standard for quality.

npm removes malicious packages, but risks mentioned above are not limited to attacks. When it's unclear whether the right package was installed, there is always potential for accidents.

Many packages on npm are designed to make changes to the file-system, such as rimraf or a code linter. In an accident, the consequences of running file-system altering code can be detrimental.

Installing the wrong package is possible with npm install as well, but the risks are higher with npm link when the footguns above come together:

Package names can collide. It's possible to link a local package with a name that's on the npm registry. This can happen when developing and testing a new or private package before realizing the name is already taken.
No local resolution error. If the package being linked can't be locally resolved, it will get resolved from the npm registry. If a package with the same name is found, an unexpected package can get globally installed.
Binaries are installed. If the wrong package is installed, it's unintuitive that binaries get installed and to realize it needs to be uninstalled globally. This leaves unexpected binaries to be left installed and accidentally invoked.

Use `npm install` instead

A better alternative to npm link is npm install using a package path:

$ npm install --no-save <package-path>

This creates a symlink to the package without installing it globally. This behavior is probably closer to what most people expect from npm link. The --no-save flag is to prevent the package path from getting saved in package.json.

However, this command still comes with a drawback. Like npm link, running npm install multiple times will remove previous links. To link multiple packages, pass in the package paths as arguments:

$ npm install --no-save <package-path-a> <package-path-b> ...

Introducing `npx link`

An even better alternative to npm link is npx link, a tiny tool I developed to tackle the problems addressed in this post.

Using the command is simple:

$ npx link <package-path>

npx link doesn't globally install the linked package or its binaries. It doesn't remove previous links. And it works across different versions of Node.js because it makes direct symlinks. It also has a clear fail-state when it can't resolve the package path.

If you want to use binaries from the package, they will only be installed locally and will only be executable with npx or via package scripts.

As an added benefit for the community, package linking will still work for those who accidentally type in npx link instead of npm link!

4 tips from my SWE promotion at Square

Hiroki Osame — Wed, 13 Apr 2022 21:17:36 +0000

I'm a Software Engineer at Square, and I was promoted to a Level 6 (team leader) Individual Contributor (IC) two years ago.

I didn't have a chance to reflect on my learnings, but a colleague recently reached out regarding my promotion packet. He was working on his application and wanted to see mine as an example and get some pointers.

I'm publicizing the advice I shared with him in hopes that others who are also applying for IC promotions can find it helpful.

Here are 4 tips I learned from my promotion application.

1. Prioritize collaboration over personal achievement

It took me two tries to get the L6 promotion.

One of the reasons my first application was rejected was because I showcased an incident where I independently "saved the day". I helped my team meet a Monday deadline by working over the weekend to fix a problem I discovered late Friday evening.

This came as a surprise to me because I felt like I was demonstrating competence, determination, and ability to carry the team.

However, the promotion committee argued that I was enabling poor work-life balance by setting a precedent for others to also work overtime. This was a project underestimation issue, and I shouldn't have carried the consequences independently.

Given that L6 is a team leader, it was important I operated at a higher level of awareness and leadership. Instead of acting alone, I should have discussed with the team so the deliverables and target dates could be adjusted and not disrupt anyone's work-life balance.

To prepare for my next promotion packet, I put collaboration work before independent contributions and sought opportunities to lead and raise others.

2. Collect testimonials from everyone you work with

The promotion packet has a section to request peer feedback from engineers at the target level I was applying to. In my case, L6 engineers. This section was tough for me to complete because I hadn't worked with many L6s. Not being able to ask most of my colleagues for feedback made me feel like I was at a disadvantage.

After struggling with this, I eventually came to the realization nothing was stopping me from including quotes & testimonials from others outside of the peer feedback section.

I started reaching out to everyone I worked closely with: L5 engineers, product managers, designers, and engineer managers. And I asked them for a testimonial on their experience working with me. My promotion packet was scattered with quotes from my peers in varying roles, adding lots of colors and boasting collaboration.

As an added benefit, personally reaching out for feedback allowed me to have more control over the process. The official feedback had to be solicited by my manager through a Google Forms survey, and sometimes, this results in unengaging or irrelevant feedback. Reaching out directly created opportunities to follow up for details/examples.

3. Share your Hype Doc when soliciting feedback

People don't often remember your contributions.

This is understandable given I struggled to remember my own as I put together my promotion packet. And when peer reviewers don't remember, they dread writing your feedback because it takes them a lot of work to revisit old documents and conversations. As a result, they might end up submitting low-quality feedback near the deadline.

To make it easy for anyone providing me feedback, I published my Hype Doc internally—a document of my accomplishments to track my growth. I made it into a website and hosted it on GitHub Pages, but a scrappy Google Doc with bullet points will suffice. The point of having it hosted online is to reduce friction for the reviewer to remember, learn, and talk about your work. Try to write less, make it skimmable, and prefer showing over explaining (via links and screenshots). I linked my Hype Doc everywhere, even in my Slack & GitHub statuses.

Making my contributions easily accessible helped me get insightful and relevant feedback in a timely manner.

4. Develop documentation habits

I struggled to find high-quality examples when trying to find links to demonstrate my work.

Collaboration often happened on temporary and unlinkable mediums like instant messaging or meetings. And the links I found had no description and required work to understand when linking them was supposed to make it easier. Worse case, they made me look difficult to work with.

I felt I missed a lot of opportunities at the cost of a minute to document them.

This frustration inspired me to adopt better habits that help me document my work with little effort:

Improve transparency & discoverability

I started moving discussions from private to public platforms.

When appropriate, I moved conversations away from direct messaging, email, and meetings. Instead, I started encouraging conversations on JIRA tickets, GitHub issues/discussions, and Google Group threads.

As a result, my conversations and contributions became preserved, discoverable, and linkable.

Improve readability

I started to broaden my target audience in my writing and paid extra attention to spelling, grammar, and formatting. Whenever possible, I would add links, screenshots, and context.

To write more inclusively, I adopted a great tip to think of the tools used by non-English speakers and people with disabilities: Are your sentences simple enough for Google Translate? Can your spelling and punctuation be dictated by a screen reader?

I also generally avoid slang or niche references (eg. jokes) as they can confuse outside participants.

These practices helped my writing become more accessible and easier to read. And as a bonus, my contributions looked credible and started to get referenced by others.

Improve skimmability

I started to adopt templates and structure to my writing to make them easier to skim.

For general writing, I adopted the Pyramid Principle—a technique to get my point across faster.

For commit messages, I adopted Conventional Commits—a convention for creating an explicit commit history.

For pull requests, I fill out "Problem", "Changes", and optionally "Other info".

As a result, the documentation I wrote was not only faster to read but also faster to write.

Parting thoughts

These tips helped me get the most out of my work to showcase in my promotion packet.

Interestingly, I also found them to have a real impact on me as an engineer. They helped me communicate better, collaborate more, and expand my reach.

After all, the engineering ladder is supposed to reflect your competence as an impactful engineer.

I hope you find these tips helpful to your growth as well.

Good luck on your promotion!