After the initial release I found some time to do some quality of life changes, among them are:
- Visualizing the dependency tree
- Refactored search to be future proof
- Updated the dataset
What is npmbomb?
npmbomb is a little website that I built where you can try and guess the number of total dependencies for popular npm modules to shine a light on the growing transitive dependencies number of npm modules. If you add 1 dependency to your project, that dependency might contain a mountain of other dependencies that you don't know of but now will be part of your project.
It does this by simply following every dependency of the dependencies field in the package.json down to the very last one. As such, the calculated numbers might surprise you.
If you want to know more, you can find an introduction here
Visualizing the dependency tree
There is now a Dependency Tree tab, switching to it will reveal the 🎉 dependency tree.
It will show the transitive dependency count as well as provide a link to npm by clicking on the link icon.
How are the numbers calculated?
The number on the right on each line is the transitive dependency count or total dependency count.
In case of React that number is 8.
Here's how it is calculated:
3 direct dependencies of React itself + all transitive dependencies of loose-envify (1) and prop-types (4). object-assign doesn't itself define any other dependencies so doesn't contribute to the transitive dependencies count.
To keep things snappy, the tree uses react-virtualized.
Apart from that the tree component is custom built.
While most trees would work just fine without react-virtualized since they are not that deep and big, Jest for example made the browser sweat.
Since the react-virtualized module was already in use on the search page, it was also used for the tree rendering to solve potential rendering bottlenecks.
Coming back to Jest and its humongous dependency tree, the very first working version resulted in a 20MB JSON payload just for the tree data.
The straight forward format looked like this:
interface IDependencyTree {
name: string;
version: string;
transitiveCount: number;
loop: boolean;
dependencies: IDependencyTree[];
}
Even though a dependency tree like Jest is not the norm, clearly that's too much. Even gzipped it still was 5MB.
When I reduced the length of the keys to single chars it was still 16MB.
So I changed it to provide a lookup table instead and only reference numbers(id) for the actual nested format:
The tree value looks like this:
export interface IDependencyTreeConfig {
//lookup
data: ITreeData[];
//nested tree structure
tree: IDependencyTreeStructure;
}
export interface ITreeData {
id: number;
name: string;
version: string;
count: number;
}
export interface IDependencyTreeStructure {
id: number;
dependencies?: IDependencyTreeStructure[];
}
I could potentially reduce it further by reducing the length of the keys here as well but already with this approach the payload went down to 7MB, while compressed with Brotli it's now around 47kb. For an outlier like Jest, I think it's acceptable.
Refactored search to be future proof
The long term goal of npmbomb is to have data for any(most) npm modules.
As of now the dataset is limited to a handful of the most popular modules.
As such the architecture of the search was tailored for this limited data set so it wouldn't scale with growing data.
To remedy this, the search now uses react-virtualized to display its results (just like the dependency tree), supporting any number of search results.
With the old search architecture everything was in memory so it wasn't prone to ajax race conditions and such, so it's implementation was really simple but again with a growing dataset this approach wouldn't be feasible any more. Now you would need to take care of Ajax race conditions, only fire the search request after the user stopped typing etc, dismiss the ongoing ajax request if the user starts typing again, etc, to handle all of this cleanly the search architecture is now powered by RxJs.
Here's the new search:
In fact, it looks like the old one, all the changes are behind the scenes ;)
Updated the dataset
The original data was based on July 2019 so I thought it was fitting to provide an update after a year, so now the data is based on July 2020. While this is already "outdated", the goal of npmbomb is not to provide up to date data. In fact, this would be very challenging to provide, as any new version of any module could alter any existing dependency tree. It's highly volatile. Instead the goal of npmbomb is to provide a ballpark number from mostly up to date data.
Interestingly the npm dataset grew within this 1 year from 23.9GB to 42.2GB.
And the total number of modules grew from 1 007 928 to 1 332 134. So within this 1 year npm saw 324 206 brand new modules.
What's next?
Things that I have on my mind for npmbomb:
Increase dataset
One of the immediate next steps is to increase the dataset aka looking at options to host the data. The dataset is currently hosted together with the webapp via netlify. While an awesome service, it eats into the free tier usage, I would rather like to move the data hosting to somewhere else and only host the web app on netlify. I'm open for suggestions.
Dependency tree improvements
Breadcrumbs for tree view
Show the path upon hovering over tree nodes, as with large trees you can get easily lost.
Filtering
Allow the user to search for specific modules.
Highlight modules where they appear in the tree
More information
Display more information for a module, like
- maintainer
- release date
- show source code
- etc.
More E2E testing
While there is E2E testing, it's not on a level where I would like it to be. The bulk of the testing is done via unit tests.
A11Y
Since the project is past the prototype stage, with a growing audience it makes sense to invest into accessibility.
Housekeeping
With the tree view added and other small tweaks here and there, it is now a good opportunity to take a step back and do some housekeeping.
You see there are a lot of ideas and things that I want to do. This is also a friendly reminder that npmbomb is totally Open Source.
Go checkout the new tree view: https://npmbomb.tmkn.dev/
Feedback highly appreciated, whether on Twitter or GitHub or here 🙃
Top comments (0)