Introducing the Node.js package analyzer

Introduction

I'm proud to present to you the Node.js package analyzer.

tmkn / packageanalyzer

A framework to introspect Node.js packages

It's a framework designed to easily introspect a Node.js package.

Think of it as eslint for your Node.js package/project.

It's also what powered the data behind npmbomb.

Package Analyzer

While the JS ecosystem is quite mature and diverse, there is no linter for a Node.js project. If you want to do a license compliance check, it's a separate tool. If you wan't to check the health of a dependency, that's a separate tool. If you want to check for unneeded polyfills/deprecated packages, that's yet another tool.

The aim of the package analyzer is to provide the framework and toolset to easily answer questions like above within a single tool.

All while being flexible enough to answer any questions surrounding a Node.js package like:

• calculate the number of all dependencies
• find newest/oldest dependency
• find most included dependency
• get weekly downloads
• check for a new version
• release velocity
• etc

That's why the package analyzer is both an API and a CLI

Another reason why I started this project was security.

Personally I think there's to much trust put into adding dependencies.

Any Node.js package that you install can run a postinstall script and people tried to extract credentials via this way for some time now, yet there is no straight forward way to highlight packages with a postinstall script.

I also envision a system that highlights which API's a particular dependency is using. fs, http etc. or highlights differences between version upgrades: A new sub dependency was added? By whom?

Ideally the package analyzer will be able to answer those questions.

Architecture

The package analyzer is written in TypeScript so TypeScript types are a first class citizen.

Package

At the heart of all of it, is the Package class. After you've traversed a package and all of its dependencies you'll get a single instance of the Package class back. Important points of the API are as follows:

interface IPackage<T> {
    //each package parent's can be easily accessed
    parent: T | null;

    //vital information is directly accessible
    name: string;
    version: string;
    fullName: string; //`name@version`

    //dependencies are listed here
    directDependencies: T[];

    //convenience functions to iterate over the dependency tree
    visit: (callback: (dependency: T) => void, includeSelf: boolean, start: T) => void;

    ///easily find dependencies
    getPackagesBy: (filter: (pkg: T) => boolean) => T[];
    getPackagesByName: (name: string, version?: string) => T[];

    //access package.json data
    getData(key: string): unknown;

    //access custom data
    getDecoratorData<E extends IDecoratorStatic<any, []>>(decorators: E): DecoratorType<E>;
}

For more information about the architecture you can checkout the Architecture.md in the GitHub repository.

CLI

If you install the package analyzer globally:

npm install -g @tmkn/packageanalyzer@0.9.4

You will get a pkga command where you can easily introspect packages.

Some of the things it does currently are as follows:

Print Metadata

You can use the analyze option to print metadata.
If you don't provide a version number it will use the latest release.

pkga analyze --package react //use latest version
pkga analyze --package react@16.12.0 //use specific version

Use the --full option to print additional data like oldest/newest package.

pkga analyze --package react --full

If you want to analyze a local project use the --folder option:

pkga analyze --folder path/to/your/package.json

Print Dependency Tree

pkga tree --package react

Print Weekly Downloads

The downloads command will print the weekly downloads for a package for NPM

pkga downloads --package react

Cyclic Dependencies

Use the loops command to print cyclic dependencies in the dependency tree:

pkga loops --package webpack@4.46.0

API

In addition to the CLI, the Package Analyzer also offers an API.
All the commands in the CLI are done via this API as well.

Example

Here's how you can use it to list all dependencies of fastify that come with built int TypeScript support.
Type declarations are either marked via the types or typings field in the package.json, so all we need to do is ask if those fields are set and collect them:

const { Visitor, npmOnline, OraLogger } = require("@tmkn/packageanalyzer");

(async () => {
    try {
        const visitor = new Visitor(["fastify"], npmOnline, new OraLogger());
        const pkg = await visitor.visit();

        const matches = new Set();

        pkg.visit(dep => {
            if (dep.getData("types") || dep.getData("typings"))
                matches.add(dep.fullName);
        }, true);

        console.log("Built in TypeScript support:")
        for (const name of matches)
            console.log(name);
    }
    catch (e) {
        console.log(e);
    }
})();

        const visitor = new Visitor(["fastify"], npmOnline, new OraLogger());

First we need to create a Visitor that traverses the dependency tree.

The 1st argument is a tuple that specifies the package, if you don't provide a version. e.g.["fastify", "3.14.1"] it will default to the latest like here.

The 2nd argument is the Provider. Whenever the Visitor wants data about a package it will ask the Provider. In this case we are asking the NPM registry, but you could also write a Provider that gets the information from the filesystem e.g. node_modules folder.

The 3rd argument is the logger. All output is routed to this logger.

        const pkg = await visitor.visit();

Then we simply call the async visit function from the Visitor to start traversing the dependency tree. We will end up with a single top level Package class.

        pkg.visit(dep => {
            if (dep.getData("types") || dep.getData("typings"))
                matches.add(dep.fullName);
        }, true);

This Package class provides utility functions like visit to iterate over each dependency and the getData method to access the respective package.json. So here we iterate over each dependency and check if the respective package.json contains entries for types or typings. If yes, collect dep.fullName which is a string formatted to packagename@version.

        console.log("Built in TypeScript support:")
        for (const name of matches)
            console.log(name);

Afterwards we just print out our findings, done.

If you want to know more I recommend to check out the Architecture.md in the GitHub repository. But do note that the API is not yet stable and will likely undergo changes as it's a first preview.

What's next

Since I have a lot of ideas where to take this and since it's a first preview I would really appreciate feedback :)

In the beginning I said to think of it as eslint for packages, however defining your own set of checks that you want to run is not yet possible and something that I want work on next.

I'm also thinking about a web interface, that allows to better visually present the results and where you can jump back and forth between packages/reports in an easy manner.

Apart from that, any suggestions are highly welcomed.

For updates you can follow me on Twitter or follow the project on GitHub: https://github.com/tmkn/packageanalyzer