DEV Community: Pavel Zeman

Analyzing Storage Consumption in Sonatype Nexus npm Repositories

Pavel Zeman — Tue, 20 May 2025 10:34:52 +0000

I've been using Sonatype Nexus Repository Community Edition (or just Nexus) for some time as a npm repository. While using it, I've observed, that it requires quite a lot of storage. So I decided, to analyze the storage usage and I present my results in this article.

Configuration

For this article, I've downloaded the latest version of Nexus and analyzed its storage requirements with the following configuration:

Version: 3.79.1-04
Database: Embedded H2
Blob store type: File

npm repository types

For npm, Nexus provides the following repository types, each serving a distinct purpose and having specific properties:

Hosted
Proxy
Group

Hosted repository

Hosted repository is designed to store npm packages that are published internally by an organization. This type of repository provides a private space for proprietary or custom packages, allowing users to upload, manage, and share their own npm packages securely within the organization.

Hosted repositories are writable, meaning users with appropriate permissions can publish new packages or update existing ones directly to Nexus. These repositories do not automatically fetch or cache external npm packages; their content is limited to what has been explicitly published to them.

Proxy repository

Proxy repository, on the other hand, acts as an intermediary between Nexus and a remote registry such as https://registry.npmjs.org. When a package is requested from a proxy repository, Nexus fetches it from the remote registry if it is not already cached locally. Once retrieved, the package is stored in the proxy repository, making subsequent requests for the same package faster and reducing external bandwidth usage.

Proxy repositories are read-only from the perspective of users, i.e. they cannot be directly published to. Instead, they serve as a transparent cache, improving reliability and performance for external dependencies, and ensuring that previously downloaded packages remain available even if the remote registry is temporarily unreachable.

Group repository

Group repository aggregates multiple repositories of any type. Typically, it combines one or more hosted and proxy repositories into a single unified endpoint. The group repository presents a consolidated view to npm clients so that users only need to configure a single registry URL in their npm settings. When a package is requested from a group repository, Nexus searches its member repositories in a defined order and serves the first match it finds. This configuration simplifies development and CI/CD workflows by allowing both internal and external npm packages to be accessed seamlessly through one URL.

In Nexus Community Edition, group repositories are read-only, i.e. they cannot be directly published to. All the packages must be published to a hosted repository, which can then be included in a group repository.

Storage principles

Nexus stores all its data in the following storages - relational database and blob store.

Relational database (H2 for this article) is used to store metadata about so-called assets. In Nexus terminology, asset in a npm repository is either a package root or a tarball. Package root is a single JSON containing information about the package itself, i.e. its name, versions, license, etc. (see for example package root of the react package). For each version, the package root also contains a link to its tarball, i.e. a compressed file containing the code of that specific package version. As an example, you can get tarball for react version 19.0.0.

Blob store contains actual asset data, i.e. a single JSON file for a package root and a compressed tgz file for a tarball. The package root JSON file is not compressed and can be quite large, because it contains metadata for all package versions. For example, current package root of the react package has about 5.7 MB. The blob store content can be stored either directly in a filesystem or in a cloud storage (Amazon S3 or Google Cloud Storage). In this article, we will focus on the filesystem storage.

As the relational database stores only asset metadata, its size is expected to be much smaller than the size of the blob store. As a result, when calculating the Nexus storage requirements, it should be sufficient to consider only the blob store. For example, my proxy repository with about 2.5 thousand assets requires 4 MB of storage in the relational database and more than 400 MB of storage in the blob store, i.e. the blob store is more than 100 times larger than the relational database.

Storage analysis

The amount of data stored by Nexus differs between repository types.

For hosted repositories, the following rules apply:

Nexus creates package root, when a first version of a package is published.
Nexus updates the package root every time a version of the package is published or removed.
Nexus creates/removes tarball, when a package version is published/removed.

Total size of the assets for a single package can then be calculated as a sum of the size of the package root and sum of all tarballs for all its versions. This size can be reduced by removing specific versions or the package itself. Another option is to define a cleanup policy, which automatically removes package versions (i.e. tarballs) based on defined criteria. However, I've found no way, how to automatically remove packages themselves.

For proxy repositories, the following rules apply:

Nexus creates package root, when package metadata or a package version is requested.
Nexus creates tarball, when a package version is requested.

Similarly to hosted repositories, cleanup policies can be defined to automatically remove package versions.

Group repositories are virtual by nature, they just group packages stored somewhere else. However, even group repositories require storage. Nexus does not directly reuse package roots from other repositories included in the group, but creates a local copy of the package root instead. On the other hand, the tarballs are reused, i.e. they are not stored in the group repository. Similarly to proxy repositories, package root is created, when package metadata or a package version is requested for the first time. Unfortunately, I've found no way, how to remove obsolete package roots other than completely recreating the group repository. I've also found no way, how to check, which package roots are physically stored in the group repository other than querying the metadata in the relational database.

A summary of the storage analysis is provided in the following table.

Repository type	Package root	Tarball
Hosted	Created when first package version is published	Created/removed when package version is published/removed
Proxy	Created when package metadata or first package version is requested	Created when package version is requested
Group	Created when package metadata or first package version is requested	Reused from other repositories included in the group, i.e. no extra storage required

To analyze storage of a specific repository, we can use Nexus GUI, which provides limited information, but we can also query the relational database to get all the details. For example, the following SQL query provides a summary of all assets and their sizes grouped by repository and asset kind (package root or tarball).

SELECT
    r.name, a.kind, count(*) c, sum(ab.blob_size) bytes
FROM
    npm_asset a JOIN
    npm_asset_blob ab ON a.asset_blob_id = ab.asset_blob_id JOIN
    npm_content_repository cr ON cr.repository_id = a.repository_id JOIN
    repository r ON r.id = cr.config_repository_id
GROUP BY
    r.name, a.kind
ORDER BY
    r.name, a.kind

Key takeaways

Nexus supports 3 types of npm repositories (hosted, proxy and group), each serving a distinct purpose and behaving differently in terms of storage.
Group repositories are virtual by nature, but they still require storage for package roots.
Blob store size is typically the dominant factor in overall storage requirements.
Cleanup policies can be used to automatically remove package versions, but not packages themselves.

The Pitfalls of Streamed ZIP Decompression: An In-Depth Analysis

Pavel Zeman — Mon, 12 May 2025 22:07:06 +0000

Wikipedia says it clearly: "Tools that correctly read ZIP archives ... must not scan for entries from the top of the ZIP file". In other words, streamed decompression of ZIP archives is not possible. Still, there are some libraries (e.g. unzipper), which support it. Is it safe to use them? Does it work? Let's try to analyze it.

Motivation

As a motivation and basis for further discussion, let's create a simple ZIP archive using the following simple script (you can clone it from my GitHub repository):

const JSZip = require("jszip");
const fs = require("fs");

const zipItem = new JSZip();
for(let i = 0; i < 3; i++) {
  const content = Array.from({ length: 100 }, (_, j) => String.fromCharCode(j)).join("");
  zipItem.file(`dummy-${i}.txt`, content);
}

zipItem.generateAsync({ type: "nodebuffer", streamFiles: true })
  .then((content) => {
    const zip = new JSZip();
    for (let i = 0; i < 2; i++) {
      zip.file(`invalid-item-${i}.zip`, content);
    }
    zip.generateNodeStream({ type: "nodebuffer", streamFiles: true })
      .pipe(fs.createWriteStream("invalid.zip"));
  });

The script creates a ZIP archive invalid.zip, which contains nested ZIP archives in the following structure:

invalid-item-0.zip - 664 bytes
- dummy-0.txt - 100 bytes
- dummy-1.txt - 100 bytes
- dummy-2.txt - 100 bytes
invalid-item-1.zip - 664 bytes
- dummy-0.txt - 100 bytes
- dummy-1.txt - 100 bytes
- dummy-2.txt - 100 bytes

We can verify, that the ZIP archive is valid using unzip:

user@localhost:~$ unzip -tl invalid.zip
Archive:  invalid.zip
    testing: invalid-item-0.zip       OK
    testing: invalid-item-1.zip       OK
No errors detected in compressed data of invalid.zip.

Now let's try to decompress the ZIP archive as a stream using the unzipper library and list all files inside it together with their sizes:

const fs = require("fs");
const unzipper = require("unzipper");

fs.createReadStream("invalid.zip")
  .pipe(unzipper.Parse())
  .on("entry", async (entry) => {
    let size = 0;
    entry.on("data", (chunk) => size += chunk.length);
    entry.on("end", () => console.log(`File: ${entry.path}, size: ${size} bytes`));
  })
  .on("finish", () => console.log("Finished processing all entries"))
  .on("error", (err) => console.error("Error during processing:", err));

The output is as follows:

File: invalid-item-0.zip, size: 141 bytes
File: dummy-1.txt, size: 100 bytes
File: dummy-2.txt, size: 100 bytes

We expect the output to contain just 2 files - invalid-item-0.zip and invalid-item-1.zip. The size of each of them should be 664 bytes. Instead, we get 3 files. The first one has invalid size and the others are read from inside of invalid-item-0.zip, which is completely nonsense. Additionally, you can see, that the finish event is never processed and its log record is missing.

If you try to modify the archive content, you may get other results including various errors. And if you search for issues of the unzipper library, you can find about 10 of them mentioning similar problems.

All of these have the same root cause - zipped data cannot be reliably decompressed as a stream. The only way to reliably decompress zipped data is to save it to a file and then decompress the file.

We can use the same library and decompress the same ZIP archive as a file as follows:

const unzipper = require("unzipper");

(async () => {
  const directory = await unzipper.Open.file("invalid.zip")
  for (const file of directory.files) {
    let size = 0;
    const stream = file.stream();
    stream.on("data", (chunk) => size += chunk.length);
    await new Promise((resolve) => stream.on("finish", resolve));
    console.log(`File: ${file.path}, size: ${size} bytes`);
  }
  console.log("Finished processing all entries");
})();

Now we get the expected output:

File: invalid-item-0.zip, size: 664 bytes
File: invalid-item-1.zip, size: 664 bytes
Finished processing all entries

ZIP archive structure

In order to understand the problem, let's analyze the ZIP archive structure. The complete description is available in Wikipedia and all the details are available in PKWARE Inc. ZIP File Format Specification. This text contains only a brief summary needed to understand the presented problem.

The ZIP archive (usually) starts with a series of entries, each of them representing a single stored file. Each entry consists of the following items:

Local file header - Contains signature (4-byte constant), file name, compressed and uncompressed size, and other metadata. Compressed and uncompressed sizes are optional and can be set to 0, when they are not known during compression. This is used, when the compression is streamed and the compressed and uncompressed sizes are not known in advance.
Actual compressed data - Byte stream containing the compressed data.
Optional Data descriptor - Contains signature (4-byte constant), compressed and uncompressed size and other metadata. It is present only when the compression is streamed and in this case, the sizes are not optional, because even when streaming, they are already known.

At the end of the archive, there is a central directory. It consists of the following items:

Central directory file header - An extension of the Local file header for each stored file. It always contains real compressed and uncompressed size, and it also contains offset of the Local file header for the file inside the archive.
End of central directory record - Data structure, which must be always present at the very end of the archive. Among others, it contains offset of the start of the central directory.

The End of central directory record is the only data structure, that has a guaranteed position in the archive - it must be at the very end. Positions of all other data structures are not guaranteed, although the specification mentions, that they should be in the order mentioned in this description with no gaps between them.

The whole structure is summarized in the following diagram. Again, please note, that the archives should be created with data structures in the order shown in the diagram, but it is not guaranteed. For example, an archive with gaps between entries is perfectly valid and must be correctly decompressed.

ZIP archive decompression from a file

To decompress a ZIP archive, we just need to follow the arrows in the previous diagram. You may notice, that all the arrows go bottom-up, which means, that the archive needs to be read from the end to the beginning as follows:

We read the End of central directory record. It is always located at the very end of the archive. Among others, it contains the offset of the start of the central directory.
We scan the central directory to get list of all files from Central directory file headers. We get file names, offsets in the archive, and compressed and uncompressed sizes.
For each file, we get its compressed data and decompress it using the ZIP decompression algorithm (details about the compression method are stored in the Central directory file header as well).

To summarize, the only reliable method to read a ZIP archive is to read it from the end to the beginning. This makes it impossible to decompress the zipped content as a stream, because it can be only read from the beginning to the end.

ZIP archive decompression from a stream

But wait. The unzipper library is actually able to decompress a ZIP archive from a stream. How is it implemented?

The library is based on a simple assumption: The ZIP entries start at the beginning of the archive and follow one by one with no gaps between them. This is quite safe assumption, since the ZIP file format specification states, that all tools should create ZIP archives exactly this way.

With this assumption in mind, we can design an alternative algorithm to decompress a ZIP archive from a stream:

We read the Local file header of a ZIP entry (the first one is at the beginning of the stream, the other ones follow without gaps). It contains the file name as well as its compressed and uncompressed sizes.
Based on the compressed size from the previous step, we read the compressed data and decompress it using the ZIP decompression algorithm.
If there is a Data descriptor present, we can either skip it or use it to verify the uncompressed data checksum (CRC-32).
We continue from the first step until we reach the start of central directory. This can be easily recognized using the signature of the Central directory file header or End of central directory record, if there is no file.
We skip all central directory entries as well as the End of central directory record. These are not needed anymore.

This algorithm works, but there is a catch. The sizes in the Local file header are optional, and they are not set, when the ZIP archive is created as a stream. As a result, we do not know the size of the compressed data. How to solve this? Let's check the library source code. The relevant part is located in parse.js at lines 181 through 187:

if (fileSizeKnown) {
  entry.size = vars.uncompressedSize;
  eof = vars.compressedSize;
} else {
  eof = Buffer.alloc(4);
  eof.writeUInt32LE(0x08074b50, 0);
}

We can see, that if the compressed size is known, it is used. Otherwise, the library searches for the end of the compressed data based on a 4-byte signature (0x08074b50). This is the signature of the Data descriptor, which is located immediately after the compressed data. Searching for the compressed data using the 4-byte signature may work, but it fails, when the signature is present in the compressed data itself. As we can hardly assume anything about the compressed data, I consider this approach to be too risky.

To summarize, the library makes the following assumptions in order to decompress a ZIP archive from a stream:

The ZIP entries start at the beginning of the archive and follow one by one with no gaps between them.
One of the following is true:
1. The archive was not created as a stream (i.e. all the sizes in the Local file header are known).
2. The compressed data does not contain the signature of the Data descriptor.

Based on these assumptions, the algorithm to decompress a ZIP archive from a stream can be refined as follows (this is the algorithm used by the library):

We read the Local file header of a ZIP entry (the first one is at the beginning of the stream, the other ones follow without a gap). It contains the file name as well as its compressed and uncompressed sizes.
If the compressed data size from the previous step is known, we read the compressed data of that size. If it is unknown, we read the compressed data until we find the Data descriptor signature.
We decompress the compressed data using the ZIP decompression algorithm.
If there is a Data descriptor present, we just discard it.
We continue from the first step until we reach the start of central directory. This can be easily recognized using the signature of the Central directory file header or End of central directory record, if there is no file.
We skip all central directory entries as well as the End of central directory record. These are not needed anymore.

invalid.zip archive analysis

Now, it should be clear, why our invalid.zip archive cannot be decompressed as a stream. The first assumption is satisfied, but the second one is not. The invalid.zip archive is intentionally created as a stream (notice streamFiles: true in the source code) and its compressed data contains the signature of the Data descriptor.

The last point does not have to be clear at first sight, so let's analyze it in more detail. First of all, notice that there is no compression level specified in the source code, when creating the invalid.zip archive. As a result, compression level 0 (i.e. no compression) is used by default. This means, that the contents of the invalid-item-0.zip and invalid-item-1.zip files are simply copied to the invalid.zip archive, which leads to the structure shown in the following diagram.

This structure is then processed as follows:

We read Local file header of the first file in the archive, i.e. invalid-item-0.zip.
The Local file header contains unknown data size, so we search for the signature of the Data descriptor.
The first Data descriptor is the first Data descriptor of the first file inside invalid-item-0.zip, i.e. file dummy-0.txt. This is not the Data descriptor that we want, but we don't know that, so we finish the processing of the first file.
We read the Data descriptor and discard it (the library does not use it in any way).
We continue processing with the next Local file header, which is the Local file header of dummy-1.txt. Please note, that now we process a file, that does not exist in invalid.zip archive, it exists only inside invalid-item-0.zip.
We search for the following Data descriptor, find it and finish processing of dummy-1.txt.
In the same way, we process dummy-2.txt.
Now, we identify a Central directory file header based on its signature. This means, that we are at the end of the archive.
We drain all following Central directory file headers until End of central directory record is reached.
End of central directory record denotes the very end of the archive, so we stop processing here without any error, but without processing invalid-item-1.zip at all.

Trying to improve it

Based on the previous text, it should be clear, that streamed decompression of zipped data is a bad idea and can never be reliable. The ZIP file format is simply not designed for it. Still, it is tempting to use it, when it works in many cases. So how to improve the unzipper library so that it provides better results than presented in this text?

I would suggest the following improvements:

Documentation - Add a big red warning to the documentation stating, that streamed decompression of zipped data is not reliable and users of this feature do it at their own risk.
Verify compressed data size based on the Data descriptor - Currently, the library does not use the Data descriptor, which contains useful information - among others there is the real compressed data size. We can leverage it and compare it with the amount of data already processed. And if there is a difference, we can either fail with a reasonable error message stating, that the archive cannot be decompressed as a stream, or we can continue processing the archive until we read all the compressed data.
Verify CRC-32 of the uncompressed data based on the Data descriptor - Same as the previous one, but in this case CRC-32 of the uncompressed data is verified.

Alternative libraries

There are other libraries, which can be used to decompress ZIP archives. Their support of streamed decompression is as follows:

adm-zip - Streamed decompression is not supported. The API only accepts a file name as its input.
decompress - Streamed decompression is not supported. The API only accepts a file name or a buffer as its input.
decompress-zip - Streamed decompression is not supported. The API only accepts a file name as its input.
extract-zip - Based on yauzl, so streamed decompression is not supported.
jszip - Streamed decompression is not supported. The API only accepts a file name or in-memory data as its input.
node-stream-zip - Streamed decompression is not supported. The API only accepts a file name as its input.
unzip-stream - Based on unzipper, so streamed decompression is supported. And it works better than unzipper, because it implements my second improvement mentioned above. Thanks to it, it is even able to successfully decompress the invalid.zip file as a stream. However, the documentation clearly states, that streamed decompression of a ZIP archive is not supported and the library may fail in some cases.
yauzl - Streamed decompression is not supported. The documentation explicitly states, that this is intentional, because streamed decompression of a ZIP archive is not possible.
zip-lib - Streamed decompression is not supported. The API only accepts a file name as its input.

Key takeaways

The ZIP file format is not designed for streamed decompression, because the archive must be read from the end to the beginning.
Avoid streamed decompression of zipped data as much as you can. Always prefer to store the stream to a file or a memory buffer and decompress it from there.
Streamed decompression of zipped data is reliable, when you can make certain assumptions about the archive, that you decompress. But this is rarely the case.
The unzipper can be improved so that it at least fails with a reasonable error message, when the archive cannot be decompressed as a stream.
When streamed decompression is required, consider using unzip-stream, which provides better results than unzipper.

Demystifying npm package installation: Insights, analysis and optimization tips

Pavel Zeman — Tue, 22 Apr 2025 21:24:33 +0000

When analyzing issues with our internal npm registry, I was quite surprised how the package installation in npm works. This post summarizes the principles, analyzes the behavior using a sample package.json and points out some gotchas that you may encounter.

Disclaimer: I'm aware, that there are other (and newer) package managers available, but npm has still been widely used and at least for my work, I have no other choice.

Package registry

Before delving into the details of npm install, let's look briefly at the package registry, which provides the packages to install. You are well aware of the public registry at https://registry.npmjs.org/, but there are many other public and private registries as well.

API specification

The package registries provide simple REST API to get package metadata as well as actual package contents:

GET /{package} - Gets package metadata for all package versions.

For example, https://registry.npmjs.org/react provides metadata about the react package. Notice, that the metadata is quite large. In this case, it is about 5.6 MB (1.2 MB compressed) at the time of writing. This is caused by the fact, that the metadata contains details for all package versions, which means 2,280 versions in this case.
GET /{package}/{version} - Similar to the previous one, but returns metadata just for a single version. The version must really be a single version, no floating versions like ^19.0.0 are supported.

For example, we can get metadata for react version 19.0.0 using https://registry.npmjs.org/react/19.0.0. As there is just a single version, the metadata is much smaller - about 2 KB in this case.
GET /{package}/-/{package}-{version}.tgz - Gets actual package content as a single compressed tarball. This endpoint is not documented (and does not have to be), because it is referenced from the package metadata.

For example, we can download react version 19.0.0 from https://registry.npmjs.org/react/-/react-19.0.0.tgz.

There are also endpoints used e.g. for package publishing, but these are not needed for package installation, so they are not mentioned here.

API behavior

It is obvious, that in general npm install can use only two out of these endpoints - the first one to get package metadata for all versions and the last one to get package content. The second one cannot be used in many cases due to floating package versions being requested. When evaluating floating versions, npm must use the first endpoint to get metadata for all versions and then evaluate the floating version locally.

Metadata endpoint

As we will need it later, let's analyze behavior of the first endpoint using curl:

curl -I -H 'accept: application/json' https://registry.npmjs.org/react

HTTP/2 200
date: Sun, 20 Apr 2025 19:52:23 GMT
content-type: application/json
cf-ray: 93372e8c4bdbf99e-PRG
cf-cache-status: HIT
accept-ranges: bytes
access-control-allow-origin: *
age: 141
cache-control: public, max-age=300
etag: "25b33cc1cc58fb5a156c19f2ec3408ca"
last-modified: Sun, 20 Apr 2025 01:17:26 GMT
vary: accept-encoding, accept

The endpoint returns package metadata (not shown here) and a couple of headers. Out of these headers, we will further need the following ones:

cache-control - based on this header, the metadata should be cached for 300 seconds, i.e. 5 minutes. I have checked several packages and this value is the same for all of them, so it seems, that it is a common default.
etag, last-modified - values of these headers can be used in the future to check, whether there is a new version available.

Let's try to use the last-modified header in another request:

curl -I \
  -H 'accept: application/json' \
  -H 'if-modified-since: Sun, 20 Apr 2025 01:17:26 GMT' \
  https://registry.npmjs.org/react

HTTP/2 200
date: Sun, 20 Apr 2025 19:59:19 GMT
content-type: application/json
cf-ray: 933738b08aa0b377-PRG
cf-cache-status: HIT
accept-ranges: bytes
access-control-allow-origin: *
age: 254
cache-control: public, max-age=300
etag: "25b33cc1cc58fb5a156c19f2ec3408ca"
last-modified: Sun, 20 Apr 2025 01:17:26 GMT
vary: accept-encoding, accept

The result is quite surprising. With the if-modified-since header, the server should have returned HTTP code 304 Not Modified, but we get 200 instead. You can try to use etag with if-none-match and the result is the same. Even when specifying both if-modified-since and if-none-match, HTTP 200 is still returned.

This means, that the public npm registry does not support effective checks of new versions. As we will see later, after the cache expires (i.e. after 5 minutes), the complete metadata has to be re-downloaded even if there is no change.

For the purposes of this post, I also have my own local npm registry running on Sonatype Nexus. The registry is set up to mirror content from the public https://registry.npmjs.org, so we can easily repeat the previous test:

curl -I \
  -H 'accept: application/json' \
  -H 'if-modified-since: Sun, 20 Apr 2025 01:17:26 GMT' \
  http://localhost:8081/repository/npmjs/react

HTTP/1.1 304 Not Modified
Date: Sun, 20 Apr 2025 20:24:56 GMT
Server: Nexus/3.79.1-04 (COMMUNITY)
X-Content-Type-Options: nosniff
Content-Security-Policy: sandbox allow-forms allow-modals allow-popups allow-presentation allow-scripts allow-top-navigation
X-XSS-Protection: 1; mode=block
ETag: W/"25b33cc1cc58fb5a156c19f2ec3408ca"

Now, HTTP code 304 is correctly returned. This means, that if the metadata is not modified, only a few HTTP headers (and no content) are transferred resulting in substantial bandwidth savings.

There is one more difference when comparing the behavior of my local Nexus with the public npm registry - my local Nexus does not return cache-control header. As a result, the client application (npm in this case) does not know, how long the returned data can be safely cached, so it has to use a heuristic algorithm, as we will see later.

Package content endpoint

The package content endpoint works similarly:

curl -I -H 'accept: text/html' \
  https://registry.npmjs.org/react/-/react-19.0.0.tgz

HTTP/2 200
date: Sun, 20 Apr 2025 20:57:35 GMT
content-type: application/octet-stream
cf-ray: 93378e0f7e5771ef-PRG
cf-cache-status: HIT
accept-ranges: bytes
access-control-allow-origin: *
age: 1705545
cache-control: public, immutable, max-age=31557600
etag: "7860ab2d152873bfbc3e990b2bbc62da"
last-modified: Thu, 05 Dec 2024 18:10:24 GMT
vary: Accept-Encoding

We can see similar headers as for the package metadata:

cache-control - based on this header, the metadata is cached for 31,557,600 seconds, i.e. 1 year. Again, this value seems to be the same for all packages.
etag, last-modified - values of these headers can be used in the future to check, whether there is a new version available

If we try to use the last-modified value to check for new version, the behavior is the same as for the package metadata endpoint - we get HTTP code 200 with the public npm repository and 304 with my local Nexus. And the cache-control header is also missing for Nexus.

Package cache

To optimize package installation, npm maintains local cache of package metadata and package content. This is described in the npm documentation. However, the documentation does not mention anything about the cache policy. It is not clear when a cache entry is considered stale and needs to be refreshed.

To analyze the cache policy, we can check the source code. The cache policy is implemented in the http-cache-semantics package in its index.js file. After analyzing this file, we can easily find out, that the cache behavior depends on the HTTP cache headers. Without any technical details, the principles can be summarized as follows:

If there is cache-control header with max-age parameter, then this value is used. This means, that package metadata fetched from https://registry.npmjs.org is cached for 5 minutes, while package content is cached for 1 year. 5 minutes for package metadata seems to be quite aggressive, but up-to-date metadata is probably more important than the extra bandwidth required for frequent checks. One year for package content is fine because the content should never change.
If there is no cache-control header (as is the case for my local Nexus), the caching behavior is not defined by the server. In this case, the implementation uses heuristics defined in RFC 7234, which calculates max-age as (current-time - last-modified) * 0.1. As a result, the data is cached one-tenth of the time elapsed since its last modification. For example, if the data was last modified 10 months ago, it will be cached for one month. If this behavior is not intended, it can always be overridden for example using a reverse proxy, which adds the cache-control header to standard headers returned by Nexus. Using this approach, the same behavior as for https://registry.npmjs.org can be configured.

The cache behavior can be observed when running npm install with the --loglevel=http option. The output then looks as follows:

npm http fetch GET 200 https://registry.npmjs.org/@testing-library%2fjest-dom 11ms (cache hit)
npm http fetch GET 200 https://registry.npmjs.org/@testing-library%2freact 5ms (cache hit)
npm http fetch GET 200 https://registry.npmjs.org/react 13ms (cache hit)
npm http fetch GET 200 https://registry.npmjs.org/react-dom 15ms (cache hit)
npm http fetch GET 200 https://registry.npmjs.org/@testing-library%2fuser-event 4ms (cache hit)
npm http fetch GET 200 https://registry.npmjs.org/@testing-library%2fdom 5ms (cache hit)
npm http fetch GET 200 https://registry.npmjs.org/react-scripts 4ms (cache hit)
...

Cache behavior is shown at the end of each request as cache <status>. Specific values are not documented, but based on my analysis of the npm source code, they are as follows:

cache hit - The requested data was found in the cache and was not fetched from the server. The HTTP status 200 in this case is misleading because there was no HTTP request sent.
cache miss - The requested data was not found in the cache, so it was fetched from the server.
cache revalidated - The requested data was found in the cache, but it is stale, so a request with if-none-match and if-modified-since headers was sent to the server. The server returned HTTP 304 without any data, so the cache item was marked as up-to-date. Similarly to cache hit, the log contains HTTP status 200, which is misleading.
cache updated - Similar to cache revalidated, but in this case server responded with HTTP 200 and provided up-to-date data.
cache stale - The requested data was found in the cache, it was identified as stale, but it was used anyway because, for example, cached data is preferred using --prefer-offline option. The HTTP status 200 in this case is misleading again because there was no HTTP request sent.

Installing packages using npm install and others

When installing packages using npm install (just plain npm install without any options is assumed), npm performs the following steps (see documentation):

package.json resolution - npm reads the package.json file.
Package resolution - based on the direct dependencies specified in package.json, npm calculates complete dependency tree including transitive dependencies. If there is a package-lock.json file present, npm first checks, if the locked versions satisfy version requirements from package.json. If they do, the package-lock.json is used instead of resolving the dependency tree, otherwise the conflicts are resolved using the dependency tree.
Package download - npm downloads all missing packages used in the dependency tree.
node_modules creation/update - npm creates node_modules directory based on the dependency tree and the downloaded packages. If the node_modules directory already exists, npm updates its content to reflect the dependency tree.
Scripts execution - npm runs scripts defined in installed package.json files.
package-lock.json create/update - npm creates or updates the package-lock.json file to reflect the installed packages.

But npm install is not the only command to install packages. Another command is npm update (see documentation), which behaves similarly to npm install. The key difference is in the way, how existing package-lock.json file is handled. npm update ignores the package-lock.json file, always calculates complete dependency tree and updates package-lock.json accordingly. If there is no package-lock.json file, npm update behaves the same as npm install.

The last package installation command is npm ci (see documentation). This command requires existing package-lock.json, but otherwise its behavior is similar to npm install. The key difference is in the way, how conflicts between package.json and package-lock.json are handled. npm ci does not try to resolve them, but fails instead.

Behavior of all package installation commands can be summarized using the following table.

Scenario	npm install	npm update	npm ci
No `package-lock.json`	Resolves complete dependency tree using `package.json` and installs resolved dependencies	Same as `npm install`	Fails
`package-lock.json` consistent with `package.json`	Installs dependencies based on `package-lock.json`	Same as with no `package-lock.json`	Same as `npm install`
`package-lock.json` inconsistent with `package.json`	Resolves dependency conflicts and installs dependencies based on `package-lock.json`	Same as with no `package-lock.json`	Fails

Package installation behavior analysis

Now, we have all the important theoretical information about npm, so let's try to analyze package installation in more detail. To do this, we are going to install npm packages in multiple scenarios and observe npm behavior in each of them.

Environment

Hardware
- CPU: Intel(R) Core(TM) Ultra 7 155H@1.40 GHz
- RAM: 32 GB
- Network bandwidth: 100 Mbps (this applies to my local Nexus registry as well)
Software
- OS: Ubuntu 24.04.1 LTS (in a virtual machine)
- Node.js version: 22.14.0 (latest LTS version)
- npm version: 10.9.2 (bundled with the Node.js)

Scenarios

The tested scenarios are as follows:

npm ci with no cache
npm ci with cache
npm install with no cache and no package-lock.json
npm install with no cache, no package-lock.json and --package-lock-only option
npm install with no cache and package-lock.json
npm install with up-to-date cache and no package-lock.json
npm install with up-to-date cache, no package-lock.json and --prefer-online option
npm install with up-to-date cache, no package-lock.json and --package-lock-only option
npm install with up-to-date cache and package-lock.json
npm install with up-to-date cache, package-lock.json and --prefer-online option
npm update for the same scenarios as npm install

Each of the scenarios will be run using the public https://registry.npmjs.org registry as well as my local Nexus registry.

package.json

In order to run each scenario, there must be a package.json file present. For this analysis, we will use the following package.json file. This is a real package.json file, that I've used when creating a simple demo of Recharts library.

{
  "name": "recharts-random-data",
  "version": "0.1.0",
  "private": true,
  "dependencies": {
    "@testing-library/jest-dom": "^5.17.0",
    "@testing-library/react": "^13.4.0",
    "@testing-library/user-event": "^13.5.0",
    "react": "^18.2.0",
    "react-dom": "^18.2.0",
    "react-scripts": "5.0.1",
    "recharts": "^2.9.0",
    "html2canvas": "^1.4.1",
    "web-vitals": "^2.1.4",
    "typescript": "^4.0.0"
  }
}

You may notice dependency on typescript version ^4.0.0, which is not really needed, it is only a transitive dependency of other packages. However, without this dependency, npm install installs typescript version 5.x and the following npm ci complains, that the generated package-lock.json is not in sync with the package.json. This seems to be a bug in npm, but as a workaround, the typescript dependency can be explicitly added.

The package.json generates node_modules directory with the following contents:

Total number of packages: 1,507 (as reported by npm install)
Total number of unique packages (excluding version): 1,168
Total number of unique packages (including version): 1,317
Total number of files: 41,087
Total number of directories: 5,240
Total file size: 258 MB
Total used disk space: 387 MB

Observed metrics

For each scenario, we will observe the following metrics:

Number of HTTP requests sent to the registry for each request type (package metadata and package content) and HTTP response code - see below for more details.
Total size of the downloaded data with the same breakdown as for the number of HTTP requests - see below for more details.
Number of duplicate HTTP requests - it may seem surprising, but npm really generates duplicate HTTP requests. This is probably connected with the fact, that the dependency tree is being resolved in parallel, but it still seems like a bug.
Total duration of the whole scenario
Time to generate complete dependency tree - this information is provided when running npm with --timing parameter as a single log line with the following contents: npm timing idealTree Completed in 5800ms.

As we already know from previous sections, the npm log at the http level does not provide real HTTP response codes. Additionally, it does not provide the size of the downloaded data. And it seems, that there are no other command-line options, that would provide us with this information. So we need to add additional logging directly to the npm source code. The HTTP requests are handled by the minipass-fetch package, which can be easily extended by adding the following code to its index.js source:

let totalSize = 0;

res.on('data', (chunk) => {
  totalSize += chunk.length;
  return body.write(chunk);
})
res.on('end', () => {
  console.log("minipass-fetch", url.url, res.statusCode, totalSize, headers.get("cache-control"));
  return body.end();
});

Here we calculate the total size of the download data (without HTTP headers) and log it together with the HTTP status code.

Tooling

To run the scenarios, I've created a simple Node.js script, which runs the scenarios one-by-one and for each of them performs the following:

Cleans npm cache using npm cache clean --force command.
Removes node_modules directory and package-lock.json file, which may have been created by a previous scenario.
Runs npm install, if there are cached data or package-lock.json needed for the scenario.
Cleans npm cache or package-lock.json file, if not needed for the scenario.
Synchronously runs command specific for the scenario, measures its execution time and gathers its log.
Analyzes the log to gather statistics about HTTP requests and time to generate complete dependency tree.
Saves the log as well as all metrics to files.

Complete source code is available in my GitHub repository. The code can be run using node benchmark.js.

Results

Complete results are available in a Google sheet.

If you prefer the results directly in the article, the following table contains results for each scenario for the public https://registry.npmjs.org registry.

And the same table for my local Nexus registry follows.

The results contain the following columns:

Command - npm command being run.
Cache - indicates, whether up-to-date cache data was used.
Lockfile - indicates, whether the package-lock.json file was present.
Arguments - additional command-line arguments passed to npm.
Metadata requests - count and size of HTTP requests to get package metadata split by HTTP status code (200 and 304).
Content requests - count and size of HTTP requests to get package content split by HTTP status code (200 and 304).
Duplicate requests - number of duplicate HTTP requests, i.e. requests, that get the same data as a previously sent request.
Dep. generation time - time to generate complete dependency tree in seconds. Each scenario was run just once, so this time should be taken cautiously.
Total time - total time of the scenario in seconds. Each scenario was run just once, so this time should be taken cautiously.

Observations

The following list summarizes observations based on the results presented above:

Metadata is huge - One may expect package metadata to be smaller than package contents. The results show, that exactly the opposite is true with package content size being about 53 MB, while package metadata size being about 77 MB (or 81 MB for Nexus). This should be no surprise since metadata contains all package versions, which can easily be more than a thousand as evidenced above for the react package.
Number of HTTP requests needed to install packages is huge - When starting with no package-lock.json and no cache (or with the --prefer-online option), there are about 2800 requests generated. And even though npm runs them in parallel (15 by default), the requests take some time depending especially on your bandwidth and latency.
Dependency tree generation is slow - The time to generate complete dependency tree is about 6 seconds even when all the metadata is cached. For 1,507 packages in total, this seems to be quite slow. And this is one of the pain points, that other package managers try to address.
Total metadata size with Nexus is higher than with the public registry - I suppose, that the metadata of the public registry is gzipped with maximum compression and then stored on a CDN to be available for end users. Nexus itself supports no metadata compression, so I configured dynamic compression at a reverse proxy. Due to its nature, this compression is probably not as efficient as the one-off compression used by the public registry.
Missing support of HTTP 304 by the public registry - As already explained above, the public registry does not support HTTP 304 status code. As a result, when you ask for up-to-date data using the --prefer-online option, be prepared to download everything - complete package metadata as well as package content. On the other hand, with Nexus and up-to-date local cache, the traffic is low, because only HTTP headers are transferred.
--prefer-online tries to fetch not only up-to-date package metadata but also package content - This does not make sense in most cases, because the package content should not be modified. If a package content needs to be modified, new package version should be published instead.
Many duplicate requests - The number of duplicate requests is quite high, especially in some scenarios. For example when using the --prefer-online option with no package-lock.json and the public registry, there are 611 duplicate requests out of 3,112 requests in total, which is about 20%. In this case, the duplicate requests also result in the increase of transferred data from 77 MB to 106 MB.
npm update behaves the same as npm install without package-lock.json - This confirms the principles of npm update described above.
npm ci behaves the same as npm install with package-lock.json - This confirms the principles of npm ci described above.

Key takeaways

Package metadata is huge, so be prepared to download a lot of data when installing packages.
Check the caching headers and HTTP 304 support of your npm registry to understand, how your packages are cached and refreshed.
When you need to install up-to-date packages, npm install with --prefer-online option can be used. However, this approach tries to refresh package metadata as well as package content. A better approach might be to use npm install --package-lock-only --prefer-online to generate up-to-date package-lock.json and then use npm install --prefer-offline to install packages based on the package-lock.json file. This way, the package content is only downloaded, when it is not present in the cache.