A canary that re-benchmarks npm, pnpm, yarn, and dep every time one of them ships

Package manager benchmarks have a shelf life of about one release. Someone runs npm install five times on their laptop, screenshots a bar chart, publishes a blog post — and three weeks later npm ships a faster tarball extractor, pnpm changes its default linker, and the chart is quietly wrong forever.

While building dep, I wanted the opposite: a benchmark that cannot go stale, because it re-runs itself whenever any of the tools it measures publishes a new version. That became depjs/canary — a repository whose front page is a results table rewritten by a bot after every run.

This post walks through how it works and, just as importantly, what I did to keep the comparison honest.

What it measures

Every triggered, a check job polls the npm registry for the latest versions of npm, pnpm, @yarnpkg/cli-dist (yarn berry), and dep, and diffs them against a committed versions.json. If nothing changed, the run ends there — scheduled runs are cheap no-ops most of the time. If any of the four released, the full matrix fires:

{npm, pnpm, yarn, dep} × {react, next, express, vite, jest}

Each cell measures the same fixture across four scenarios — every combination of {cold, warm} cache × {without, with} lockfile:

1. cold cache, no lockfile — a from-scratch install; this also generates the lockfile and warms the cache for the scenarios that follow
2. warm cache, lockfile — the CI-like fast path
3. cold cache, lockfile — resolution skipped, downloads still needed
4. warm cache, no lockfile — resolution needed, downloads cached

Each scenario runs 5 times and the median is kept, so one slow registry round-trip or a noisy runner neighbour can't skew a published number.

The fixtures pin nothing — every dependency is "*":

{
  "dependencies": {
    "react": "*",
    "react-dom": "*"
  }
}

That's deliberate. It means the canary doesn't just exercise the newest package managers; it exercises the newest publish of react, next, express, vite, and jest too. If any of them ships something that breaks installation, the canary catches that as well.

It's a canary first, a benchmark second

Timing alone doesn't tell you the install actually worked. A package manager can be blazingly fast at producing a broken node_modules. So after the timings, each fixture runs a smoke test that loads the installed package for real:

const express = require('express')

const app = express()
assert.equal(typeof app.listen, 'function')

The react fixture renders markup with react-dom/server, the jest fixture runs the actual jest bin, and so on. A cell in the results table only counts if the thing it installed can do its job.

The self-updating README

After every matrix run, a publish job collects the per-cell result JSONs, renders the comparison table, and rewrites the section between two markers in the README with a bot commit. The front page always shows the most recent run — including failures. A red cell gets published just as prominently as a green one.

There's one small trick I like here: the same commit advances versions.json, but only when the whole matrix passed. If a release fails, versions.json stays behind — so the next scheduled poll sees the "new" version again and re-runs the matrix. A broken release keeps getting retried, and keeps failing loudly on the front page, until it's actually fixed. Failure isn't a one-time event you can miss; it's a state the repo stays in.

Keeping the comparison fair

This is the part I sweated over most, because canary lives under the depjs org — it benchmarks the tool it belongs to. That's exactly the situation where you should be most suspicious, so the methodology is designed to be inspected:

Every tool produces the same output. Yarn berry runs with nodeLinker: node-modules, so all four managers build the same layout and the same smoke test applies to all of them.

Behavioural differences are levelled, not hidden. pnpm ≥ 11 hard-fails on unapproved build scripts, so it gets --dangerously-allow-all-builds to match what npm, yarn, and dep do by default. npm gets --no-audit --no-fund to strip work no other tool performs.

Cold means cold. Caches and stores live under the runner's temp directory, and cold scenarios wipe the npm cache, the pnpm store, and yarn's global folder before every repetition — not just before the first one.

Disadvantages of my own tool are printed, not smoothed over. dep keeps no cache by design, so its "warm" numbers measure the exact same work as its cold ones. The table says so, and dep's warm cells don't get the discount every other column enjoys.

What the numbers don't prove

An honest benchmark needs an honest limitations section:

• Each cell is a separate CI job. The four managers in a row ran on different shared runners, at slightly different times, over different network conditions. The median-of-5 protects against noise within a job, not between jobs. Compare tools within a row as a strong signal, not a lab measurement.
• Absolute numbers are meaningless. 1.9s on a shared GitHub runner is not 1.9s on your laptop. Only the relative shape carries information.
• Floating "*" dependencies cut both ways. They're what makes the canary catch real-world breakage, but they also mean two columns could theoretically resolve different dependency trees if a library publishes mid-run.

Run a cell yourself

The whole measurement is one bash script, so you can reproduce any cell locally:

$ RESULT_FILE=/tmp/result.json bash scripts/canary.sh dep react
$ node scripts/report.mjs /tmp

Swap dep for npm, pnpm, or yarn, and the same script measures the same four scenarios with the same prep between repetitions.

Takeaways

If you maintain anything whose performance you want to claim publicly, I'd recommend this shape over a one-off blog-post benchmark:

1. Trigger on releases, not on a calendar — poll cheaply, run the expensive part only when something actually changed.
2. Publish failures as loudly as wins — and make a failing state persist until it's fixed.
3. Verify, don't just time — a smoke test that loads the artifact turns a benchmark into a canary.
4. Write down what would make your own numbers wrong — the limitations section is what makes the rest believable.

The repo is at github.com/depjs/canary — the front page has today's numbers, whatever they are.