Abdelrahman Farag

Posted on May 22 • Edited on May 25

We analysed 396 breaking dependency releases. Here's what they have in common.

#npm #devops #javascript #opensource

GitHub “Finish-Up-A-Thon” Challenge Submission

Renovate opened 23 upgrade PRs in your repo this week. One of them will break your CI. You just don't know which one yet.

That's the problem DepCast solves.

The idea

Most dependency tooling tells you what changed — semver, changelogs, release notes. None of it tells you how risky that change is before you merge.

DepCast computes a Compatibility Risk Score (CRS) for any npm release and rates it SAFE / WAIT / AVOID:

CRS = w₁·V(r) + w₂·E(r) + w₃·D(t) + w₄·H(m)

Signal	What it measures
V(r)	API surface volatility — fraction of exported symbols removed
E(r)	Downstream exposure — normalised weekly download count
D(t)	Observed failures — GitHub issues opened in first 24h post-release
H(m)	Maintainer history — SIR propagation R₀ from prior releases

Try it in 30 seconds

npx depcast-check --package chalk --version 5.0.0

Output:

DepCast CRS Check
-----------------------------------------------
Package:  chalk@5.0.0  (prior: 4.1.2)
-----------------------------------------------
V(r):  0.000  [....................]  API volatility       pattern C
E(r):  0.611  [############........]  Downstream exposure  (439M weekly downloads)
D(t):  0.000  [....................]  Observed failures    (0 issues/24h)
H(m):  0.030  [#...................]  Maintainer history   (R0=1.162)
-----------------------------------------------
CRS:   0.186   SAFE
-----------------------------------------------
Recommendation: Release looks safe. Proceed with publish.

With a GitHub token you get the live D(t) propagation signal:

npx depcast-check \
  --package moment \
  --version 2.0.0 \
  --github-token $GITHUB_TOKEN

Drop it into GitHub Actions

- name: DepCast compatibility risk check
  run: |
    npx depcast-check \
      --package ${{ env.PACKAGE_NAME }} \
      --version ${{ env.PACKAGE_VERSION }} \
      --threshold 0.60 \
      --fail-on avoid \
      --github-token ${{ secrets.GITHUB_TOKEN }}

Exit code 1 blocks the publish. Exit code 0 lets it through. No config files. No setup. One workflow step.

The research behind the score

The weights aren't guessed — they come from an empirical study of 396 confirmed breaking releases across three ecosystems, fitted via logistic regression.

Validation: AUC-ROC = 0.853 on a held-out set of 346 releases (306 breaking + 40 non-breaking controls).

Cross-ecosystem R₀ comparison

We fitted a SIR epidemiological propagation model to every release, measuring how fast a breaking change spreads through the dependency graph.

Ecosystem	n	R₀ median	Pattern C	First issue (median)
npm	306	1.44	62.4%	1.0h
PyPI	25	0.99	24.0%	1.1h
pub.dev	25	10.3	72.0%	1.1h

PyPI is sub-critical (R₀ < 1): breakage is self-limiting. The pip ecosystem's explicit pinning culture absorbs shocks.
npm is moderately super-critical: one breaking release propagates to ~1.4× as many downstream consumers each generation.
pub.dev is massively super-critical (R₀ ≈ 10): Flutter's null-safety migration forced a simultaneous ecosystem-wide update wave, creating a near-perfect propagation storm.

The Pattern C finding

Here's the surprising one: 62.4% of breaking npm releases have V(r) = 0.

That means the API surface looks identical between versions — no exported symbols removed, no type signature changes. Pure semver-based tooling scores these as safe. They are not.

These are Pattern C releases: breaking changes hidden in behaviour, not surface. The only signal that catches them is D(t) — the live issue rate after the release goes out. This is why the propagation signal matters, and why we built the aggregator.

The top AVOID-rated releases in the dataset

Package	Version	CRS	Why
glob	9.0.0	0.631	Complete API surface removed + high issue rate
semver	7.0.0	0.622	718M weekly downloads — highest E(r) in dataset
moment	2.0.0	0.580	D(t)=0.395 — 39.5% Dependabot rejection rate

The Renovate integration

If you use Renovate, you can run DepCast as a consumer-side gate — scoring every upgrade PR Renovate opens before you merge it.

A PR to add DepCast to the Renovate community tools list is currently open: renovatebot/renovate#43563.

The consumer gate composite action is in packages/depcast-consumer/ — it labels PRs with the CRS rating and blocks AVOID-rated upgrades before they reach your main branch.

What's next

depcast-check@1.0.0 is live on npm — install it now.
The full paper (7 pages, IEEE format) is on Zenodo: 10.5281/zenodo.20361569
Submitting to MSR 2027 (Mining Software Repositories).
Phase 5.6: reaching 500 opt-in repos to build a statistically significant live D(t) signal. If you add the consumer gate to your repo, you help make the model better for everyone.

GitHub: github.com/ahafarag/depcast
npm: npm install -g depcast-check
Preprint: doi.org/10.5281/zenodo.20361569

DEV Community