Beta Stories — Episode 09
The promise of the modern package ecosystem was a kind one: you do not have to write everything yourself. Stand on the shoulders of giants. Reuse, do not reinvent. Anyone maintains it.
The reality, measured this morning on a fresh laptop, is the avalanche this episode is named for.
The Reality, in Two Numbers
npm install express on a clean directory pulls Express 5.2.1, declares 28 direct dependencies, resolves a total of 65 packages across the dependency tree, and produces a node_modules of 3.6 MB. That is the smaller end of the modern web.
npx create-next-app@latest with the recommended defaults (TypeScript, Tailwind CSS, ESLint, App Router) creates a project whose package.json declares 11 packages, resolves a transitive tree of 644 packages, and writes a node_modules of 463 MB. The application, at this stage, renders a single page that says "Welcome to Next.js".
Six hundred and forty-four pieces of someone else's work to render eighteen characters of text. Each package authored, in principle, by a stranger; in practice, sometimes by a small group of strangers; in a handful of unpleasant cases, by an account that has been waiting to be useful.
For comparison: the same minimal HTTP responder in Rust uses hyper (the lower-level HTTP foundation) or axum on top of it, and keeps its dependency tree small and explicit in Cargo.toml, where every crate is signed off in writing before it lands. The Go equivalent uses the standard library's net/http, pulls zero external dependencies, and builds, statically linked, to a single 7 MB binary. FreeBSD ships a base userland audited as one source tree, and a ports collection where every port has a named maintainer and a fully declared dependency graph (nginx and friends arrive that way, complete with provenance). All three have existed for years. None asks the team to import a stranger's recursion.
The Mechanism
The phrase that does the damage, repeated almost reflexively, is "we do not maintain it; the upstream maintainer does."
Which means, when one reads it carefully: nobody on the team has read the code. Nobody has reviewed the commit history. Nobody has audited the build script. Nobody has checked who has commit access. Nobody has asked what the package was last week, or what it will be next week. The entire audit duty has been outsourced, in writing, to a name on an npm registry page.
A 644-package install is 644 outsourced audits. The team that ships it has, by default and in good faith, agreed that some other unnamed group will do the reading on their behalf. The other unnamed group, when one goes to look, is mostly volunteers who themselves have not been paid to read the code below them either. The audit duty has been outsourced so many times that nobody is left holding it.
This is the Beta Stories mechanism: software gets worse not by mistakes, but by accumulation. The decay is in the count. Every package added is a piece of code that the team has decided not to read.
The Audit That (Barely) Happened
In January 2021, an account named Jia Tan was created on GitHub. It submitted small, useful patches to xz-utils, the compression library that ships in essentially every major Linux distribution and underlies a great many compressed-archive operations across the Unix world. The patches were good. The maintainer at the time, Lasse Collin, accepted them, as one accepts good patches.
By the summer of 2022, three accounts (Jia Tan, Dennis Ens, Jigar Kumar) were active in the xz-utils mailing lists and issue tracker, applying coordinated pressure on Lasse Collin to add an additional maintainer with commit rights. Lasse, by his own admission, was burnt out, unpaid, and had been carrying the project alone for years. Jia Tan was eventually granted those rights.
In February and March 2024, Jia Tan committed the backdoor. The payload was inserted into xz-utils 5.6.0 (released 24 February) and 5.6.1 (released 9 March), hidden inside test fixtures and triggered through the autotools build system in a way that left the source tarball pristine to a casual reader. The build, when invoked in a particular way that distribution maintainers happened to invoke it, linked malicious object code into liblzma. liblzma was, in turn, linked into sshd on systemd-based distributions via the systemd-notify integration. The result was a remote-code-execution backdoor in OpenSSH, triggerable by any client with the right key. CVSS 10.0.
By 29 March 2024, the backdoored versions had already reached Debian unstable, Fedora 40 rawhide, openSUSE Tumbleweed, Ubuntu testing, and Kali Linux. They were days, in some cases hours, away from rolling into stable releases that would have been on hundreds of millions of servers.
On 28 March 2024 (the public posting was on the 29th), Andres Freund, a PostgreSQL maintainer at Microsoft, noticed that his SSH login on a Debian unstable system was taking approximately 500 milliseconds longer than usual. He had recently been benchmarking Postgres builds and was attentive to small latency drifts. He ran the login under valgrind, found memory-access errors pointing at liblzma, traced the chain back through the autotools build, found the obfuscated payload in the test fixtures, and posted the disclosure to the oss-security mailing list that night.
The internet was saved by one engineer noticing half a second of latency he did not expect.
The Signal
Two and a half years of patient social engineering against an unpaid maintainer. A chain of distribution maintainers signing off on routine version bumps. CI pipelines, test suites, code-review tools, all returning green. SBOM generators producing clean reports. The audit duty had been outsourced so many times that nobody was holding it.
The signal one watches for, after XZ, is not "is there malicious code in your dependencies." That signal is impossible to read directly. The signal is who is doing the reading. If the answer is "the upstream maintainer," ask: who is the maintainer, what is their funding, what is their burnout level, who has commit access, what has changed in the past six months. If those questions cannot be answered for a package one ships to production, one is in the avalanche zone.
The Boring Counter-Move
The counter to dependency-avalanche is not "audit every package", which is impossible at 644 packages. It is fewer packages. Each one read, each one chosen, each one with a known maintainer model.
The Go standard library is one approach: a curated, audited, batteries-included foundation that removes 80 percent of the reasons one would reach for a third-party package in the first place. Rust takes the opposite end of the same idea: a deliberately small standard library plus a Cargo.toml that any reviewer can read in a single sitting, with every dependency declared in writing and every transitive crate visible in Cargo.lock. The FreeBSD base system is a third: a coherent userland built and audited as a single project, where what ships in /usr/sbin/sshd is the same source tree as what ships in /usr/bin/find and /usr/bin/awk, maintained by the same project, released together. The Python standard library, before the wheel-and-PyPI culture took over, was something similar. So was the venerable Unix toolkit.
None of this scales to "build any product, in any language, in any week". It scales to "build a sustainable system, with a known maintenance posture, over decades". The Beta Stories question is whether one has been paying for the first while pretending it is the second.
The Closer
XZ is the post-mortem the industry will keep referencing because it nearly worked. The patient, careful, well-funded version of the same attack will work, and one will not notice in time. The reason is not that the security tools have failed. The reason is that nobody is reading the code, and nobody has been for some time.
Next time it might not be 500 milliseconds. Next time it might be 50. Next time it might be no perceptible drift at all, until the breach notification arrives in the inbox.
Read the full article on vivianvoss.net →
By Vivian Voss — System Architect & Software Developer. Follow me on LinkedIn for daily technical writing.
Top comments (0)