DEV Community

Cover image for Left-pad incident explained: how 11 lines of JavaScript broke npm
Nikoloz Turazashvili (@axrisi)
Nikoloz Turazashvili (@axrisi) Subscriber

Posted on

Left-pad incident explained: how 11 lines of JavaScript broke npm

#opensource #node #javascript #npm

The left-pad incident is the npm outage every JavaScript developer has heard of and few have read the paperwork for. On March 22, 2016, one developer unpublished 273 packages from npm, one of them an eleven-line function called left-pad, and builds of Babel, Atom and "many thousands of projects" started failing at "hundreds of failures per minute", in npm's own words. It took two and a half hours and an unprecedented restore from backup to fix. The mechanism behind it, transitive dependencies resolved live from a registry anyone can delete from, is still how most of us build software.

TL;DR

  • A naming dispute with Kik, the messaging app, over an npm package called kik ended with npm handing the name to Kik. Its author, Azer Koçulu, then unpublished all 273 of his packages.
  • One was left-pad: eleven lines of code, downloaded about 2.5 million times the month before, pulled in by Babel and Atom through a package called line-numbers that pinned exactly version 0.0.3.
  • A stranger republished left-pad within ten minutes, and nothing changed, because nothing asked for his version. npm restored the original 0.0.3 from backup, which its registry normally cannot do. Total disruption: 2.5 hours.
  • npm's postmortem the next day: "Unrestricted un-publishing caused a lot of pain" and "We dropped the ball". A week later it shipped a 24-hour unpublish window, later widened to 72 hours.
  • The lesson for today: commit your lockfile, and treat eleven lines as a paragraph you write yourself.

What is left-pad?

left-pad pads a string on the left to a given length. This is the whole of index.js in version 0.0.3, verbatim from the registry tarball:

module.exports = leftpad;

function leftpad (str, len, ch) {
  str = String(str);

  var i = -1;

  ch || (ch = ' ');
  len = len - str.length;


  while (++i < len) {
    str = ch + str;
  }

  return str;
}
Enter fullscreen mode Exit fullscreen mode

Eleven lines of code, seventeen with the blanks. According to npm, via The Register, it had been downloaded 2,486,696 times in the previous month. Nearly all of those were not people choosing left-pad. They were machines installing something that installed something that needed it.

How a trademark email started the left-pad incident

The fuse was a package name. Kik's head of messenger, Mike Roberts, later published the full email thread, so the sequence is on the record.

On March 11, Kik's patent agent asked Azer to rename his kik package. Azer said no. Sixty-six minutes later came the line that made the story: "our trademark lawyers are going to be banging on your door and taking down your accounts". Azer answered with a price, "$30.000", and Kik emailed npm support the same day. Kik's own note on the thread says "Bob is our patent agent, not a lawyer", and that Kik had "decided to use a different name for an upcoming package … even when we were told we could have the name Kik".

On March 18, npm's CEO Isaac Schlueter ruled for Kik: "most users who would come across a kik package, would reasonably expect it to be related to kik.com". On March 20 Azer wrote: "I want all my modules to be deleted including my account". Two days later he did it himself, and explained why in a post titled I've Just Liberated My Modules: "NPM is someone's private land where corporate is more powerful than the people".

npm stood by the naming call in its postmortem: "It was abrupt unpublishing, not our resolution policy, that led to yesterday's disruptions."

Left-pad timeline (UTC)

When What happened
Mar 11 Kik's patent agent asks for the kik name; "lawyers banging on your door"; Azer asks $30,000; Kik emails npm
Mar 18 npm's CEO transfers the name to Kik
Mar 20 Azer asks for all his modules to be deleted
Mar 22, ~21:30 273 packages unpublished; builds start failing, hundreds per minute
Mar 22, 21:42 Cameron Westland publishes a functionally identical left-pad 1.0.0
Mar 22, 23:03 npm's CTO Laurie Voss: "we are un-un-publishing it"
Mar 22, 23:55 The original 0.0.3 is back, restored from backup
Mar 23 npm's postmortem; Kik publishes the emails
Mar 29 npm's new unpublish policy

The failures start "shortly after 2:30 PM Pacific". Within ten minutes Cameron Westland republished the function as 1.0.0 (the registry records 21:42 UTC), and the builds kept failing. Then npm's CTO tweeted:

Laurie Voss (@seldo), Mar 22, 2016:

npm called the restore "unprecedented": "re-publishing isn't otherwise possible". It was done at 4:55 PM Pacific, two and a half hours after the first failures.

Why did left-pad break Babel? Transitive dependencies

The detail most retellings skip is why a republished left-pad didn't fix anything. Babel and Atom did not depend on left-pad. Per npm's postmortem, they pulled it in through line-numbers, and line-numbers asked for exactly 0.0.3. A simplified sketch of the chain:

your project
└── babel (or atom)
    └── line-numbers
        └── left-pad  "0.0.3"   <- exact version, not a range
Enter fullscreen mode Exit fullscreen mode

An exact pin means only that one version satisfies the dependency. Westland's 1.0.0 was the same function under a different number, so the resolver ignored it and kept returning 404 for 0.0.3. The only fix was to bring back the exact bytes, which is why npm restored from backup rather than waiting for a new release.

Kik got caught in the same chain. From Roberts' post: "our builds started failing because we use … JSCS. Through a long chain of dependencies, JSCS relied on left-pad@0.0.3". The company that asked for the name broke its own builds on the result.

Three properties of the ecosystem made this possible, and the video counted them the same way:

  1. Any author could delete any version, instantly. npm did not check who depended on a package before removing it.
  2. Dependencies are transitive and resolved live. Babel knows line-numbers, not left-pad, and the whole tree is fetched again on every install.
  3. No lockfile by default in 2016. npm shrinkwrap existed but was opt-in. package-lock.json arrived by default with npm 5 in May 2017. Until then, every CI run asked the registry again what the tree should be.

npm's unpublish policy: what changed after left-pad

npm's postmortem, published the next day, reads like a good postmortem should: "Unrestricted un-publishing caused a lot of pain", "npm needs safeguards", "If these had been in place yesterday, this post-mortem wouldn't be necessary", "We dropped the ball", and "It took us too long to get you this update".

On March 29 npm published the new unpublish policy:

  • You can unpublish a version only if it is less than 24 hours old.
  • Older than that, you contact support, and support checks who depends on it.
  • A package removed completely is replaced by a security placeholder package.

The policy page itself notes it was updated on January 30, 2020, when the window became 72 hours. The principle has not changed: after the grace period, removing something other people build on is a conversation with a human.

The blame split

In the episode I split the blame the way I do for every Postmortem, from the sources:

  • npm, Inc., 60 %. The registry let any author delete any version with no dependency check. npm said so itself: "Unrestricted un-publishing caused a lot of pain". It also made the name call that lit the fuse.
  • Kik, 25 %. A patent agent, "not a lawyer", promising lawyers at the door over a package name Kik had already decided not to use, escalated to the registry the same afternoon.
  • The dependency habit, 15 %. Everyone who installed eleven lines instead of typing them, including Kik.

Azer is not on the list. He did what the registry allowed, publicly, and explained why.

How to protect your builds from the next left-pad

The npm-specific hole is closed, but the pattern, a build that depends on something you don't control resolved at build time, is the same shape as every npm supply chain problem since. Practical takeaways:

  • Commit the lockfile (package-lock.json, yarn.lock, pnpm-lock.yaml) and install from it in CI (npm ci fails if the lockfile and package.json disagree, instead of resolving a new tree).
  • Know your transitive tree. npm ls left-pad shows every path by which a package reaches you. Most of your dependencies are ones you never chose.
  • Cache or mirror what you build from. A registry outage or a deletion should cost you nothing for packages already in your cache.
  • Write the small ones yourself. Modern JavaScript has String.prototype.padStart built in, so left-pad's job is now one call: 
'5'.padStart(3, '0'); // "005"
Enter fullscreen mode Exit fullscreen mode

The Hacker News thread that best captured the mood was David Haney's NPM and Left-Pad: Have We Forgotten How to Program?, at 1,725 points. The honest answer is no. We had forgotten that every dependency is someone else's decision.

Verdict: SHIP IT

The Postmortem verdict judges the response to the incident, and npm's response earns SHIP IT. It restored the original version from backup in two and a half hours, published a postmortem the next day that said "we dropped the ball" without hedging, and within a week shipped a rule that is still the rule, tightened in 2020. The Monday line from the episode stands: commit your lockfile, and if a function is eleven lines, it is a paragraph you write yourself.

FAQ

What was the left-pad incident?
On March 22, 2016, Azer Koçulu unpublished 273 packages from npm, including left-pad. Babel, Atom and thousands of projects depended on it indirectly, and their builds failed for about two and a half hours.

Why did Azer Koçulu unpublish left-pad?
npm gave his kik package name to Kik after a trademark dispute. He responded by removing all of his packages.

Can you still unpublish an npm package?
Only within a time window. The 2016 policy allowed 24 hours; npm's policy page says it was updated to 72 hours in 2020. After that, npm support checks dependents.

Does a lockfile prevent a left-pad outage?
It stops your tree from changing between installs. A package deleted from the registry still needs a cache or mirror to install, which is why npm now blocks late unpublishing.

Sources

This article expands on an episode of **The Daily Diff, a five-minute daily video on what shipped and what broke in tech.
Watch the episode · Subscribe on YouTube · the written diff lands in your inbox every morning at thedailydiff.dev.

Top comments (0)