I was checking Twitter today and came across this πΆπ₯πΆspicyπΆπ₯πΆ post:
And when I read a spicy post on tech - especially one on Node.js - I'm often compelled to respond, and this is no exception. I definitely disagree with this take, and as someone who has published one of the most notoriously spammy modules ever to bless NPM, I might even be an expert.
As usual: I have no beef with Ayane, and in fact I think if I did my job right while writing this post that it'll be clear that I want to encourage them to write even more takes - which, I like my takes how I like my coffee. <_<;
The Argument for Judicious Publishing
I'd like to take a moment to summarize Ayane's arguments, not just so that readers can nod and say, "yes, Josh understood the blog he read", but also: this is a common stance among JavaScripters, and has been a common stance for as long as NPM has been around. NPM may have only recently reached a million packages, but people were just as concerned about this at the 10k packages mark.
Exhibit A, Sturgeon's Law:
90% of everything is crap.
Ayane has a similar stat in their post: "about 30% or so of those are doing the same thing" - both of these numbers are made up, but the point is fairly clear: 90% of what's on NPM is crap, and if there are a million modules on NPM then that means there are 900,000 crappy modules! That sounds really bad, right?
Let's define crappy: Ayane doesn't actually call anything crap in their post. What they do talk about is utility: "useless stuff," modules "doing the same thing [as other better modules]", abandoned modules, and "packages with no useful purpose".
They don't talk about the ramifications of this vast sea of not-very-useful modules, but if I had to guess (because I've heard it before): When you search for a package on NPM, you'll find dozens of modules to choose from, 90% of which aren't very good - old and busted, half-assed, no users or support, whichever. This means that people searching for packages have to spend more time sorting through bad packages to find the diamond in the rough.
This experience is very real! However, I think "don't publish your work" is the wrong solution to these problems.
When I Say Everything, I Mean Everything
Sturgeon's Law is quite old - according to Wikipedia, it was originally cited as Sturgeon's Revelation in the late 50s. Back in the days of yore, Theodore was talking about science fiction novels, but it's quite generalizable and applies to: science fiction movies, romance novels, comedies, self-help books, tweets, webcomics, fanfics, youtube videos, twitch streams, github repos, ruby gems, and more. This is in fact the point of Sturgeon's Law - not the 90%, but the everything. Critics derided science fiction in the late 50s as being overall crappy, just as we're deriding NPM for having low module quality. The more things change, the more they stay the same.
Barriers of Entry: Or, Why Sturgeon's Law is True Now More Than Ever
My formative personal experience with Sturgeon's Law comes not from software, but webcomics. In the mid aughts, I had grown from a child that was obsessed with newspaper comics into a teenager that was obsessed with webcomics. I even drew one fairly consistently for a good 6 months - and no, I will not be sharing the link.
Webcomics in 2005, the age of the Gamer Comic and the Sprite Comic - as well as the Sprite Gamer Comic, lord help us - were widely considered very bad. Of course, most comics are pretty bad - newspaper comics are notoriously bad and comic books are responsible for Rob Liefeld - but these comics were at least widely considered to hit most of the fundamentals more often than not, Rob Liefeld's anatomy excepted.
So what was different? The answer we in the webcomics community had at the time was barriers of entry. Before webcomics, in order for people to see what you worked on, you either had to convince a bunch of stodgy editors at a newspaper comics syndicate that old people would consistently get a light chuckle at your drawing and never ever write in, or you had to convince a bunch of stodgy editors at a comics publisher that teens would get excited about your totally extreme superhero. When costs are high people are careful to get value out of what they spend, and printing and distributing comics all cost a bunch of money.
With webcomics, you needed no such convincing: if you could afford the site hosting (as low as $0) then you could upload your comics and subject the world to them. Suddenly, an entire gaggle of nerdy teens with severe ADHD and a pirated copy of Photoshop 6 could make and upload comics after doing their homework - and suddenly, there were a lot more amateur hour comics out in this wild and crazy world.
The mechanics of NPM are similar. Decades ago, distributing software meant paying for servers, buying CD-Rs, buying postage stamps and buying ads. This meant that only software that was polished enough to justify those costs had any exposure to a wide audience. Today object storage is exceedingly cheap - you hardly need to feel bad about wasting it, I promise - and all you need to get your code out there is a free NPM account and, perhaps, a free GitHub account. In 2020, an entire gaggle of nerdy teens with severe ADHD and a text editor they didn't even have to pirate can make and upload NPM modules after doing their homework - and here we are.
Barriers of Entry are Fickle
The flip-side of bad things not getting filtered out is that good things don't get filtered out either. I mentioned newspaper comics earlier: if we were stuck getting our comics from the newspaper, we would all be convinced that comics are a huge waste of time. Because people on the business side of syndicates are extremely risk-adverse, potentially great comics would never see the light of day. For every The Far Side and Calvin and Hobbes, there are dozens of people who drew a mildly successful comic for their college paper, were unable to get a syndicate to pick it up, and had to put it on a shelf - and we're worse off for it.
Analogously: Because nobody can tell you what you can and can't upload to NPM, sometimes projects that on the face of them would seem like a bad idea - or at least redundant - turn out to be huge hits. Ayane mentions Grunt. Did you know that I thought Grunt was a terrible idea? Probably not. But I did, and I was pretty vocal about it. Imagine if I was in charge of what could and couldn't get published on NPM - there's a world where Grunt never had the opportunity to take off because it wasn't deemed useful enough to merit publishing. Or Gulp: What if Gulp never got put on NPM because Grunt already existed? Gulp as it turns out was also a huge hit, despite "reinventing the wheel".
Oh yeah? Well Define Purpose
Ayane talks a lot about modules having a purpose, a raison d'etre. In this world, a module needs to pass a test before it goes live: What problem of Ayane's is it trying to solve, and would they want to use it in the face of a dozen other packages working in the same space?
I say this a little tongue in cheek. Obviously, our intrepid author didn't deem themselves to be the arbiter of what has purpose. But if not them, then who?
In mid 2012 - very different times in the lifecycle of NPM! - we, and by "we" I mostly mean @substack, were very concerned about large utility modules, particularly underscore. In our minds, these broad modules necessarily shipped with a big, fat kitchen sink that you weren't going to need. When bundle size is at a premium, why do that to yourself? Why not let people who need a particular utility publish a targeted module on NPM, and use browserify to pull it into your bundle? A sensible position today as then.
Of course, I was quite the jokester in 2012, so I wrote a module that I'm sure will raise Ayane's blood pressure by 30 points when they learn of it:
This module made fun of grab bag modules like underscore by being the ultimate grab bag. The package.json for the module was autogenerated based on the _all_docs route on NPM - then a couchapp! - and included every module currently on NPM. The index.json then used a call to global.__defineGetter__ to lazily require any module just by referencing its name as a global variable. NPM was much smaller then - it wouldn't be tenable to make a package.json with all million NPM modules today - and this was workable, at least until Isaac discovered that starring it was a DOS vector.
Very clever! Also, very much not practical. Even then, people were absolutely furious.
Out of all the modules on NPM, this may have been the most "useless" one - and yet, it made me and a bunch of other developers quite happy:
Node is fun, because screwing around is taken seriously.
There are a bunch of modules that are quite silly (like caps-lock-script),
hoarders is one of the greatest of the silly modules.
- @dominictarr
The point here is this: this module had an explicit purpose - to make people laugh - and while it made some people angry, it made me happy, and it made a number of other people happy as well. On those merits, it was wildly successful.
Other "useless" modules on NPM are certainly more useful than Hoarders - an umpteenth logger still logs. People also publish these modules for a reason: The other (umpteen - 1) loggers failed to scratch a certain itch, or didn't play nice with their favored framework, or hell, maybe they were like me in 2005 and had an urge to create.
People should be allowed to enjoy things, and if what they enjoy is releasing open source software, they should be allowed to do that.
Discoverability
I wrote earlier of the challenge of finding a package on NPM. This general set of features - finding good things in a sea of bad - is what I call "discoverability".
This framing is the correct one to use when talking about the problem inherent in Sturgeon's Law and low barriers of entry. If 90% of everything is shit and NPM is really big, how do you find the good parts?
Going into the details of how to think about these kind of features is outside the scope of this post, but suffice it to say that this is something that software product designers think about often. My own experiences have centered around building good search; using NLP to automatically categorize and tag content; making it easy for users to tag their own content; building engagement metrics like downloads and stars and using them to detect trends; designing collaborative filtering algorithms that surface recommendations based on what you and others have downloaded and used; leveraging manually curated lists such as the "Awesome X" series; and much more.
In short, there are technical tools, user experience design paradigms and editorial techniques that we can throw at this problem, all without increasing barriers of entry.
Times Change - Or Do They?
It's true that the world of JavaScript in 2012 isn't the same as the one in 2020.
For one, there really are more packages on NPM. This does mean that finding packages can now be harder than ever, leaving us frozen in a paradox of choice. As Ayane notes, the JavaScript core APIs have gotten better too, and many a module that was hot shit back then has been deprecated and abandoned - a number of mine included.
In 2012, we were all anxious to just publish up a storm, and a big chunk of that really was because the modules we were looking for simply didn't exist. But the fact of the matter is that it was bigger than that.
What we were concerned about was having a good time, creating, and being true to what Marx called Gattungswesen. The Ruby community had a saying: "Matz is nice and so we are nice," or minaswan. Ours was jifasnif: "JavaScript is fun and so Node is fun."
Let's endeavor to keep it that way.
Top comments (2)
Loved your take on this, and no I don't mean I don't agree with your opinions, but my only alms and grudge was how NPM is not doing on some of the problems it has, but I am open for dialogue on what's the better approach for this problem - despite my suggestions being more on the authoritarian stance. I do agree taking fun out of publishing also ruins the point on what made Open Source and NPM successful as a whole, so really sorry for those who were rather mad about me.
I encourage people to do their own takes on the problem as well, and present their own solutions, after all, my point of opening such a opinion is to garner a open dialogue on what can be done on some packages, spare the joke packages (I know I said that phrase one last time because I was a bit too malicious with them).
With regard to joke modules: I'm honestly not sure if I would have written Hoarders in 2020. The person I was 8 years ago was much more likely to do something "for the lulz" than I am today.
For what it's worth, I think you got a lot of pushback, including from my post, but I definitely think you have more allies than it appears in this one. Many people have been saying the same sorts of things for many years - it's just that there isn't much content inherent in a "yeah I agree" post.
I thought your suggestion for keeping modules local was a good one, by the way. I've taken this approach the last few years with my own loggers, and I've found it deeply satisfying to work on loggers without having to worry about meeting the use cases of other users, or hell even myself in other contexts. I think this strategy is unsung and particularly relevant to logging.