You don't need a significant portion to spoof for the metric to be useless.
Let's say you need a package, you go look at exxpress. It has 30M downloads, therefore it's probably the popular package you wanted. So you're good to go with npm install exxpress right?
I agree... You could argue the same with security:
I can create an exploit in an npm package -> everyone does -> npm is fundamentally unsafe.
That is kind of true, but no one stops using npm for this reason. I guess the same goes for every metric. Who guarantees you that a project github stars dont come from a clickfarm ?
When using a npm package, you're trusting its author, to some extend.
(This is fun to see that npm doesnt even try to protect itself against this, though)
That's not what the post is getting at (to my reading). The equivalent would be, I think:
There happens to be wild fluctuations in the number of exploits accidentally appearing in npm packages due to cosmic rays -> I can demonstrate getting an exploit into an hyperbolic number of npm packages to prove a point -> npm is fundamentally unsafe.
It's talking about how the metric is useless even in telling you how many unique users downloaded a package, or how often something caches it or runs a build job.
In this context however, the "caching" part of that description is rather noteworthy as it means the majority of downloads from yarn users won't contribute to download counts on NPM due to cache hits.
Slight correction - in 2017 we switched from a Cloudflare caching to a simple CNAME to the npm registry. So downloads via Yarn are counted just like any download from npm.
While I've no doubt that NPM could create a popularity metric that aggregates a number of different attributes of a package (npms.io has already done it)
AFAIK the popularity metric that npm uses is from npms.io 😅
The source I have is that I worked at a competitor when that change happened and was friends with folks at npm at the time. You're correct that there's apparently now deviation, and I'm not sure what that is - if npm continued using the original scoring and npms moved on, if npm moved on, or something else.
the funny part is you dont even need to try to inflate it. just publish a version and mirrors + scanners give you ~100+ downloads for free. publish 5 versions in a week and suddenly your package looks "popular"
it bugged me enough that I built something to at least partly undo it: humanloads.dev
it removes the spike every new version gets and the flat bot traffic, then shows real installs as a range. not perfect (cant catch someone looping npm install on purpose like you did lol) but for normal packages the difference is pretty eye opening
I don't get the logic here
Your argument is based on your exploit, which makes an illogical jump
I can spoof downloads -> Everyone spoofs -> the metric cannot be trusted
Unless you can proof a significant portion of users spoof (like in appstore reviews), then the argument is moot
You don't need a significant portion to spoof for the metric to be useless.
Let's say you need a package, you go look at exxpress. It has 30M downloads, therefore it's probably the popular package you wanted. So you're good to go with
npm install exxpressright?Sorry if I've made this unclear, there are two main points I'm trying to emphasise. Just in case you missed it 👇
In terms of the exploit side of things, my personal opinion is that data which can be manipulated to this degree shouldn't be given any weight.
Edit: I've updated the conclusion to remove the emphasis on the latter point - hope that clears up any confusion!
I agree... You could argue the same with security:
I can create an exploit in an npm package -> everyone does -> npm is fundamentally unsafe.
That is kind of true, but no one stops using npm for this reason. I guess the same goes for every metric. Who guarantees you that a project github stars dont come from a clickfarm ?
When using a npm package, you're trusting its author, to some extend.
(This is fun to see that npm doesnt even try to protect itself against this, though)
That's not what the post is getting at (to my reading). The equivalent would be, I think:
There happens to be wild fluctuations in the number of exploits accidentally appearing in npm packages due to cosmic rays -> I can demonstrate getting an exploit into an hyperbolic number of npm packages to prove a point -> npm is fundamentally unsafe.
It's talking about how the metric is useless even in telling you how many unique users downloaded a package, or how often something caches it or runs a build job.
Slight correction - in 2017 we switched from a Cloudflare caching to a simple CNAME to the npm registry. So downloads via Yarn are counted just like any download from npm.
Thanks for this! Fixing this now 👍
Edit: Removed mention of yarn in registries section.
AFAIK the popularity metric that npm uses is from npms.io 😅
Do you have a source for this?
Looking at the package on npms.io it has a much lower popularity rating (3% compared to 14% on NPM).
Edit: just to clarify, I also had this assumption but assumed I was misremembering after seeing the difference.
The source I have is that I worked at a competitor when that change happened and was friends with folks at npm at the time. You're correct that there's apparently now deviation, and I'm not sure what that is - if npm continued using the original scoring and npms moved on, if npm moved on, or something else.
same thing happening with my package .
npmjs.com/package/viwerjs-ang
its rapidly increasing downloads but i can't able to belive in that statistics.
this post lives rent free in my head honestly 😄
the funny part is you dont even need to try to inflate it. just publish a version and mirrors + scanners give you ~100+ downloads for free. publish 5 versions in a week and suddenly your package looks "popular"
it bugged me enough that I built something to at least partly undo it: humanloads.dev
it removes the spike every new version gets and the flat bot traffic, then shows real installs as a range. not perfect (cant catch someone looping npm install on purpose like you did lol) but for normal packages the difference is pretty eye opening
Very interesting article!
I would've thought NPM would at least check the IPs 🤷♀