DEV Community

Cover image for 1,715 npm downloads and zero sales: I measured the whole funnel
SAMUELFMEDEIROS
SAMUELFMEDEIROS

Posted on Edited on

1,715 npm downloads and zero sales: I measured the whole funnel

I built a scorer that grades a coding agent 0-100 before you ship its work. Then I watched 1,715 npm downloads produce zero sales. Here is the whole funnel, measured, because the funnel is the post.

Every number below comes from the store's own SQLite (orders.db, tables instalacoes, cli_eventos, hits, orders) and from the Stripe session list, read on 2026-11-17 at 12:21 (-03). Test and probe installations are filtered out by the collector; two obvious ones are excluded, including one I created myself that logged 142 runs and would have inflated everything I am about to say.

The download number is a lie you tell yourself

Last week: 1,715 npm downloads and 365 PyPI downloads of the free package (agent-test-loop / ai-test-loop).

That reads like traction. It is not. Package downloads include CI runners, mirrors, and re-installs by the same machine. The number that means something is the one the CLI reports when it actually runs.

45 distinct installations sent at least one telemetry event. Of those, 6 ever ran an evaluation. 29 evaluation runs total, 63 error events, average score 93.7.

So: 1,715 downloads, six machines that scored anything. One of those six is almost certainly my own box — 24 runs, 20 help calls and 17 errors spread over eight days, which is what my own debugging looks like. Strip it and you have five external runs, ever.

The tool works, which is the annoying part

Average score across the runs: 93.7. Minimum 50, maximum 100.

The product is not the problem. A tool that grades an agent's output before you merge it is useful, and the scores are real project scores.

The problem is the first five minutes. Most people typed --help and stopped. And 63 error events across 15 installations happened before a single score was printed.

I assumed the drop-off was pricing. It was not pricing. Nobody reached pricing.

The checkout page has traffic and no buyers

30 days of Stripe: 610 checkout sessions, 0 payments.

Before you ask, those sessions are almost certainly not shoppers. The buy routes have an anti-bot guard that creates a real Stripe session only for a plausible browser user agent, and October showed 100 sessions in four days clustered between 3 and 5 AM. That is scanners walking links, not humans deciding.

The storefront itself: 60 human sessions in 30 days, with the owner already excluded in the classifier (there is a separate owner bucket, and I am not counting it here). Referral hosts for the window, human-classified: 898 direct, 279 self-referral, 22 Bing, 20 from my pet site, 16 Google, 13 Hacker News, 11 Telegram, 9 from my blog.

That is the whole map. There is no traffic to convert, which means every hour I spent "optimizing conversion" on this store was an hour spent polishing the rim of a bucket with no water in it.

What I got wrong, in order

  1. I treated downloads as audience. Downloads count builds, not people.
  2. I treated checkout sessions as intent. With an anti-bot guard they are mostly scanners.
  3. I fixed CTAs before I fixed the first five minutes. A better button on a page nobody reaches changes nothing.
  4. I wrote for a channel whose lifetime total is 468 views across 25 posts. That is not a distribution channel, that is a diary.

What the numbers prescribe

The bottleneck is not the offer, the price, or the checkout. It is that almost nobody who installs reaches a first score, and nobody who reaches a score is asked for anything.

Two fixes, in this order, and neither one is a marketing campaign:

Fix the first run. 63 errors across 15 installations, before any score. That is the highest value per hour in the whole funnel, because the person is already holding the package and already willing.

Then ask. A run that ends at "score 93" and says nothing else wastes the one moment where the user has proof the tool did something real.

Everything else — pricing tests, landing page rewrites, a newsletter, a Product Hunt launch — sits behind those two.

If you run a coding agent

The free part is the honest part of this. It scores your agent's output 0-100 with a per-item breakdown, so you find out before your users do:

npx ai-test-loop --config loop.json
Enter fullscreen mode Exit fullscreen mode

Or inside Claude Code:

/plugin install ai-test-loop --marketplace Samuelfmedeiros/ai-test-loop
Enter fullscreen mode Exit fullscreen mode

The paid guards are separate, one-time, no subscription: MCP server preflight, screenshot diffing that does not produce false positives, long-training GPU watchdogs. Full pack, R$147.

Run the free one first. If it does not catch something real in your project within ten minutes, the pack will not either, and I would rather hear that from you than take your money.

The part I am publishing before it is fixed

I am writing this while the first-run leak is open, on purpose. Next time I post numbers, either the first-run rate moved or it did not, and the post will say which.

Sources: orders.db (instalacoes, cli_eventos, hits, orders) and the Stripe session list, both read 2026-11-17 12:21 -03. Free tool: npm ai-test-loop, PyPI agent-test-loop.

Top comments (0)