I spent two months building an AI bug-fixing extension for VS Code. Extension in TypeScript, FastAPI backend, Supabase, Stripe subscriptions, GitHub Actions publishing every release. Shipped solo.
Then I spent a week telling people it had 500+ installs.
It had 4.
Nobody lied to me. I lied to myself with a dashboard, and it took an embarrassing amount of evidence to admit it. Here's exactly how it happened, because I don't think I'm the only one doing this.
The number that felt great
The VS Code Marketplace publisher dashboard has a big, friendly headline metric called Total Acquisition. Mine said 525.
So I did what anyone would do. I put "500+ installs" in my freelance profile. I put it in a proposal. I mentioned it on LinkedIn. It felt like traction — proof that the thing I built in my bedroom was landing with real people.
There was a small detail I kept not thinking about: my API logs were quiet. Very quiet.
If 500 developers had installed a tool that calls my backend to fix bugs, my server should have been busy. It wasn't. I explained that away for days — maybe they installed and forgot, maybe onboarding is bad, maybe the free tier isn't obvious.
Maybe. Or maybe the number was fake.
The three pieces of evidence
1. The public listing said something different.
My dashboard said 525. My public Marketplace page — the one every visitor sees — said 4 installs, 0 reviews.
That's the number clients see. That's the number anyone clicking my "verify my work" link would see, right next to my claim of 500+. I had been sending people to a page that contradicted me.
2. The funnel was mathematically impossible.
The dashboard showed a conversion funnel: 160 page views → 525 acquisitions. That's a 328% conversion rate.
More than three downloads per human visit. Funnels don't work that way. A number over 100% doesn't mean you're crushing it — it means the events aren't coming from the page at all.
3. The chart legend told the truth.
The acquisition graph has two series: Install from VSCode and Download from Marketplace.
Mine was almost entirely "Download from Marketplace" — raw .vsix file fetches. Real users install through the editor. Almost nobody downloads a VSIX by hand.
But you know who does? Automated scrapers and archive bots that pull every version of every extension the moment it's published. I had shipped a dozen releases. Every release triggered a wave.
The spikes in my "growth" chart lined up exactly with my publish dates. I hadn't found product-market fit. I'd found a mirror.
Fixing it: stop trusting dashboards, instrument your own truth
Vanity metrics are ambiguous by design. So I stopped arguing with the dashboard and made my backend answer the question directly.
I added one log line to my fix endpoint:
logger.info(
f"[REAL_USER] fix/auth device={device_id} ip={ip} tier={tier} lang={lang}"
)
device_id is a stable hash — sha256(key_hash)[:8] — so I can tell 10 different people apart from 1 person clicking 10 times. That distinction matters more than any total.
Now my real user count is a command, not a feeling:
grep
[REAL_USER]→ count uniquedevice=values → subtract my own device ID.
That last step is not optional. Your own testing will absolutely show up in your metrics and quietly make you feel successful. Know your own ID and exclude it.
The bug that almost fooled me twice
I deployed the logging, ran a fix, checked the logs — nothing. Zero [REAL_USER] lines. For a moment I thought I'd proven I had no users at all.
But my server was logging POST /v1/fix → 200. Requests were arriving. The tag just wasn't there.
The reason: I'd only added the logging to the keyless public endpoints. My own editor had an API key configured, so my traffic went to the authenticated route — which had no logging.
Two lessons in one bug:
- Instrument every path, not the one you happened to be thinking about. Partial instrumentation is worse than none, because it looks like data.
- Test your telemetry the same way you test your code. "I added logging" and "logging works" are different claims.
What the real numbers actually told me
Once it was honest, the picture was brutal and useful:
- Real users: ~0 (just me)
- The public listing: 4 installs
- Nobody had ever used the keyless free trial
That reframed my entire problem. I'd been optimizing onboarding — first-run experience, auto-demo, walkthrough — assuming people were installing and bouncing.
They weren't bouncing. They were never arriving. My problem was never activation. It was distribution. Nobody knew the thing existed.
You cannot fix the right problem while you're looking at the wrong number.
Then one real person showed up
A few days later, this appeared in my logs:
[REAL_USER] fix/public device=624f526c ip=196.191.x.x lang=python
[REAL_USER] fix/public device=624f526c ip=196.191.x.x lang=python
[REAL_USER] fix/auth device=6e193b74 ip=196.191.x.x tier=free lang=javascript
Someone in Ethiopia found it. Tried it keyless, twice, in Python. Then signed up for a free account and used it three more times in JavaScript.
Five real fixes. One real human who had never heard of me, walking the exact funnel I designed: try it free → like it → sign up → keep going.
That single line of log output meant more than 525 ever did. Not because it's big — it's tiny — but because it's real, and I can watch it grow from a number I trust.
What I'd tell you if you're building something
Know which of your numbers are load-bearing. If a metric would change your decisions, it needs to be instrumented by you, not inferred from someone's dashboard.
Be suspicious of numbers that only make you feel good. My install count never once caused me to do anything differently. That should have been the tell.
Any conversion rate over 100% is a bug, not a win.
Instrument every code path, then verify the instrumentation itself.
Exclude yourself from your own metrics. You are your product's most active fake user.
And check what your public page says about you. I was telling people to verify my work at a link that flatly contradicted my claims. That's the kind of mistake that costs trust at the exact moment you can least afford it.
Going from 525 to 4 was the most useful thing that happened to this project. Fake momentum kept me polishing a product nobody had found. Real zero made me go get users.
The number's still small. But it's mine, and I can count it.
Top comments (3)
525 vs 4 is a gut punch of a title. i track my own numbers way too closely and i've definitely trusted a counter that was lying to me. what tipped you off that the real number was that different?
thank you — honestly that title was the whole reason I finally wrote it down.
what tipped me off was the silence. if 500 people had installed a tool that calls my API to fix bugs, my server should've been at least a little busy. it was dead quiet. and I explained that away for way too long — bad onboarding, people install and forget, etc.
what finally broke it was the conversion funnel: 160 page views → 525 "acquisitions." that's 328%. I stared at it going "that's impossible" before it clicked — a number over 100% doesn't mean you're crushing it, it means the events aren't coming from humans at all. the chart even split it: almost all "downloads" (scraper bots grabbing every version I publish) vs almost no real "installs from the editor," and the spikes lined up exactly with my release dates.
the humbling part is I had all that data the whole time. I just liked the big number too much to look under it.
sounds like you know the feeling though — what's the counter that lied to you?
"what tipped me off was the silence" is such a good tell. i'd have done the exact same thing, blamed onboarding, blamed people forgetting, anything but the number. the funnel finally not letting you lie to yourself, oof.