DEV Community

kyb8801
kyb8801

Posted on

D+5 of a 14-day product test — the mid-experiment temptation

#productivity #metrology #experiment #mcp

Five days in. Mid-experiment is the honest part — too early to declare success,
late enough that the absence of signal is itself data.

D+5 dashboard (raw)

Verifiable / public metrics (no spin):

  • Gumroad sales: dashboard confirmation pending (verdict variable: at least 1 by D+14)
  • GitHub MCP repo stars: 0 (D+0 baseline: 0)
  • MCP server health: 200 OK at /health, 401 OAuth at / (both correct by design)
  • dev.to total views (9 articles): 44 (+31 since D+0, +238% growth)
  • Gumroad storefront: both products visible (EN $19, KR ₩30,000)

User-dashboard metrics (not API-public):

  • Beehiiv subscribers: dashboard confirmation pending
  • Beehiiv Week 9 open rate: pending
  • Cold DMs sent: campaign starts mid-window

The mid-experiment temptation

I keep wanting to lower the price to $9. Force a sale. Buy my own product. Run
a 5x affiliate promo. Each of these would invalidate the experiment.

The discipline at D+5 is: don't change variables. Watch what you set up. Trust
the original design enough to let it speak.

I'm bad at this.

What is actually informative

Even at D+5 with the sales counter still pending dashboard read, here is what
the data tells me:

  1. GitHub stars growth correlates with audience trust building, not with
    purchase intent.     People discover the free MCP repo without ever clicking
    through to Gumroad. The free thing has reach; the paid thing has friction.

  2. dev.to traffic without conversion is real-world data. Engineers read
    the engineering story but don't convert because the call-to-action is buried.
    I can fix this on D+6 by moving the Gumroad link to the article opening.

  3. The MCP server endpoint returning 401 is a feature, not a bug. /health
    returns 200; the root requires OAuth. My collect script initially thought
    that was a failure. It was working correctly; my monitoring was wrong.

What I'm NOT doing at D+5

  • Not changing the $19 / ₩30,000 price
  • Not adding a "first 10 buyers get a free cert generator" promo
  • Not pivoting to a different niche
  • Not adding new features

The verdict on D+14 is the only thing that means something. The temptation to
mid-correct is the temptation to make the data unfalsifiable.

What I'm watching

  • Single-buyer attribution: if a buyer arrives, where did they come from?
  • Page-view-to-buyer ratio per channel: which audience is closest to the buyer?
  • Beehiiv open rate on D+7 mid-verdict: 35%? 50%? proxy for "audience cares"

The narrative is in the email I will publish on June 5. The data is in the
emails I am resisting sending until then.

Nine more days.

Top comments (0)