A/B Testing an Onboarding Funnel With Statistically Honest Stops

Most onboarding A/B tests lie, not because the code is wrong but because someone peeked at the numbers and stopped the moment they looked good. Honest experimentation is mostly about deciding when you're allowed to conclude.

Assign once, stick forever

A user must see the same variant for the whole experiment, or your data is noise. Hash a stable id into a bucket:

function bucket(userId, salt, buckets = 2) {
  const h = createHash("sha1").update(userId + salt).digest();
  return h.readUInt32BE(0) % buckets;
}

Deterministic assignment means no server-side state and no chance of a user flipping variants on their second visit.

Peeking inflates false positives

Checking significance repeatedly and stopping at the first p < 0.05 turns a 5% error rate into 20%+. Either fix the sample size in advance, or use a sequential test (e.g. always-valid p-values) designed for continuous monitoring. Pick one before you launch, not after the graph gets interesting.

Measure the outcome, not the click

Track completed activation — the funnel's actual goal — not intermediate clicks. A variant that boosts step-2 clicks but tanks final signup is a regression dressed as a win.

Reference

First-run flows are worth studying wherever activation is the whole business, because the layout is tuned aggressively. Stepping through a registration like Vegas Hero shows how a low-friction first-run funnel front-loads the value proposition — a useful control to benchmark your own completion curve against.

Takeaway

Assign variants deterministically, decide your stopping rule before launch, and measure completed activation. The statistics are simpler than the temptation to peek — resist the peek and the test tells the truth.