Two readings of the same experiment.
Day 4. Variant B is up 12.7% on conversion rate. 12,506 visitors in the variant, 12,418 in control. The number is green. Somebody screenshots it and drops it in Slack.
Day 11. Variant B is up 12.7%. Same test, same split, same traffic mix.
Same experiment, same lift. The only thing that changed is how long it was allowed to run. One of those two readings is a result. The other is a coin that has come up heads four times.
Every dashboard I have used will render both of them identically, with a green arrow.
The thing that goes wrong is peeking
A standard frequentist A/B test is a fixed-horizon test. You commit to a sample size before you start, you look once when you reach it, and the α you chose (usually 0.05) is the probability of a false positive for that one look.
The moment you look more than once, that guarantee is gone.
This is not a subtle effect and it is not our finding; it is the classic result on repeated significance testing. Each additional look is another chance for the random walk of the difference between two proportions to wander across your threshold. Look ten times at α = 0.05 and the real false-positive rate lands far closer to 20% than to 5%. Monitor continuously and, in the limit, you will cross the line eventually with probability approaching 1, whether or not there is any effect at all.
Now think about how experimentation actually works in a company. The test is live. The dashboard is open in a tab. There is a standup every morning and a roadmap review on Thursday. Nobody looks once.
So the tooling is not neutral here. A dashboard that renders a day-4 lift the same way it renders a day-11 lift is not reporting a result, it is handing someone a screenshot to win an argument with.
What we show instead
Three decisions, all of which make our product look worse in a demo.
1. The verdict panel says "Still collecting" and means it.
Not "leading". Not "trending positive". Until the decision boundary is crossed, the headline on the results page is that the test has not concluded, and the lift figure sits underneath it rather than above it. Whoever opens that page has to read the word before they read the number.
2. The confidence interval is always shown next to the sample size.
A lift is meaningless on its own. +12.7% with a 95% interval of [-2.1%, +28.4%] at n = 12,506 is a different object from +12.7% with [+8.9%, +16.6%] at n = 180,000, and the only way to stop people conflating the two is to refuse to print one without the other. No bare percentages anywhere in the UI.
3. Sample ratio mismatch is on the results page, not in a settings drawer.
If you asked for 50/50 and you are getting 50.4/49.6 across 200,000 visitors, something upstream is broken: a redirect that fires before the assignment, a bot filter that treats variants differently, a cache that serves control to a subset. The check is a chi-square goodness of fit against the expected allocation, and a low p-value there invalidates everything above it on the page.
SRM is the most useful check in experimentation and the one most often buried. If the split is broken, the lift is not wrong, it is meaningless. We put the flag at the top of the result, in red, before the number.
Sequential is the real answer to peeking
Telling people "don't look" loses to reality every time. The better answer is to use a method designed for looking.
Sequential tests spend your error budget across the monitoring period rather than at a single endpoint, so continuous monitoring is valid by construction rather than in spite of the maths. You give up some power relative to a perfectly executed fixed-horizon test, and in exchange you get a test that survives the way teams actually behave.
Our default engine is frequentist because that is what most teams already reason in. Sequential and Bayesian are available on the higher plans for teams that want to monitor continuously or reason about probability-to-beat-control directly. Whichever engine you pick, the method version is locked when the test starts, so nobody can switch engines halfway and pick the flattering one.
Why any vendor would build this
Here is the part that is uncomfortable to write.
Every incumbent's renewal conversation depends, in some form, on the customer believing their testing programme is working. A dashboard that surfaces wins is commercially aligned with that. A dashboard that says "still collecting" for nine days is not. "Still collecting" is a worse demo than a trophy, and I have watched it be a worse demo.
We can afford to build it because we are small, we publish our prices, and we are not defending a renewal number against a board. That is a structural advantage rather than a virtuous one, and it will not last forever.
But if you sell a statistics product, refusing to overclaim is the product. Everything else is a rendering detail.
ABTestly is A/B testing for teams that write their variations in code rather than in a visual editor. Docs at docs.abtestly.com, and the pricing is published at abtestly.com/pricing.
Top comments (0)