Two quantities decide how long a test runs, and the calendar is not one of them.
How many visitors per variation the test needs, and how fast you supply them. That is the whole projection.
Our duration estimator returns 15 days for a particular test. The schedule you should commit to is 21.
The gap between those two numbers is the interesting part, and it is where most test plans go wrong.
What the projection actually computes
Required sample per arm, divided by arrival rate. That is it.
required is always computed at 95% confidence and 80% power. Not at whatever you set in the experiment. At 95 and 80, every time.
So if you configured the test at 99% confidence, or you are running four variations under a Bonferroni or Sidak correction, you need more than the projection says.
Treat the number as a floor. It is the arithmetic, not the plan.
Four things it does not know
It does not round to whole weeks.
Behaviour varies across the week. A run that stops mid-week hands one variation an extra Saturday.
Round your own end date up to a whole number of weeks so every day of the cycle appears the same number of times in both arms. That is why 15 becomes 21.
It does not hold a business-cycle floor.
If traffic is heavy enough that the sample arrives in two days, the projection will happily say two days.
Two days is one narrow slice of your audience and one mood of the market. Hold a floor of at least one full cycle, preferably two, so you can watch the effect survive a second week.
If your purchase cycle is longer than a week, stretch the floor to match it.
It does not model novelty or primacy.
Returning visitors react to a change because it is new. Some engage more than they will once it is ordinary. Some are thrown by an unfamiliar layout.
Both effects fade. Neither is visible to a projection that only counts visitors.
It does not know your confidence level or your correction.
Covered above, and it is the one people are most surprised by.
The trap at the other end
There is a status on the runway line called alreadySignificant, and it short-circuits the projection to reached.
The moment the leading variation clears 95% on the gated p-value, the line stops showing a date and starts saying you have enough.
On day 4 of a planned three-week run, that is exactly the reading a fixed-horizon test cannot support.
So we wrote the caveat into the product rather than leaving it in a blog post. The status describes the state of the evidence right now. It is not permission to stop.
If you want a runway you are allowed to watch continuously, that is what a sequential test is for. A fixed-horizon test plus a live dashboard is not the same thing, however much it looks like it.
Why we gate small numbers instead of hiding them
There is a floor of 100 visitors and 5 conversions before certain figures are judged at all.
But the rate, the interval and the lift are never withheld, at any sample size.
Those three are honest at any n, because the interval is the thing that says so. A 95% interval of [-19%, +44%] is not a weak result being hidden from you. It is the result.
What we refuse to do is print a bare percentage off a handful of visitors, because a bare percentage is the thing people screenshot.
The revenue-per-visitor interval uses the same bar, since a revenue interval is at its noisiest exactly where that bar is not met.
One thing a confidence interval is not
It is not the probability that the true value lies inside it.
Under the frequentist reading, the true value is fixed and the interval is what is random. A 95% interval is a procedure that brackets the truth in 95% of repetitions.
The sentence people actually want sounds like "there is an 87% probability this variation is better". That is a posterior probability with a credible interval beside it, and it comes from a Bayesian engine, not from a frequentist CI with the words swapped.
It is also not a guard against bias. An interval describes sampling noise and nothing else. A test with a sample ratio mismatch can have a beautifully tight interval around a meaningless number.
The practical version
Before you launch, compute the required sample at the confidence and power you are actually using, including any correction.
Divide by your real arrival rate, not your best week.
Round up to whole weeks. Apply a floor of one full business cycle, two if you can afford it.
Write the end date down before the test starts, and treat every look before it as monitoring rather than deciding.
Then, when someone asks on day 4 whether it is winning, the answer is that the test is still running, and that is a complete answer.
ABTestly is A/B testing for teams that write their variations in code rather than in a visual editor. The duration model and its caveats are documented at docs.abtestly.com, and the prices are published at abtestly.com/pricing.
Top comments (0)