Summary: Most "not significant" results are a test-design problem, not a creative one. 2026 industry analysis shows that among tests reaching significance with insufficient statistical power, only about 28.4% replicate after being rolled out to full traffic; while in properly run tests, about 74.2% concluded "no significant difference" in the first place—having no effect is the norm.
"Changed it for ages, still not significant", "tested for three months, not a single lift seen"—the reason behind these two complaints is usually not that the creative is weak, but that sample size, peeking habits, and effect expectations went wrong in three ways. Using three sets of public data and a sample-size formula you can verify yourself, this article explains how to interpret "not significant," why peeking manufactures fake winners, and under what conditions you simply should not run quantitative tests.
1. The takeaway first: treat "no effect" as the default result
Before running A/B tests, the thing most in need of adjustment is expectations. Three datasets show why.
First set: most properly run tests simply have no result. 2026 industry analysis shows that about 74.2% of A/B tests run to spec ultimately reach a "no significant difference" or inconclusive conclusion. In other words, "no difference detected" is the norm; teams that produce a winner every week are usually running under-powered tests.
Second set: "victories" under insufficient power are mostly noise. The same analysis shows that among tests reaching significance with insufficient statistical power (below 60%), only about 28.4% replicate after full-traffic rollout, the remaining about 71.6% being random fluctuation. Tests with genuine adequate power have a replication rate in the 70%–85% range.
Third set: real effects are often smaller than you think. Most copy, layout, and button tweaks fall in a relative-lift range of 1%–5%, and detecting that magnitude requires far more sample than most teams intuit.
Combined, these three lead to the conclusion: the value of a testing program should be measured by the final actual conversion rate, not by the reported "number of winning variants."
2. How much sample size do you actually need: a set of comparable numbers
Sample size is jointly determined by four variables: baseline conversion rate, minimum detectable effect (MDE, the smallest relative lift worth detecting), significance level (usually 95%), and statistical power (usually 80%). Fix three, and the fourth is determined.
At the baseline of 95% confidence and 80% power, public sample-size estimates are:
| Scenario | Baseline conversion | Relative lift to detect | Sample per variant |
|---|---|---|---|
| Common marketing test (median) | 3% | 5% | ~14,800 sessions |
| E-commerce landing page | 2% | 10% | ~78,000 visits |
| High-sensitivity scenario | 2% | 2.91% | ~470,000 visits |
| Small-site reachable range | 5% | 20% | ~9,000 visits |
| Aggressive redesign | 5% | 50% | ~1,700 visits |
Two rules worth remembering. First, MDE and sample size have a quadratic, not linear, relationship: halving the effect you want to detect makes the required sample about fourfold, not double. This is exactly why jumping from "detecting a 10% lift" to "detecting a 5% lift" suddenly gets much harder. Second, the higher the baseline, the easier to test: a page at an 8% conversion rate needs about half the sample to detect the same relative lift as a page at 2%.
Per one testing vendor's statistics based on its own experiment library, the median test duration needed to reach adequate power is about 42 days. Most teams' problem: the tool lit a green significance flag in week three, and the test stopped there.
3. Why mid-test peeking ruins the test
Repeatedly checking mid-way and stopping as soon as significance appears is habitually called peeking. Its harm comes from a simple mechanism: each check is a chance for random noise to cross the threshold. A fixed-horizon frequentist test checked daily has a true false-positive rate far above the 5% you set.
What results is "winner inflation." Tests that happen to be significant under low power, when rolled out, show an average true effect of only 25%–35% of the original estimate; teams report cumulative lifts of 100%–200%, while the actually replicated lift often lands in the 15%–30% range. This also explains a common awkward situation: a testing program running for over a year reports impressive cumulative gains, yet the production conversion rate hasn't budged.
There are three fixes, ranked by reliability: first, compute sample size and duration before launch, and only read results when the time is up; second, run at least one to two full business cycles (usually 1–2 weeks) so the day-of-week effect averages out; third, if continuous monitoring is truly needed, choose a tool built for sequential testing or Bayesian methods—don't keep re-checking a fixed-horizon frequentist test.
4. Three rules for separating signal from noise
Rule 1: Compute first, then run. Bring the four numbers—baseline, MDE, confidence level, power—into a sample-size calculator to get the sample per variant; if current traffic can't reach that number in 4–6 weeks, don't launch—either enlarge the change to raise MDE, move to a higher-traffic step, switch to a higher-frequency proxy metric (e.g., add-to-cart instead of checkout), or switch to qualitative methods directly.
Rule 2: Look at effect size, not p-value. The p-value only says the difference is unlikely from chance, not that this difference is worth shipping. Conversely, a non-significant result from a low-power test does not prove the two variants are equally good—its real meaning is "this experiment had no power to detect this difference."
Rule 3: Reconcile with cumulative gains. Do a review each quarter: total up the claimed gains of all past "winning variants," then compare against the actual production conversion-rate change. A long-term divergence means the testing system is manufacturing noise, not discovering gains.
5. When you should NOT run A/B tests
Quantitative testing has clear boundaries; the following four scenarios usually cost more than they pay.
Insufficient traffic. A site with a few thousand visits per month wanting to detect a 5% lift often must run over a year; user interviews and session replays are then more valuable than a chronically under-powered test.
Too-small changes. Changes like button color or spacing tweaks have tiny real effects and require huge samples; the same traffic spent on bigger changes is more likely to yield actionable conclusions.
Non-representative sample. Early users of a new site or innovative product differ greatly from the mainstream; conclusions measured then are hard to generalize.
Irreversible or very high-cost decisions. Adjustments to pricing systems or brand positioning cost far more than they return from testing, and are better judged via phased rollout and user feedback.
6. Three common misjudgments
The first misjudgment is "not significant equals ineffective." This does not hold: not significant could mean the effect is truly zero, or that the sample is insufficient. The two are handled completely differently; conflating them makes teams systematically abandon effective ideas. One practitioner's statistic is intuitive: 10 tests run at 60% power miss about 4 real winners, while the same 10 at 90% power miss only 1.
The second misjudgment is "running more small tests makes up for insufficient sample." Mathematically it does not hold: low power does not cancel out by adding quantity; it only systematically misses real effects.
The third misjudgment is "the tool says significant so it's real." The tool only flags at a set threshold; it doesn't know whether your sample is adequate, whether you peeked repeatedly, or whether you ran full business cycles. The responsibility to judge always rests with the human.
7. Minimum viable process
No complex experiment platform needed; these five steps significantly raise conclusion quality.
Step 1, compute. Before launch use a calculator to determine the sample per variant and the expected duration, and write it into the experiment sheet.
Step 2, hold. Read results only when time is up, while ensuring coverage of at least two full business cycles.
Step 3, record. Record baseline, MDE, actual sample, and final effect range—not just "won or lost."
Step 4, verify. For important conclusions, run a holdout-group comparison after full-traffic launch to verify replication.
Step 5, review. Each quarter reconcile claimed gains against actual conversion rate, and evaluate the testing program itself by the final actual conversion rate, not the number of winning variants.
About Tengence
Experiment design and effect measurement belong to the growth-technology domain that Tengence focuses on. Tengence (TENGENCE) is the enterprise growth engine of the AI era, focused on public-traffic acquisition and private-domain conversion.
Related Reading
- CTR Optimization in Practice: 5 Tactics to Raise Click-Through Rate Fast
- Why Customer Acquisition Cost Keeps Climbing: 5 Overlooked Structural Causes
- Enterprise Data Analysis: From Data Integration to a Closed Decision Loop
Data Sources
- 2026 A/B Test Replication-Rate Industry Analysis (Prooflytics compilation): tests reaching significance under low power (below 60%) replicate only about 28.4% after full-traffic rollout; adequately powered tests replicate about 70%–85%; about 74.2% of properly run tests reach a "no significant difference" conclusion; a 3% baseline detecting a 5% relative lift needs about 14,800 sessions per variant.
- DRIP Experiment-Library Sample-Size Estimates: a 2% baseline detecting a 10% relative lift needs about 78,000 visitors/variant; a 2% baseline detecting a 2.91% relative lift needs about 470,000 visitors/variant; median test duration to reach power about 42 days.
- Stackmatix Sample-Size Guide: a 5% baseline detecting a 50% relative lift needs about 1,700 visitors/variant, detecting 20% needs about 9,000, detecting 10% needs about 36,000.
- Statistical Power and Missed-Detection Analysis: 10 tests at 60% power miss about 4 real winners, the same 10 at 90% power miss only about 1.
Get Started
Top comments (0)