Most experiments that come back "no clear winner" were unwinnable on the day they launched. The data could not resolve an effect that size, and no amount of extra runtime was going to change that. You can find this out in about two minutes, before you spend anything, with one formula and a resampling pass over your own data.
Here is the check, in three steps.
Step 1. Compute the smallest lift your data can see
For a two-arm test on a conversion rate, the smallest lift detectable at 95% confidence and 80% power is a one-liner:
from math import sqrt
Z_ALPHA = 1.96 # two-sided 95%
Z_BETA = 0.84 # 80% power
def mde(baseline_cvr: float, n_per_arm: int) -> tuple[float, float]:
"""Minimum detectable effect: absolute (pp) and relative (%)."""
se = sqrt(2 * baseline_cvr * (1 - baseline_cvr) / n_per_arm)
abs_lift = (Z_ALPHA + Z_BETA) * se
return abs_lift * 100, abs_lift / baseline_cvr * 100
At a 3% conversion rate:
| clicks per arm | smallest lift you can detect |
|---|---|
| 5,000 | +32% relative |
| 20,000 | +16% relative |
| 100,000 | +7% relative |
Read the middle row twice. Twenty thousand clicks per arm is a serious amount of traffic for a mid-market account, and a real 15% improvement still lands inside the confidence interval. The report will say "inconclusive," and the team will read that as a verdict on the idea. It is a verdict on the instrument.
Invert the same formula and the planning question gets easier: at 3% baseline, detecting a 10% lift needs about 51,000 clicks per arm, and detecting a 5% lift needs about 203,000. If your account produces 8,000 clicks a month, you now know the honest answer to "how long should we run this."
Step 2. Stop assuming your conversions are independent
The formula above treats every click as an independent coin flip with the same probability. Account data does not behave that way, and the gap is not small.
In a corpus of 31 advertiser accounts I maintain for diagnostic work (9.46 million search term rows, roughly $133M of spend, September 2024 to February 2025), the median account draws 62% of its conversions from the top 1% of search terms by cost. The middle half of accounts sits between 51% and 72%, and the full range runs from 28% to 83%.
When most of the outcome rides on a few terms, the effective sample size is far below the row count, and a textbook calculator quietly promises resolution you do not have. Do not guess the correction. Measure it by resampling terms rather than clicks:
import numpy as np
def bootstrap_se(conversions, clicks, draws=2000, seed=0):
"""Empirical SE of account CVR, resampling whole search terms."""
rng = np.random.default_rng(seed)
n = len(clicks)
rates = np.empty(draws)
for i in range(draws):
idx = rng.integers(0, n, n) # resample terms, not clicks
rates[i] = conversions[idx].sum() / clicks[idx].sum()
return rates.std(ddof=1)
def design_effect(conversions, clicks):
p = conversions.sum() / clicks.sum()
binomial = sqrt(p * (1 - p) / clicks.sum())
return (bootstrap_se(conversions, clicks) / binomial) ** 2
Required sample size scales with the design effect; the detectable lift scales with its square root. A design effect of 2 turns that 20,000-clicks-per-arm row into 40,000 and pushes the detectable lift from 16% to about 23%. I would treat any account that has never run this as having an unknown, and probably optimistic, power calculation.
Step 3. Check that your baseline holds still
A test compares two arms measured at the same time. A before-and-after comparison compares one arm against a moving target, and the movement is larger than most planning documents assume.
In the same corpus, across 1,950 campaigns running Target CPA on about $42.2M of spend, only 34% landed within plus or minus 10% of the target CPA they were given, and 36% overshot it by more than 20%. That is the platform missing a number it was explicitly instructed to hit, on its own optimized traffic, with no experiment running at all.
So the drift in an untouched account is routinely wider than the 10% to 15% effect you are trying to detect. Any month-over-month readout is measuring your change plus that drift, and it cannot tell you which is which. If randomizing users is not available, split geographies instead and run both arms in the same weeks, so the drift hits both.
A test you cannot power is not a cheap test. It is a slow way to buy a coin flip.
The four ways this check gets thrown away
- Peeking. Watching the p-value daily and stopping at the first significant reading inflates false positives badly. Fix the sample size before launch and read once.
- Unit mismatch. Randomizing at the campaign level and analyzing at the click level understates variance. Randomize and analyze at the same unit.
- Attribution shorter than the sales cycle. If the window is 7 days and the median purchase takes 20, the treatment arm's conversions land after you stopped counting.
- Two changes, one test. New creative plus new landing page gives a directional result and no mechanism, which is the expensive kind of win because you cannot repeat it.
What to do when the number says do not run it
Three moves, in the order I would try them.
Raise the floor. More observed outcome per click is worth more than more clicks. Server-side tagging, offline conversion imports, and cleaning bot traffic all reduce the noise term directly. This is the only one of the three that compounds: every future test inherits the better floor.
Change the unit. If a keyword-level test cannot power, an account-level geo holdout usually can. Aggregating trades granularity for resolution, which is the right trade when the alternative is measuring nothing.
Decide without the test, on purpose. Some changes are cheap to reverse and not worth 200,000 clicks of evidence. Write down the belief and the reversal trigger, then ship it. That is a decision, not an experiment, and confusing the two is what fills a test log with entries nobody can act on.
Where this does not apply
The resolution problem binds when the effect is small relative to the noise. It relaxes in three cases. Logged-in commerce that records almost every outcome carries so little noise that anything worth shipping is already visible. Small single-channel accounts do not generate enough decisions to repay the arithmetic. And one-way-door choices such as pricing or a rebrand are governed by the cost of being wrong, not by the value of the measurement, so you buy insurance instead of significance.
Two honest limits on my own numbers. The corpus is micro-SaaS heavy and skewed toward search, so the 62% concentration figure is a starting prior for your account, not a constant of nature. And the normal approximation behind the formula gets shaky below roughly 30 conversions per arm, which is exactly where most of these arguments happen.
The habit worth building is small: before any experiment, write down the lift you are hoping for and the lift your data can see. If the second number is bigger, you have learned something real for free, and you have not spent a quarter learning it the slow way. Finding which stage of a measurement loop is actually the binding one, instead of optimizing the stage that is easiest to touch, is the work I call Profit Forensics.
Igor Ivitskiy is an expert in Google Ads profitability analysis and the creator of Profit Forensics, ranked #6 in The PPC Survey's 2026 list of the 50 most influential PPC experts. Adapted from my LinkedIn article "The Diffraction Limit of Marketing Measurement" in the M.A.T.H. series: https://www.linkedin.com/pulse/diffraction-limit-marketing-measurement-igor-ivitskiy-nf2me
Method paper (SSRN): https://dx.doi.org/10.2139/ssrn.6258238 . Open-access mirror on Zenodo: https://doi.org/10.5281/zenodo.18552246
Top comments (0)