Somewhere in your second month, you will want to A/B test something. The paywall headline, the button colour, the onboarding order. It feels like the rigorous thing to do, and every growth article you read assumes you are doing it.
Here is the uncomfortable arithmetic: with fewer than about 10,000 users a month, A/B testing is largely unreliable, because only an improvement north of 30% would register as a winner. Between 10,000 and 100,000, you need roughly a 9% improvement to detect anything trustworthy. Most early apps are well below the first threshold, which means most early A/B tests produce a number that looks like an answer and is actually noise.
That does not mean you fly blind. It means the tool you reach for should match the traffic you have.
Why small tests lie
An A/B test is a way of deciding whether a difference you observed is real or chance. Whether it can do that depends on two things: how big the true effect is, and how much traffic you feed it. They trade off directly.
A genuinely dramatic change, an entirely different paywall, a restructured first session, can reach significance on a few thousand users per variant, because the effect is large. A subtle change, a different button colour or a reworded subtitle, might be a real 2% improvement, and at your traffic you would need months to distinguish that from randomness.
What happens in practice is worse than "no answer." Founders run a test for five days, see variant B at 14% versus A at 11%, declare a winner, and ship it. With a few hundred users per arm, that gap is entirely consistent with a coin flip. You have now made a permanent decision on noise, and you believe you made it scientifically.
The two failure modes to know by name: peeking, where you check daily and stop the moment the numbers look good, which manufactures false winners; and testing effects so small that your app will never generate the traffic to detect them.
The honest test for whether to test
Before running anything, ask three questions.
1. How many users per week will actually enter this experiment? Not total users. The ones who reach the specific screen you are changing. If your paywall sees 200 views a week, a paywall test is not viable.
2. How big an effect am I expecting? Be honest. If your answer is "a few percent," stop. You cannot detect it. If you genuinely expect a structural change to move something by a third, you might.
3. Can I run it for at least two weeks? Below a few thousand users per variant, tests need one to four weeks, and always run to a pre-set confidence threshold rather than a date. If you need eight weeks to reach significance, the test will be stale before it finishes and your product will have changed underneath it.
Three yeses: run it. Anything else: use one of the approaches below, which are not consolation prizes. At low traffic they are simply better instruments.
What to do instead
Ship the obviously better thing. Much of early product work does not need a test. If your onboarding buries the core action three screens deep and users tell you they cannot find it, you do not need an experiment to authorize moving it. Reserve testing for genuine coin-flips between defensible options, not for decisions you could make by looking.
Use before-and-after comparison, carefully. Change one thing, watch the metric for two weeks against the two weeks prior, and be honest that seasonality, a marketing push, or a store feature could explain the move. This is weaker evidence than a controlled test, and it is often enough for a decision you can reverse cheaply. The discipline that makes it work: change one thing at a time, and write down what you expect before you look.
Move up the funnel. If your subscription conversion is too rare to test, test something upstream with a much higher base rate: the tap-through on a screen, completion of onboarding, the click on an email. Micro-conversions need far smaller samples because they happen far more often. You are testing a proxy, so pick one you have reason to believe leads to the outcome.
Watch five people use it. Five session recordings or five live walkthroughs will find more real problems in an hour than a month of underpowered testing. Numbers tell you where people fall out; watching tells you why. At small scale, why is the scarce information.
Read your reviews and support email as data. Both are unprompted, specific, and free. Three people describing the same confusion is a stronger signal than a 12% versus 14% difference on 300 users.
When testing does make sense early
Two exceptions worth knowing.
Store listing experiments. Both app stores offer their own listing experiments, and these often have far more traffic than your in-app screens, because store impressions include everyone who sees you in search, not just installers. A different first screenshot or icon is exactly the kind of large, structural change that can reach significance, and the conversion effect is meaningful.
Ad creative. If you are running paid acquisition, the platform shows your variants to enough people quickly, and the differences between genuinely distinct angles are usually large. This is testing you get cheaply because someone else supplies the traffic.
Both share a pattern: high volume at the top of the funnel, and big differences between variants. That is the condition under which testing works, and it is worth remembering as the general rule rather than a pair of special cases.
If you do run one, run it properly
Decide the metric and the stopping rule first. Write down the primary metric, the minimum effect you care about, and how long you will run it, before you start. This one habit prevents most self-deception.
One variable at a time. Two changes in one variant means you learn nothing about which one mattered.
Do not peek and stop early. Checking is fine; stopping because today's numbers look good is not. Run to the plan.
Run whole weeks. Weekday and weekend users behave differently. A test covering Tuesday to Saturday has a bias baked in.
Accept "no difference" as a result. It is common, it is useful, and it means you can pick either option and move on. Founders who only accept winners keep testing until randomness produces one.
The real point
The reason this matters is not statistical purity. It is that early-stage time is your scarcest resource, and underpowered testing consumes it while producing confident nonsense. A founder who spends six weeks testing button variants has spent six weeks learning nothing, and could have spent them talking to twenty users and fixing the three things those users kept hitting.
Test when you have the traffic and the effect is big. Otherwise, look at your funnel to find where people leave, and talk to people to find out why. Somewhere past a few thousand weekly users on the screen in question, testing starts earning its place. Until then, judgment plus evidence beats a coin flip with a p-value attached.
Originally published at https://foundyra.com/news/ab-testing-small-app
Top comments (0)