I bet you have heard about A/B testing before, but are you actually using it? If not, maybe because it doesn't fit your project, but for a great number of developers controlled experiments (experiments which run A/B tests) are definitely relevant. In this piece I'd like to go through some great reasons to try them out!
We don't have enough users!
First of all, let me debunk the myth that controlled experiments are suitable only for tech giants, on the contrary!
The key point to understand is that these big companies are looking for small, incremental conversion rate (CR) improvements somewhere between +0.1-2%. But if you are a startup and you want to get that additional funding, you are looking for rapid, big improvements on the scale of 10% and (much) more which require proportionately less users to verify in an A/B test.
Example: You have CR rate 20%. To detect an absolute 0.1% improvement, you'd need 2.500.000 users. But to see a 5% (25% relative) you'd need only 1.000 users over the course of a few weeks! That's absolutely doable even for small startups.
Improve decision culture
With that out of the way, let's talk decision culture. Unfortunately, in many companies a (no)launch decision is being made based purely on opinion without any backing in the data. This of course can lead to launching features that worsen UX.
Using A/B tests to validate new features shifts the decision culture from opinionated to proof-based, which brings confidence into improving the product. Moreover it ends unproductive arguments in the team by providing evidence of which features work the best.
Lastly, A/B test results are often interesting for the stakeholders and increase management's trust in feature team's decisions since they are backed by data.
Learn
Learning from your experiments is at least as important as making that (no)launch decision. Each experiment tests a certain hypothesis and the outcome should say in detail whether it was correct and why. Over time, you'll be able to accumulate enough learnings to have an intuition of what works and what doesn't, which makes developing new features more effective.
Also, should someone else have the same initiative in mind that you already performed a while back, going back to those learnings will steer her in the right direction.
I also think that articulating the hypothesis and looking for the learning during the experiment helps the team understand better how their initiative ties to the company's strategy, which helps teams focus and work on things that really matter. This fact is incredibly important for companies big and small.
Conclusion
A/B testing is an important tool and a de facto standard when validating product's features. Using it will help you make better product decisions and improve communication inside and outside your team.
Sources
Kohavi R, Tang D, Xu Y. Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press; 2020.
https://blog.statsig.com/you-dont-need-large-sample-sizes-to-run-a-b-tests-6044823e9992
Tools
https://www.growthbook.io/
https://www.evanmiller.org/ab-testing/sample-size.html
Top comments (0)