DEV Community


Discussion on: A/B tests for developers

tadman profile image
Scott Tadman

Don't forget basic statistics here. If you're developing an A/B test for a case that will be used millions of times per day then a hundred tests is not going to be conclusive. You need to test a statistically significant number of times relative to your actual use case.

ben profile image
Ben Halpern

Yep. This is a conversation I've definitely had numerous times.

bgadrian profile image
Adrian B.G. Author

Yes ofc. Beside the statistically relevance there are other factors like user aquisition, cohorts have to be very similar and there is also an error margin of at least 5% which should be taken into consideration when comparing the stats.

That is part of the business part of doing tests, I tried to cover only the technical details.

The web is full of articles on how to do proper testing, but it was lacking on the implementation details, so I wrote this story.

Forem Open with the Forem app