Your team has started running experiments. Half the users see the old checkout button and half see the new one, and every Monday someone shares a chart. Version B is up three percent. Everyone seems pleased. You nod, and privately you have no idea whether three percent means anything, or whether last week it was down two.
You do not need a statistics degree for this. You need a handful of ideas, learned well enough to ask the right questions.
Start with the biggest one. If you flip a coin twenty times, you rarely get exactly ten heads. Two groups of users will almost always behave a little differently even if nothing changed at all. The whole point of the statistics is to tell a real difference from that ordinary noise. So the first question to learn to ask is not which version won, but how likely is it that we would see a gap this size by chance.
Then learn why the length of the test should be decided before it starts. If you check the results every day and stop the moment they look good, you will find winners that are not real far more often than you expect. The noise drifts up and down, and sooner or later it drifts past the line. Agree in advance how many users or how many weeks, and wait. Running for whole weeks also helps, because people often behave differently at weekends.
Learn what happens when you measure many things at once. Track twenty metrics and one of them will usually look significant purely by luck. Decide beforehand which single result matters most.
And learn about novelty. Something new can get clicked because it is new, and the effect can fade after a few weeks.
The best way to learn all this is not a course. Find whoever on your team or in the company knows it well, often a data analyst, and ask if you can sit in when they read the next result. Ask them to think aloud. What do they look at first? What would make them suspicious?
Then try one habit of your own. Before each test, write down what you expect to happen and why. Comparing your guesses with the results teaches you more about your users than any chart, and it keeps you honest about the experiments you were hoping would win.
– Asael Shinder
Top comments (0)