Forget Statistical Tests: A/B Testing Is All About Simulations
How simulations outperform traditional stats in that they are easier to understand, more flexible, and economically meaningful
Controlled experiments such as A/B tests are used heavily by companies.
However, many people are repelled by A/B testing due to the presence of intimidating statistical jargon including terms such as “confidence”, “power”, “p-value”, “t-test”, “effect size”, and so on.
In this article, I will show you that you don’t need a Master in Statistics to understand A/B testing — quite the opposite. In fact, simulations can replace most of those statistical artifacts that were necessary 100 years ago.
Not only this: I will also show you that the feasibility of an experiment can be measured using something that, unlike “confidence” and “power”, is understandable by anyone in the company: dollars.
Starting from the OEC
Your website has a checkout page. The ML team has come out with a new recommendation model. They claim that, by embedding their recommendations into the checkout page, we can increase revenues by an astounding 5%.
