![]() Why Is There a Need for a New Statistical Method in A/B Testing? Using these flexible early stopping rules results in an average gain starting from ~20% in the worst case scenario (true difference is just below the minimum effect of interest) and reaching ~80% in the best case where the true lift of the tested variant is significantly higher or lower than the minimum effect of interest. The efficiency gains compared to fixed sample tests come from the use of rules for early stopping for efficacy and for futility (lack of practically-significant effect, non-superiority). Having a reliable measure of the effects of any AB testing treatment and the uncertainty of the measure enables decision-makers to make smart choices and move their business in the right direction with greater confidence. The result from applying the AGILE approach to A/B testing is a greatly improved efficiency and thus ROI (Return on Investment) while maintaining the ability to control the uncertainty (error probability) in the experiments. Post-test estimates for % true lift, confidence intervals and p-value with reduced conditional bias.Fail fast! Statistical guidance to terminate a test early when it is unlikely to see a minimum desired effect.Optional stopping issues are solved with statistically rigorous rules for early stopping for efficacy.Great flexibility in monitoring results and decision making.classical fixed-sample tests ( 50-60% on average*) Significant efficiency improvements: 20%-80% less users needed vs.to optimize the amount of resources, committed to an experiment, relative to the expected utilityĪnother reason to prefer statistical methods developed for medical testing is that they undergo severe scrutiny before being used in practice and most statistical procedures AGILE is built upon have been in use in the medical field for years or sometimes decades. ![]() ![]() to minimize exposure of test subjects to ineffective treatments.to ship/implement effective treatments as quickly as possible.In developing AGILE we did a very extensive research in the theory and practice of frequentist sequential testing ultimately focusing on the medical field due to immense similarities between the business case in AB testing and the scientific research case in medical testing. We believe it also addresses most if not all of the major statistical hurdles in doing proper AB and MVT tests in conversion rate optimization, landing page optimization, e-mail marketing and online advertising optimization in general. ![]() The result of our research we call the “AGILE A/B Testing Approach”. AGILE offers significant flexibility and greatly increased efficiency in performing any kind of A/B test. In this post we’ll cover briefly the need for a new method, some highlights of the method we propose and a brief introduction to the software tool we’ve developed to help apply it. After many months of statistical research and development we are happy to announce two major releases that we believe have the potential to reshape statistical practice in the area of A/B testing by substantially increasing the accuracy, efficiency and ultimately return on investment of all kinds of A/B testing efforts in online marketing: a free white paper and a statistical calculator for A/B testing practitioners. ![]()
0 Comments
Leave a Reply. |