Power Analysis for AB Testing

Power analysis for AB testing. The calculations are based on the Welch's unequal variances t-test, which is generally preferred over the Student's t-test when sample sizes and variances of the two groups are unequal, which is frequently the case in AB testing. In such situations, the Student's t-test will give biased results due to using the pooled standard deviation, unlike the Welch's t-test.

pwrAB is a simple package to perform power analysis for AB testing. There are a number of R packages for power calculations, such as the widely used pwr package, but to my knowledge none of them allow the variance and sample size to simultaneously vary across the two groups. Instead, they are based on the pooled t-test, and will provide biased results when the variance and sample size are not equal across groups.

The two functions in the pwrAB package use the Welch's unequal variance t-test instead, and will provide valid results even when sample sizes are unequal. Delacre et al. (2017) expands on this topic in more detail for those that are interested (http://www.rips-irsp.com/articles/10.5334/irsp.82/)

Much of the code/syntax of this package is borrowed from the pwr package by Champely et al. (https://cran.r-project.org/web/packages/pwr/index.html). Many thanks to the authors for making their code publicly available.


Reference manual

It appears you don't have a PDF plugin for this browser. You can click here to download the reference manual.


0.1.0 by William Cha, 4 years ago


Report a bug at http://github.com/williamcha/pwrAB/issues

Browse source code at https://github.com/cran/pwrAB

Authors: William Cha [aut, cre]

Documentation:   PDF Manual  

GPL (>= 3) license

Imports stats

Suggests testthat

See at CRAN