R Tutorial: Designing an Experiment - Power Analysis

now that we have a good sense of our baseline numbers we're ready to design our experiment here we use our knowledge of seasonality along with power analysis to figure out how long we need to run our experiment in preparing our experiment we learn about historical conversion rates on average conversion rates are about 28% but that can change throughout the year what does this mean for building our experiment well it would be bad to run the control condition in August and the test condition in September because the control may look better simply due to seasonality not because it's actually a better condition this is why a be testing experiments try to run both conditions simultaneously to ensure both conditions are exposed to similar seasonal variables we also need to consider seasonal effects for knowing how we expect our control condition to perform if the experiment is run in January we expect the control to have a conversion rate of roughly 20% but if it's run in August the controls should be closer to 50% with this knowledge we use a power analysis to determine how long we should run our experiment experiment length is one of the big questions and a/b testing if you stop too soon you may not get enough data to see an effect too long and you may waste valuable resources on the failed experiment one way to safeguard against this is with a power analysis a power analysis will tell you how many data points or your sample size that you need to be sure an effect is real once you have your sample size you can figure out how long you will need to run the experiment to get your number of required data points this will depend on variables such as how many website hits you get per day running a power analysis is also good because it makes you think about what syste chol test you want to run before starting data collection when running a power analysis you should know one the plants difficult test 2 the value of the control condition and 3 a desired or expected value of the test condition you also need to know one the proportion of the data from the test condition ideally 0.5 or half to the significant threshold or alpha generally 0.05 and 3 the power generally 0.8 terms such as alpha and power should be familiar to already from data camps course on experimental design there are several packages you can use to run a power analysis in our here we'll use the power mediation package the first thing we need to decide is what statistical test will be running since the value for collecting is binary clicked or it didn't click we'll run a logistic regression to run a power analysis for logistic regression we'll use the function s size logistic bin we'll also save the result of our power analysis to a variable total sample size now we need to fill in each of the pieces of our equation to get our final sample size we'll work backwards to figure out each of our variables for sample proportion Beta Alpha and power we'll use the most common values zero point five zero point zero five and zero point eight for a conversion rate for our control condition P one let's say we're going to run the experiment starting in January so we expect roughly a 20 percent conversion rate now the last and hardest part is deciding the expected conversion rate for the test condition p2 normally this is backed by previous data but for now let's guess in dream big we'll say a conversion rate of 30 percent a 10 percent boost we see that we need 587 data points in total or roughly 294 per condition let's get some more practice running power analyses in Chapter two we'll get some results from our experiment and get to analyze them with religious