This calculator will help you determine the minimum sample size needed per variation in your experiment. Start by entering your values into the inputs. The Baseline Conversion Rate and Minimum Desired Effect will vary with each of your experiments; whereas, the confidence and power levels will usually remain as the default values-- unless the determined sample size is unachievable. After the values have been entered, click on "Calculate" to view a table of your results.
The Baseline Conversion Rate (BCR) is synonymous with your benchmark conversion rate. Most people use historical data to determine their BCR by dividing the number of conversions by the number of impressions or people for a given time period. Minimum Desired Effect is the smallest percentage change you desire to achieve after the experiment runs. For example, if your Baseline Conversion Rate is 0.05% and you hope that at the end of the experiment, you reach 0.06%, then your Minimum Desired Effect is 20%.
This calculator uses a right-tailed hypothesis test to estimate the necessary sample size. A one-sided hypothesis test was used because the majority of our clients prioritize the significance of a winner in their experiments (positive change in conversion rate)--not the significance of any change (positive or negative changes in conversion rate).
Detecting significance in small changes of conversion rates requires very large sample sizes and some online traffic does not allow for the desired sample size. If this is the case, you can decrease your minimum sample size by reducing your confidence level, reducing your power level, or increasing your Minimum Desired Effect. The most common method is reducing the confidence level, but we do not suggest using a confidence level less than 80%
The unit used in the denominator of your conversion rate calculation is the same unit for the results in the sample size calculator. Traditionally, conversion rate is calculated as number of conversions / number of impressions. With Thunder’s people-based approach, conversion rates can also be calculated as number of conversions / number of people. If you used impressions to calculate the conversion rate used in the sample size calculator, and resulted in 1MM per variation, then you would need 1MM impressions per variation for your experiment. On the flip side, if you used people to calculate the conversion rate used in the sample size calculator, and resulted in 1MM per variation, then you would need 1MM people per variation in your experiment.
The effect size is a measure of the magnitude of the treatment effect in your experiment group. The smaller the effect you have, the more samples are needed to confidently determine whether the difference in your results is due to actual signal or merely noise. For example, compare an energy drink (which has a high caffeine effect) to regular tea (which has a low caffeine effect). You would need to consume less of the energy drink than the tea before feeling the effects of caffeine simply because the effect from the energy drink is much higher.
Setting a very high minimum desired effect cuts down on the required sample size for your experiment, but most of the time, it is unachievable- thereby, wasting time and resources to run the experiment in the first place. By making sure that your minimum desired effect is the lowest value you expect to achieve with your experiment, you are essentially planning for the worst-case scenario. Once the experiment runs, you can be confident about your results at the end for any lift greater than or equal to your initial minimum desired effect- thus, optimizing on the cost and benefit of running the experiment.