How to expedite A/B testing with smaller sample sizes

A/B tests taking too long to reach statistically significant results? You're not alone. This problem frustrates many, but there are ways to quicken the process.

In practice, actual business decisions are often made based on a risk/reward analysis.

While the underlying statistics are complex, you can understand them at a level where you are capable of making effective, data-driven decisions based on probability and your attitude toward risk.

To speed up the testing process while still obtaining reliable results, consider these three factors:

1. Higher Risk: 

Accept lower statistical significance. The less confident you need to be in your result, the smaller the sample size required, thus reducing the testing duration. Even when you have large traffics, there are valid situations where your appetite for risk might be higher such as when testing changes that are simple and cost-free. Read more on how to configure test confidence in Mida .

2. Higher Minimum Detectable Effect (MDE): 

Determine what specific uplift you’re looking for. Want to test if B’s conversion rate is 20% better than A? A higher MDE means fewer participants are needed.

3. Focus on Micro Goals: 

These often lead to a higher base conversion rate. For instance, a micro conversion could be the CTR of ‘Add-to-Cart’ button. Higher base conversion rates also decrease sample size requirements. Read more on how to analyze results with multiple goals in Mida .

After finalizing the above, use this free sample size calculator to ensure your experiments in Mida have the right number of participants (MTUs) while still obtaining reliable results fast.

If your MTU quota is running low for your next experiment, please feel free to contact LiRou or Donald for a top-up.