How to configure test confidence

What is a confidence level?

A confidence level is a statistical concept that quantifies the degree of certainty that a given prediction or hypothesis is correct. It's typically represented as a percentage, reflecting how strongly we believe our prediction is likely to be accurate.

For instance, a 95% confidence level indicates that one is 95% certain that the stated prediction or hypothesis is accurate. In other words, if the same situation were repeated 100 times, we would expect our prediction to be correct 95 times out of 100. Therefore, a confidence level provides a measure of trust in the validity of a statistical estimate or prediction.

Understanding Probability Through A/B Testing

Consider a coin flip scenario for simple understanding. Suppose you suspect that heads occur more frequently than tails. To test this theory, you flip the coin six times and get four heads and two tails. Does this confirm your hypothesis? Certainly not, as this is not a substantial number of coin flips for verification. The outcome here is skewed due to insufficient observations or flips.

To address this, increase the number of coin flips to 10,000. The most probable result would be 5,000 heads and 5,000 tails, adhering to the 50/50 chance rule.

A/B testing parallels this concept - by observing a sample, we aim to deduce the actual probability of 'B' resulting in more conversions than 'A'.

So, two major factors come into play in this experiment: 

1. Sample Size: The volume of sessions/ participants needed for each variant to validate the result. For example, you need a larger number of coin flips to get a reliable outcome.
2. Statistical Significance: Given that you could continue flipping the coin forever, it's crucial to decide your necessary confidence level in the outcome. This level is determined by what you intend to use this information for and your tolerance for risk. This is what's referred to as statistical significance; it's an indicator of your level of confidence in whether B truly outperforms A.

These two factors are inseparable and have a significant influence on each other. For instance, the level of acceptable error in statistical significance can affect the necessary sample size.

How to Configure Test Confidence in Mida

After setting up your variant and goals, you can set your confidence level in the 'CONFIGURATION' section. 

While 95% confidence is standard, successful A/B tests can vary based on your risk tolerance. If a test costs nothing to implement, you may accept less confidence. This could lead to smaller sample sizes required and shorter test durations.

Small businesses with lesser traffic may opt for a confidence level as low as 70%. This means that if we were to reproduce the same tests 100 times, we would achieve the observed result 70 times out of 100. Having some confidence level is always better than just guessing.