How to interpret confidence interval

What is a confidence interval?

A confidence interval, in statistics, addresses the degree of uncertainty linked with an estimated value, typically an average, derived from a limited sample of a larger population. This concept underlines that any statistic or test statistic, though inherently subject to uncertainty, can still provide meaningful insights into the overall population.

Consider the scenario of a marketing study evaluating the average time spent by consumers on a new website. After tracking a select group of users' browsing sessions, the duration of these visits can be compiled and an average calculated. However, as this calculated average only relies on a subset of users, it is accompanied by a degree of uncertainty.

In this context, a confidence interval becomes pertinent. It frames a range within which the actual average browsing time of all the site's users is likely to fall. In essence, a confidence interval informs you how confident you can be that the observed results accurately reflect the values that would have been seen if it were feasible to collect data from every single individual in the population. A broader confidence interval implies more uncertainty, while a narrower interval indicates a higher degree of confidence in the estimate's accuracy.

In statistical terms, what is a confidence interval?

In simple terms, a confidence interval in statistics is a range where we expect a certain number to fall. For example, if we create a confidence interval with a 95% confidence level, it means that if we were to recreate the study 100 times, 95 times the number would fall within that range.

The confidence level we choose usually ties back to our accepted level of mistake, often called statistical significance or alpha. 

For instance, if we set an alpha of 0.05 (meaning we are okay with being wrong 5% of the time), then our confidence level would be 95% because 100% - 5% = 95%. So, we are 95% sure that our number will fall within our predicted range.

Interpreting confidence interval in Mida report

Green: This shows a winning variant. Given that the required confidence level is 95%, this test result is considered statistically significant because the statistical significance value of 99.71% surpasses the required threshold of 95%. Statistical significance refers to the probability that the differences observed in the test are not due to chance. In this context, a 99.71% statistical significance means that there is a less than 0.3% likelihood that the results occurred by chance. And, with an improvement of 233.04%, this is a meaningful lift.

Next, look at the confidence interval of the difference of means. The confidence interval provides the range of expected lift values, at the 95% confidence level. In other words, the lower bound is the “worst case” scenario of possible lift and the upper bound is the “best case” scenario. Here, you see a range from 0.44% to 1.32%. Since both numbers show a good increase compared to the Control CR (0.26%), you can feel confident about the change.

Red: This shows a losing variant. Given that the required confidence level is 90%, then the statistical significance of the losing variant should ideally be less than 10% (100%-90%).  The statistical significance value of Variant 1 is 6.71% which implies that the observed difference occurred due to randomness is roughly 6.71%, a figure that is below the required threshold of 10%. In other words, you can be (100 - 6.71) = 93.29% confident that the losing variant, Variant 1, is indeed inferior to the winning variant, Control. 

Looking at the confidence interval, we see a range from 4.69% to 5.29% Conversion Rate (CR). In simpler terms, 4.69% represent the “worst case” scenario, and 5.29% represent the “best case” scenario of Variant 1's performance. Since both values are below the 5.38% CR of Control, you can be confident that the Control is the winner.

Gray: This indicates that the test doesn't have definitive results and hasn't reached statistical significance yet. Depending on what you're trying to achieve with your experiment, here are some options you may want to consider:

1. Let It Run Longer: In some instances, you might need to allow the experiment more time to gather a larger sample size and achieve more accurate results.

2. Simplify Variations: If you have too many variations, consider reducing them. For instance, you might bring four variations down to just two or three.

3. Prioritize Brand Consistency: If the results are similar between two variations, choose the one that aligns best with your brand's guidelines.

4. Repeat the Test: Running the same test again can be beneficial for confirming your initial findings. Keep in mind that factors like the time of the year, or fluctuations in website traffic, may affect the end results.

5. Keep It As Is: Occasionally, your original design or strategy may not need any changes and is the most suitable version.

Mistakes to avoid with confidence intervals

It's a common mistake to assume that the 'true value' of what you're estimating will definitely be within the confidence interval. That's not exactly the case.

Instead, a 95% confidence interval means that if you repeated the same study many times, 95 out of 100 of these repeated studies would give results that fall within the range set by your confidence interval.

Also, as you collect more data and sample size increase, it becomes more likely that your confidence interval will include the 'real' value of your estimate. But no matter how much data you have, remember there's always a chance that the 'true value' might not be inside your confidence interval.