Understanding Bayesian A/B Testing

Traditional A/B testing (frequentist approach) requires you to wait until a pre-determined sample size is reached before checking results. Bayesian statistics takes a different approach by continually updating probabilities as new data comes in, allowing you to make decisions faster and with more intuitive results.

Key Benefits of Bayesian Testing

• Provides straightforward, intuitive answers (e.g., "There's a 95% probability the variant is better")
• Allows you to check results at any time without statistical penalties
• Incorporates prior knowledge to potentially reach conclusions faster
• Works effectively with smaller sample sizes
• Gives clear decision-making thresholds based on probability

Understanding the Calculator Results

Probability of Improvement

This represents the likelihood that your variant outperforms the control. A value of 95% means there's a 95% chance the variant is truly better. This is typically the primary metric for making decisions—most statisticians consider 95% probability sufficient evidence to conclude the variant is superior.

Expected Lift

The average expected improvement of the variant over the control. For example, a 10% lift on a page with a 2% conversion rate means the variant is expected to achieve approximately 2.2% conversion. This helps quantify the practical impact of implementing the change.

95% HDI (Highest Density Interval)

This range shows where 95% of the most likely lift values fall. If this interval includes zero or negative values (e.g., -2% to +15%), it means there's still a possibility the variant could perform worse than the control. A range entirely above zero (e.g., +3% to +12%) provides stronger evidence of improvement.

Planning Tools Explained

The calculator offers two important planning tools to help design effective tests:

Time to Significance Calculator

This estimates how long your test needs to run to detect a specific uplift with your desired confidence level. By entering your daily traffic, baseline conversion rate, and expected improvement, you can plan test durations accurately and avoid ending tests too early or running them unnecessarily long.

Minimum Detectable Effect Calculator

This calculates the smallest improvement you can reliably detect given your traffic volume and test duration. It helps set realistic expectations and prevents testing for effects too small to be measured with your available data, allowing you to focus on meaningful changes.

Practical Applications

The Bayesian calculator is especially valuable for:

• Comparing different versions of landing pages, emails, or ads
• Testing UI/UX changes on websites and checkout flows
• Evaluating promotional offers and pricing strategies
• Making data-driven decisions with limited traffic
• Optimizing conversion funnels step by step
• Determining which test variations to prioritize
• Quantifying the impact of changes with statistical rigor

Understanding Prior Knowledge

The "Prior Knowledge Strength" setting is a key Bayesian concept. It allows you to incorporate existing knowledge about conversion rates into your analysis. "None" uses an uninformative prior (no assumptions), while "Strong" applies more skepticism toward extreme results. This feature is particularly useful when you have historical data or industry benchmarks that can inform your expectations.