Free Calculator

A/B Test Calculator

Calculate statistical significance for your A/B tests or plan sample sizes for new experiments.

Test Results

Control (A)

Variant (B)

Enter your test results to analyze significance.

Understanding A/B Testing Statistics

Statistical Significance

A result is significant when there's less than 5% probability it happened by chance (p < 0.05). This means 95% confidence.

Sample Size

Smaller effects require larger samples to detect. A 1% improvement needs more data than a 20% improvement.

When to Stop

Run tests until you reach significance OR your planned sample size. Stopping early or peeking increases false positives.

Minimum Detectable Effect

The smallest change worth detecting. If a 5% improvement isn't meaningful to your business, plan for detecting 10%+.

Common A/B Testing Mistakes

  • ✗
    Stopping tests early: Ending a test when it looks good dramatically increases false positive rates. Run to your planned sample size or significance, not before.
  • ✗
    Peeking at results: Checking results daily and stopping when p < 0.05 inflates error rates. Set your sample size upfront and check only when complete.
  • ✗
    Testing too many variations: Testing 10 headlines multiplies your chance of false positives. Stick to A/B or use multivariate testing with appropriate corrections.
  • ✗
    Running tests too short: A weekend test misses weekday behavior. Run for at least one full business cycle (usually 1-2 weeks minimum).
  • ✗
    Ignoring segment differences: A variant that wins overall might lose for mobile users. Analyze results by device, traffic source, and customer type.

Frequently Asked Questions

What is statistical significance in A/B testing?

Statistical significance indicates the probability that your test results are not due to random chance. A result is typically considered significant when p < 0.05 (95% confidence), meaning there is less than 5% probability the difference happened by chance.

How long should I run an A/B test?

Run your test until you reach statistical significance OR your pre-calculated sample size, whichever comes first. Never stop early just because results look good - this increases false positives. Most tests need 1-4 weeks depending on traffic volume.

How do I calculate sample size for an A/B test?

Sample size depends on: your baseline conversion rate, the minimum effect you want to detect, desired statistical power (typically 80%), and confidence level (typically 95%). Smaller effects require larger samples. Use our calculator to determine the exact number.

What is minimum detectable effect (MDE)?

MDE is the smallest improvement you want your test to detect. If a 5% relative improvement is not meaningful to your business, set MDE higher (like 10-20%). Higher MDE means smaller required sample sizes and faster tests.

Why is my A/B test not reaching significance?

Common reasons: insufficient traffic/sample size, the actual effect is smaller than expected, or there truly is no meaningful difference. Continue running until you reach your planned sample size, then accept the results even if not significant.