HomeFree toolsA/B test calculator
A/B test calculator
Enter the visitors and conversions for version A and version B. The calculator shows each conversion rate, the uplift, and whether the difference is statistically significant at your chosen confidence level.
Worked example.
A Sheffield bike shop tested a new product page layout (version B) against its current page (version A), sending 5,000 visitors to each.
- Version A visitors
- 5,000
- Version A conversions
- 200
- Version B visitors
- 5,000
- Version B conversions
- 250
- Confidence level
- 95%
Version A converts at 4.00% and version B at 5.00%, an uplift of 25.0%. The z-score is 2.41 and the p-value is 0.016, which is below 0.05, so the result is significant at 95% confidence (it would not pass at 99%, where the p-value needs to be below 0.01).
How it works.
The maths and the rules behind the tool, so you can check it yourself.
The calculator first works out each version's conversion rate and the uplift, which is how much better (or worse) version B did compared with version A.
It then runs a two-sided two-proportion z-test. This asks: if the two versions really performed the same, how likely is it that you would see a gap this big just by chance? That likelihood is the p-value. Two-sided means it checks for B doing better or worse than A.
If the p-value is below 1 minus your confidence level (below 0.05 at 95%), the result is significant and the difference is unlikely to be random noise. A result that is not significant means the test has not shown a difference yet, which is not the same as proving there is none.
Tips.
Small changes that make the numbers more useful.
- Decide your sample size and test length before you start, then let the test run to the end. Checking every day and stopping the moment it looks significant makes false wins much more likely.
- Run tests for at least one full week, ideally two, so weekday and weekend visitors are both included.
- Change one thing at a time where you can. If version B has a new headline, new photos and a new button, a win will not tell you which change did the work.
- Split traffic at random and run both versions at the same time. Comparing this month with last month is not an A/B test, because seasons, promotions and ad spend all change.
- Look at the size of the uplift, not just significance. With very large samples, a tiny difference can be significant but still not worth the cost of changing your site.
Questions, answered.
What does statistically significant mean in an A/B test?
It means the difference between your versions is unlikely to be down to chance alone. At 95% confidence, a significant result has a p-value below 0.05: if A and B truly performed the same, you would see a gap this big less than 5% of the time. It does not guarantee the uplift will hold at exactly the same size.
What confidence level should I use?
95% is the most common choice and a sensible default. Use 99% when a wrong decision would be costly or hard to undo. Use 90% for low-risk changes where you are happy to accept more false positives in exchange for faster answers.
How many visitors do I need for an A/B test?
It depends on your current conversion rate and how big a change you want to detect. Small uplifts on low conversion rates need many thousands of visitors per version, while big changes show up sooner. If your site has little traffic, test bolder changes so the difference is easier to spot.
What is a p-value?
The p-value is the probability of seeing a difference at least as big as yours if the two versions actually converted at the same rate. A small p-value means your result would be surprising if there were no real difference. It is not the probability that version B is better.
What should I do if my A/B test is not significant?
First check the test ran for its planned length with the visitors you planned for. If it did, the change probably made little difference, so keep whichever version you prefer for other reasons and test something bolder next. Avoid extending the test again and again until it happens to cross the line.
More free tools.
Related calculators and generators. All free tools
Custom websites and landing pages built for speed and sales.
Want help with the numbers?
Tell us what you need. Charlie will come back with a plan and a price, usually within one working day.