A/B test significance calculator.
Is your result real, or luck? Enter the visitors and conversions for each version. It runs in your browser: nothing you type is sent anywhere.
The variant is beating the control by 25.0% (98% sure it's real).
| Version | Rate | 95% range |
|---|---|---|
| Control (A) | 2.00% | 1.74% to 2.29% |
| Variant (B) | 2.50% | 2.21% to 2.82% |
- The difference in conversion rate is +0.50 pts, and the real difference is likely between +0.09 pts and +0.91 pts (95% range).
- To reliably spot a 10% relative change from a 2.00% baseline (95% confidence, 80% power), plan for about 80,680 visitors per version. You have 10,000 in the smaller one. Plan a test's sample size and duration.
- Decide how many visitors you need before you start, then look once. Checking daily and stopping at the first significant result inflates false wins.
Keep the test next to the rest of your work.
Cleo keeps A/B tests in its Testing tab: numbers posted from your site or logged by the day, the same verdict you see here, and a chart that follows the test. It also turns the emails, requests and reports behind a test into clients, tasks and dates. Free to start, no card.
How the calculation works.
The calculator runs a two-sided two-proportion z-test. With n₁ and n₂ visitors, x₁ and x₂ conversions, and p̂ = (x₁ + x₂) ÷ (n₁ + n₂) the pooled rate:
z = (x₂/n₂ − x₁/n₁) ÷ √( p̂ (1 − p̂) (1/n₁ + 1/n₂) ) p-value = 2 × (1 − Φ(|z|)) confidence = 1 − p-value
Worked example: 200 of 10,000 visitors converted on the control and 250 of 10,000 on the variant. The pooled rate is 2.25%, so z = 2.38, the p-value is 0.017 and confidence is 98.3%. The variant is 25.0% ahead (2.00% against 2.50%), and a gap this size would show up by luck in fewer than 2 tests in 100.
Each conversion rate also gets a 95% Wilson score interval, which stays sensible at small counts and near 0% or 100%. The difference between the two rates gets an unpooled 95% range. These are the same calculations as the A/B tests in Cleo's Testing tab. The two use slightly different formulas, so near the 95% line the range can just reach zero while the verdict says "leaning". A z-test is an approximation: with fewer than 5 conversions or non-conversions in a version the calculator says so.
Questions about A/B test results.
What does statistical significance mean in an A/B test?
It means the gap between your two versions is larger than random variation would usually produce. The calculator reports a p-value: the chance of seeing a gap this big if the two versions really convert the same. A p-value under 0.05 is the usual bar for calling a result significant.
What does 95% confidence mean?
Confidence here is 1 minus the p-value. 95% confidence means that if the two versions were truly identical, you would see a gap this large in about 1 test out of 20. The verdict's "98% sure it's real" is this number, read loosely: strictly it is not the probability that the variant is better, and it says nothing about how big the improvement is.
How many visitors do I need?
Plan the number before you start. As a floor, this calculator will not call a result until each version has 100 visitors. To reliably spot a 10% relative change, up or down, with 95% confidence and 80% power, you need n = (1.96 + 0.8416)² × [p₁(1 − p₁) + p₂(1 − p₂)] ÷ (p₂ − p₁)² visitors per version, where p₁ is your current conversion rate and p₂ is that rate raised or lowered by 10% (whichever needs more). The calculator works that out for your control's rate, and the sample-size calculator does the same for any change. Lower conversion rates and smaller changes need more visitors.
Can I stop the test as soon as it turns significant?
Not safely. A test checked every day and stopped at the first significant reading produces false winners far more often than 1 in 20. Decide the sample size first, run until you reach it, then look once. If you must watch a test as it runs, treat early readings as hints.
Which statistical test does the calculator use?
A two-sided two-proportion z-test, with the standard error pooled across both versions, plus Wilson score intervals for each conversion rate and an unpooled 95% range for the difference. It is the same maths as the A/B tests in Cleo's Testing tab, so a result here matches what Cleo shows.
Is my data stored or sent anywhere?
What you type is never sent: the calculation runs in your browser. A share link carries the four numbers in its web address, so opening one sends them to Cleo's web server like any web address, and Cleo removes them from its page analytics. Share a link only if you are happy for the recipient, and your own browser history, to hold those four numbers.