A/B test sample size calculator.
How many visitors does your test need, and how long will it take? Enter your baseline rate and the smallest change that matters. It runs in your browser: nothing you type is sent anywhere.
You need about 80,680 visitors per version, 161,360 in all.
- Decide this number before you start, run until each version reaches it, then look once. When it is done, check the result with the significance calculator.
- A test this long drifts: the season changes, cookies expire and the page itself gets edited. Aim for a bigger change or more traffic.
Plan the test, then keep it next to your work.
Cleo keeps A/B tests in its Testing tab: numbers posted from your site or logged by the day, a target number of visitors, and a verdict in plain words. It also turns the emails, requests and reports behind a test into clients, tasks and dates. Free to start, no card.
How the calculation works.
The calculator uses the standard sample size for comparing two conversion rates, p₁ (your baseline) and p₂ (the rate you want to be able to detect), with the variance of each rate counted separately:
n = (z₁₋α/2 + z₁₋β)² × [ p₁(1 − p₁) + p₂(1 − p₂) ] ÷ (p₂ − p₁)²
Here n is the visitors needed in each version, z₁₋α/2 is 1.645, 1.960 or 2.576 for 90%, 95% or 99% confidence, and z₁₋β is 0.8416 or 1.2816 for 80% or 90% power. The result is rounded up, and never below 100 visitors a version, the least a result can be called on. Worked example: a 20% baseline and a 25% target at 95% confidence and 80% power give n = (1.960 + 0.8416)² × (0.16 + 0.1875) ÷ 0.05² = 1,091.006, which rounds up to 1,092 visitors per version (some calculators print 1,091).
The duration is the total visitors (n times the number of versions) divided by your daily visitors, shown in days and in whole weeks. The second mode runs the formula backwards: it searches for the smallest rate p₂ above your baseline whose sample size is no more than the visitors you have. This is the same formula as the sample target shown for A/B tests in Cleo's Testing tab, which sizes for a 10% relative change, up or down, whichever needs more, at 95% confidence and 80% power.
Questions about sample size.
How many visitors do I need for an A/B test?
It depends on four things: your current conversion rate, the smallest change you want to be able to detect, how sure you want to be, and how likely you want to be to catch a real change. At a 2% baseline, spotting a 10% relative rise (2.0% to 2.2%) at 95% confidence and 80% power takes about 80,680 visitors per version. A bigger change, or a higher baseline rate, needs far fewer.
What is the minimum detectable effect?
It is the smallest change in conversion rate you want the test to reliably notice. Choose it before you start: the change you would actually act on, not the change you hope for. Halving it roughly quadruples the visitors you need.
What do confidence and power mean?
Confidence (95% is usual) controls false wins: if the two versions were truly identical, a 95% confidence test calls a winner about 1 time in 20. Power (80% is usual) controls misses: if the real change is as big as the one you entered, an 80% power test catches it 4 times in 5. Raising either one needs more visitors.
Should I enter a relative or an absolute change?
Either describes the same thing. A relative change is a share of your baseline: from 2% to 2.2% is a 10% relative rise. An absolute change is in percentage points: the same move is 0.2 points. Use whichever your team talks in; the calculator converts.
How long should an A/B test run?
Divide the total visitors you need by the visitors you get each day, then round up to whole weeks so every day of the week is covered equally. Do not stop early because the result looks good: decide the sample size first, run until you reach it, then look once.
What if I do not have enough traffic?
Test a bigger change, test on a busier page, measure a step earlier in the funnel where more people convert (a click rather than a purchase), or run fewer versions. The calculator's second mode shows the smallest change your traffic can reliably detect.
Does it adjust for more than two versions?
No. It gives the visitors each version needs, and the total is that number times the versions. Testing several versions against one control raises the chance that one wins by luck, and this calculator does not correct for that.
Is my data stored or sent anywhere?
What you type is never sent: the calculation runs in your browser. A share link carries your inputs in its web address, so opening one sends them to Cleo's web server like any web address, and Cleo removes them from its page analytics. Share a link only if you are happy for the recipient, and your own browser history, to hold those numbers.