A/B test significance calculator: is the difference real?
Version B converts better. Is it really better, or is it luck? Enter the visitors and conversions of each version and get the lift, the p-value and a plain answer.
Calculator
Is the difference real?
Visitors and conversions for each version, over the same period.
Significant means the difference is unlikely to be noise. It does not mean it is large, or that it will last: keep an eye on the winner for a few weeks.
My comparison notebook
0/4
Keep up to four named runs, compare what changed and download a report. The notebook stays in this tab’s memory: export it to reopen it later.
Nothing is sent. Closing or reloading this tab clears an unexported notebook.
Images and audio keep their dedicated exports. The notebook records parameters and text results.
How the test works
The calculator runs a two-proportion z-test, the standard test for comparing two conversion rates. It pools the two versions to estimate how much the rates would vary by chance alone, then measures how far apart the two observed rates are in units of that variation:
z = (rate B − rate A) ÷ √(pooled rate × (1 − pooled rate) × (1 ÷ visitors A + 1 ÷ visitors B))
The p-value is the chance of a gap at least that large if the versions were truly the same. Under 0.05, the difference is called significant at 95% confidence.
With the default numbers, version A converts 2.00% of 5,200 visitors and version B 2.71% of 5,100: a 35% lift, with p = 0.018. Significant.
Same lift, different verdicts
Significance depends on the size of the sample as much as on the gap:
| Visitors per version | Conversions A / B | Rates | Lift | p-value | Verdict |
|---|---|---|---|---|---|
| 2,400 | 48 / 62 | 2.00% / 2.58% | 29% | 0.18 | Not significant |
| about 5,000 | 104 / 138 | 2.00% / 2.71% | 35% | 0.018 | Significant |
| 1,700 | 510 / 578 | 30% / 34% | 13% | 0.012 | Significant |
The first test shows a 29% lift and proves nothing: at 2,400 visitors and around 50 conversions each, gaps of that size happen by chance almost one time in five. The third test shows a much smaller lift and is significant, because click-through rates around 30% give many more conversions to work with.
Four mistakes that create false winners
- Stopping at the first significant reading. Rates swing early in a test. Check every day and stop as soon as p dips under 0.05, and you will crown false winners far more often than 5% of the time. Plan the sample with the sample size calculator and read the result once.
- Testing too many versions at once. Each extra comparison is another chance for luck to win. Compare each version with the original and use a stricter threshold.
- Slicing the results until something wins. Mobile only, women only, Tuesday only: with enough slices, one will look significant by chance. Decide the segments before the test.
- Confusing significant and important. With a lot of traffic, a 1% lift can be significant and still not worth the change. Look at the lift and at the cost per purchase, not only at the p-value.
After the verdict
A significant result on click-through is a good reason to switch; confirm it on purchases over the following weeks. A non-significant result is an answer too: the two versions perform about the same, so keep the simpler one and test something bolder next, such as a new angle or a new format. The A/B testing guide explains what to test first on a presell page.
Frequently asked questions
What does a p-value of 0.05 mean?
If the two versions truly performed the same, a difference at least as large as the one you measured would appear by chance 5% of the time. It is not the probability that B is better, and it says nothing about how large the lift is.
What confidence level should I use?
95%, the usual threshold, is right for most page tests. Use a stricter level when a wrong decision is expensive or hard to undo, and plan a larger sample to match.
Can I compare more than two versions?
Compare each version with the original, and be stricter: every extra comparison adds a chance of a false winner. A simple fix is to divide 0.05 by the number of comparisons, so three versions against one original need p < 0.0167.
Why is my result not significant when B looks much better?
Because the sample is small. With a few dozen conversions per version, gaps of 30% appear by chance all the time. Let the test reach the sample size you planned before you read it.
More free tools
- ROAS calculatorReturn on ad spend, cost per purchase and profit after ads, measured against your break-even ROAS.
- Break-even ROAS calculatorThe ROAS, cost per purchase and cost per click at which your ads stop losing money, fees included.
- Presell page calculatorSame budget, two funnels: straight to the product page, or through a presell page. Which one leaves more profit.
- A/B test sample size calculatorVisitors needed per version and days of traffic, from your conversion rate and the lift you want to detect.
- Discount Profit PlannerA discount's sale price, margin, contribution and required order volume, with a reusable scenario sheet.
- The refund pressure ledgerRefunded sales, inventory recovery and return costs in three exportable cohort scenarios.
- Advertorial headline generatorHeadlines for presell pages and advertorials, written for the reader's level of awareness, with copy and CSV export.
- Presell format pickerFive questions about your reader, proof, product and traffic, then the three best-fit presell formats and why.
- CTR, CVR and EPC calculatorRaw counts in, every rate out: CTR, CPM, CPC, CVR, AOV, CPA, ROAS and earnings per click.
- Funnel drop-off calculatorStep-by-step conversion from ad click to purchase, the weakest step and a what-if on one improvement.
- UTM link builderClean UTM links for one ad or a batch, with the page in utm_content and the ad in utm_term, and CSV export.
- Test budget and profit per 1,000 visitorsProfit per 1,000 visitors, break-even conversion and the budget to reach your target orders, in three scenarios.
- Free shipping threshold calculatorCompare lost shipping revenue, carrier costs and contribution targets with exportable scenarios.
- Page weight and speed budgetEstimated network load time on three connections, and the kilobytes you can afford for your goal.
- Customer acquisition cost calculatorCompare fully loaded CAC, acquisition cost lines and target budgets in two exportable scenarios.
- Message-match scorecardTen weighted checks between an ad and its page, a score, the fixes to make first and a word-overlap helper.


