A/B test significance calculator: is the difference real?

Version B converts better. Is it really better, or is it luck? Enter the visitors and conversions of each version and get the lift, the p-value and a plain answer.

Calculator

Is the difference real?

Visitors and conversions for each version, over the same period.

Lift of B over A—
Rate A—
Rate B—
p-value—
Confidence—

Significant means the difference is unlikely to be noise. It does not mean it is large, or that it will last: keep an eye on the winner for a few weeks.

How the test works

The calculator runs a two-proportion z-test, the standard test for comparing two conversion rates. It pools the two versions to estimate how much the rates would vary by chance alone, then measures how far apart the two observed rates are in units of that variation:

z = (rate B − rate A) ÷ √(pooled rate × (1 − pooled rate) × (1 ÷ visitors A + 1 ÷ visitors B))

The p-value is the chance of a gap at least that large if the versions were truly the same. Under 0.05, the difference is called significant at 95% confidence.

With the default numbers, version A converts 2.00% of 5,200 visitors and version B 2.71% of 5,100: a 35% lift, with p = 0.018. Significant.

Same lift, different verdicts

Significance depends on the size of the sample as much as on the gap:

Visitors per versionConversions A / BRatesLiftp-valueVerdict
2,40048 / 622.00% / 2.58%29%0.18Not significant
about 5,000104 / 1382.00% / 2.71%35%0.018Significant
1,700510 / 57830% / 34%13%0.012Significant

The first test shows a 29% lift and proves nothing: at 2,400 visitors and around 50 conversions each, gaps of that size happen by chance almost one time in five. The third test shows a much smaller lift and is significant, because click-through rates around 30% give many more conversions to work with.

Four mistakes that create false winners

  • Stopping at the first significant reading. Rates swing early in a test. Check every day and stop as soon as p dips under 0.05, and you will crown false winners far more often than 5% of the time. Plan the sample with the sample size calculator and read the result once.
  • Testing too many versions at once. Each extra comparison is another chance for luck to win. Compare each version with the original and use a stricter threshold.
  • Slicing the results until something wins. Mobile only, women only, Tuesday only: with enough slices, one will look significant by chance. Decide the segments before the test.
  • Confusing significant and important. With a lot of traffic, a 1% lift can be significant and still not worth the change. Look at the lift and at the cost per purchase, not only at the p-value.

After the verdict

A significant result on click-through is a good reason to switch; confirm it on purchases over the following weeks. A non-significant result is an answer too: the two versions perform about the same, so keep the simpler one and test something bolder next, such as a new angle or a new format. The A/B testing guide explains what to test first on a presell page.

Frequently asked questions

What does a p-value of 0.05 mean?

If the two versions truly performed the same, a difference at least as large as the one you measured would appear by chance 5% of the time. It is not the probability that B is better, and it says nothing about how large the lift is.

What confidence level should I use?

95%, the usual threshold, is right for most page tests. Use a stricter level when a wrong decision is expensive or hard to undo, and plan a larger sample to match.

Can I compare more than two versions?

Compare each version with the original, and be stricter: every extra comparison adds a chance of a false winner. A simple fix is to divide 0.05 by the number of comparisons, so three versions against one original need p < 0.0167.

Why is my result not significant when B looks much better?

Because the sample is small. With a few dozen conversions per version, gaps of 30% appear by chance all the time. Let the test reach the sample size you planned before you read it.

More free tools

Go further

Guides that use these numbers

All articles

Testing & growth13 min read

A/B testing presell pages: read the result clearly

Read a presell A/B test against the plan you wrote before launch. Keep the same visitor basis, check the product handoff, and separate clicks from purchases. A result below your evidence threshold remains uncertain; it does not prove that the pages perform alike.

Testing & growth12 min read

Presell page math: when the extra step pays for itself

A presell funnel pays when its orders leave more contribution than the same traffic sent straight to the store. Multiply click-through by the purchase rate, then check costs. Keep the same traffic basis, price and cost rules. More clicks or a bigger order alone do not prove a win.

From product URL to presell page in minutes

Advertorial, listicle, quiz, comparison and more, written from your product page and ready to edit.