A/B testing presell pages: read the result clearly

Read a presell A/B test against the plan you wrote before launch. Keep the same visitor basis, check the product handoff, and separate clicks from purchases. A result below your evidence threshold remains uncertain; it does not prove that the pages perform alike.

Illustration of marketers comparing two landing-page variants while monitoring a planned test.
On this page 11 sections
  1. How can a broken tag distort a page test?
  2. What change should the test isolate?
  3. How should visitors enter the comparison?
  4. Which outcome and denominator matter?
  5. What assumptions belong in sample planning?
  6. When should a fixed-horizon test stop?
  7. What does the statistical readout actually mean?
  8. What if the result is still inconclusive?
  9. How do you verify both variants before launch?
  10. What should happen after this test?
  11. Keep the engagement definition fixed across test arms

A presell test needs a defined comparison and a defensible reading of the result. An attractive page and a temporary lead are useful observations, but neither establishes that a change improves the outcome you care about.

The American Statistical Association explains that a p-value does not give the probability that a hypothesis is true and does not measure an effect's practical importance.1 Apply that distinction before reading a dashboard's winner label. This guide keeps planning, intermediate signals and a business decision separate.

ObservationWhat it supportsWhat it does not establish
A route or product handoff failsStop and investigate that operational failureA fair comparison of page performance
More visitors click a buttonA measured difference at that intermediate stepMore purchases or better contribution
A result stays above the chosen p thresholdThe test did not cross that threshold under its assumptionsEquality or equivalence of the pages
A result crosses the chosen thresholdA bounded statistical finding under the test designA probability of winning or a useful business effect

Set the financial hurdle before reading the conversion result. The promotion contribution calculator shows how much extra order volume the discount must produce at your entered costs.

Name every variant consistently with the UTM link builder.

<!-- tool27:add:start --> Before reading gross ROAS as an outcome, compare the same original orders in the returns-adjusted ROAS calculator. Its cohort table separates refunded sales, recovered inventory and reverse logistics, and exports the assumptions with the result. <!-- tool27:add:end -->

How can a broken tag distort a page test?

A comparison needs a stable definition of assignment and outcome before anyone reads a difference. If one variant's entry route changes campaign fields, a source-based report can describe different populations or omit observations. That is a measurement question to investigate, not evidence that the variant won or lost.

Before launch, follow both actual entry routes using the same kind of nonpersonal dummy fields. Record redirects, destination and the intended assignment mechanism for each. Preserve the raw test definitions and mark event collection as unknown until checked with the authorized measurement setup.

The UTM redirect troubleshooting guide explains how to isolate the first changed address. It does not replace random assignment, a prespecified outcome or the uncertainty calculations on this page.

What change should the test isolate?

Choose an effect worth learning about and a design your available traffic can investigate. A small visual edit does not guarantee a small effect, and a large redesign does not guarantee a large one. Treat expected impact as a planning assumption until the comparison provides evidence.

Rank your ideas by how much they change the reader's experience:

RankWhat changesExampleTraffic needed
1The angle: a different problem or readerMorning stiffness vs snoring partnerPlan from the assumed effect
2The formatAdvertorial vs quiz on the same anglePlan from the assumed effect
3The first screen: headline, lede, image"For six years I blamed my mattress" vs a question headlinePlan from the assumed effect
4Length and structureLong story vs short offer pagePlan from the assumed effect
5Proof and offer presentationReviews wall vs one detailed casePlan from the assumed effect
6Details: button text, colors, icons"Get my pillow" vs "Shop now"Plan from the assumed effect

Start at the top. Once the angle and the format are settled, refine the first screen, then the rest. Our guide to marketing angles explains how to build several angles for one product, and one product, six angles shows them side by side.

How should visitors enter the comparison?

A fair test sends comparable visitors to each version at the same time. Use a documented experiment workflow rather than alternating ordinary ad sets and assuming the audiences were comparable.

Record the actual assignment mechanism, eligibility rule and unit: a person, browser or session are different units. Ordinary ad sets with similar budgets do not by themselves establish independent randomized groups. The normal-approximation test below compares two binomial proportions under its stated assumptions.2

To test the page alone:

  1. Duplicate the page into a variant and change one thing (or one big thing, like the format).
  2. Duplicate the ad, and change only the destination URL.
  3. Run both as an A/B test, same audience, same budget, same dates.
  4. Send both pages to the same product page with the same offer.

If you change the ad too, you are testing the pair (ad + page). That is often what you want early on, when you are comparing whole angles: the ad and the page belong together. Just do not attribute the result to the page alone.

There is a second way to test the page alone: keep one ad and one link, and let the link share its visitors between the versions. It is handy when the ad already has comments and shares you do not want to reset with a duplicate, or when the same link gets traffic from several places. In ClickBanger, a split test shares a page's link with up to three other versions, in equal parts, and a visitor keeps seeing the same version as long as they use the same browser and network. Whichever method you use, run every version over the same days.

Calculate each rate from raw counts with the CTR, CVR and EPC calculator.

Which outcome and denominator matter?

Decide what "winning" means before the first visitor arrives, or you will be tempted to pick whichever number looks best at the end.

  • Click-through to the product page is your fast, early signal. It can provide more intermediate events than purchases, but an intermediate signal is not a substitute for the purchase outcome.
  • Purchases per visitor (or cost per purchase) is the decision. A page that excites clicks but disappoints buyers can raise the first and lower the second.
  • Guardrails: average order value and refund requests. A page that over-promises can win on orders and lose on returns.

In ClickBanger, each page and each variant has its own statistics: views, visitors, button clicks, leads, click rate, how far visitors scrolled, which buttons they clicked and the average time on page. During a split test, the statistics of the original page also compare the versions side by side: views, clicks, click rate, the change against the original and a platform-specific statistical readout. Record its definition and method; the label does not establish a probability that one page wins.

Illustration of two marketers reviewing campaign visits, clicks and leads over time.
Review the same measures over time, then compare them across page variants.

Purchases happen on your store, so read them there: with each version tagged, your store's marketing reports can show sessions and sales by UTM parameters. Shopify documents its marketing-report definitions and attribution settings.5 Record the selected model and reporting scope instead of assuming that every report uses the same default. Use the same source and setting for both versions.

The test budget planner shows what it costs to reach your target number of orders.

What assumptions belong in sample planning?

Plan from a baseline event rate, a minimum relative effect and explicit error and power settings. This worksheet fixes a two-sided 5% level, 80% power and equal independent groups; traffic only estimates the time needed for that approximate sample.

Interactive

How many visitors does your test need?

Normal-approximation planning: two-sided 5% level, 80% power and equal independent groups. The default baseline event rate is a fictional example.

Visitors per version1,693
Visitors in total3,386
Days at your traffic9 days

Change the current rate to 2% to see why tests on purchases need so much more traffic than tests on click-through.

Bookmark the standalone sample size calculator to plan each test. Two habits follow from the numbers:

  • Test bold changes. Halving the lift you want to detect roughly quadruples the visitors you need.
  • Keep a click-only finding at the click stage. If purchases are too rare for the planned comparison, record the purchase outcome as unresolved rather than calling the page a sales winner.

When should a fixed-horizon test stop?

Live readouts can tempt you to stop at the first appealing result. In a fixed-horizon analysis, that changes the plan assumed by the ordinary p-value. Evan Miller explains the repeated-testing error with worked examples.3 Record the stopping method before launch; this worksheet does not implement a sequential procedure.

Myth"B was ahead after two days, so B wins."
Reality

Early leads come and go. With small samples, random swings are large, and they are exactly what stopping early locks in.

Myth"We'll just run it until it's significant."
Reality

Run long enough and noise will cross the line at some point. Decide the size first; an inconclusive result is a valid outcome. That does not prove equivalence.

Myth"Weekends don't matter for our product."
Reality

People browse and buy differently through the week. Plan a window that covers the traffic patterns relevant to this comparison. A calendar week alone does not guarantee adequate information.

Myth"We can tweak the budget mid-test to save money."
Reality

Changing budgets, audiences or the offer during a test changes who sees each version. Change nothing until it ends.

What does the statistical readout actually mean?

When the planned sample is reached, enter the counts for a large-sample, two-sided normal approximation. Each eligible visitor contributes one binary outcome within a group; repeated clicks or several orders from that visitor are not additional independent observations. The worksheet is not an exact small-sample or sequential test.2

Interactive

Read a fixed-horizon two-proportion comparison

Unique eligible visitors and visitors with the chosen event, counted once per group. Fictional counts: 1,700 visitors per group, with 510 and 578 event visitors.

Lift of B over A13.3%
Rate A30.0%
Rate B34.0%
Two-sided approximate p-value0.012

The p-value describes a result relative to the specified equality model. It is not a probability of winning. This worksheet withholds its normal approximation below its stated count guard; it does not perform a small-sample or sequential test.

The same test is available on its own page: the A/B test significance calculator. Then write down the decision, even when it is "uncertain under this plan". A test log (hypothesis, versions, dates, sample, result, decision) is how a team stops re-testing the same idea every six months.

What if the result is still inconclusive?

Keep the uncertainty visible. A failure to cross a chosen threshold can accompany a useful effect, a harmful effect or an estimate too imprecise to settle the question; it is not proof that the pages are alike.1

Choosing the existing page can still make operational sense. Write that reason explicitly: a cheaper implementation, a broken variant or insufficient information under the plan. Those are different decisions from a demonstrated performance tie. Do not repeatedly extend a fixed-horizon test merely to obtain a smaller p-value.3

For a bounded layout hypothesis, use the presell CTA placement guide. Keep that single change separate from an offer or angle test, and retain the same product destination while assessing the page.

How do you verify both variants before launch?

Complete the same task on both published routes before interpreting performance. A wrong product destination or an unusable quiz result creates an operational failure, not an informative page comparison.

Chrome's Device Mode approximates mobile layout and performance.6 The English demonstration below helps reproduce viewport and orientation conditions. It is not a statistical tutorial, and emulation does not establish that a route ran on a physical phone.

Use the English viewport and orientation walkthrough to check both variant routes. It does not explain statistical inference or predict a winning page. Video: Simulate mobile devices with Device Mode #DevToolsTips, by Chrome for Developers, on YouTube. It loads from YouTube only when you press play.
Illustration of two coworkers checking one difference between paper mobile page outlines.
Record the changed element and complete the same task on both routes. The drawn exercise is fictional.

What should happen after this test?

After the planned readout, record what the comparison supports and what remains uncertain. A roadmap orders possible questions; it does not supply a winner for every one:

  1. Angles. Two or three angles, each as a full ad + page pair. Keep the best one.
  2. Format. The winning angle as an advertorial and as a quiz (or a listicle). Keep the best one.
  3. First screen. Two headlines and ledes on the winning page.
  4. Length. The long page against a shorter cut, on the same angle; see long-form vs short-form.
  5. Proof and offer presentation, once the traffic allows smaller effects to be detected.

A supported operational choice can become a new control. Keep the decision basis and any unresolved purchase outcome in the record. To see what a winning page is worth in money, run your numbers in our presell page math guide.

Before you launch a test

9 items

Keep the engagement definition fixed across test arms

Record the analytics settings alongside the page versions before comparing two presell variants. If one arm has a different event definition, observation window or reporting boundary, the apparent difference may not answer the content question in the test plan. A valid comparison needs a stated unit and a consistent way to count it.

Do not promote whichever variant shows the lowest bounce rate if the planned primary outcome was completed orders per acquired visitor. Engagement, product handoffs and orders can move differently. The GA4 bounce-rate review explains why one minus bounce rate cannot replace a handoff rate. Keep the original outcome definition visible when reading the diagnostic metrics.

If a measurement setting changes during the experiment, mark when it happened and investigate which records are affected. Do not silently combine the periods or invent a historical adjustment. Preserve the page versions and raw eligible counts so a later reviewer can see what the comparison actually contains.

RecordUseful readingDo not substitute
Same page questionKeep the primary outcome explicitA winner selected after reading the result
Event definition changedRecord the time and affected recordsComparable engagement across both periods
Diagnostic metrics disagreeInspect the route and nested countsA reason to choose the favorable percentage

Session-engagement terminology follows Google Analytics definition.7

<!-- shipping1112:add:start --> Before putting a delivery incentive into the offer, use the free shipping threshold calculator to separate the shipping charge you give up from the carrier cost you still pay. <!-- shipping1112:add:end -->

<!-- cac1115:add:start --> To compare acquisition economics beyond an attributed purchase, use the customer acquisition cost calculator with distinct first-time buyers and a consistent allocation of sales and marketing expenses. <!-- cac1115:add:end -->

Frequently asked questions

How much traffic does a presell A/B test need?

Plan from the baseline rate, minimum effect worth detecting, allocation, significance level and power. The worksheet is a normal approximation for equal independent groups. Its traffic estimate does not prove that the assumed effect exists.

Can I test two pages with the same ad?

A single ad and a page-level random split can isolate a page change. An ad-platform experiment is another design. Record the actual assignment method and unit; two ordinary ad sets do not automatically create a randomized comparison.

How long should a test run?

Use the stopping and observation rule chosen for the method before launch. Consider weekly traffic patterns and delayed outcomes, but do not treat one calendar week as a universal guarantee. Operational checks can still stop a broken route.

Does a nonsignificant result mean the pages are equal?

Failure to reject equality does not demonstrate equivalence. The sample may be too small or the estimate too uncertain. You may keep the control for operational reasons while recording that performance equivalence was not established.

Is one minus the p-value my chance of winning?

A p-value and its complement do not give the probability that a hypothesis is true. The readout here omits that misleading confidence label. Review the method, effect and uncertainty rather than treating a percentage as a verdict.

Which reporting source should I use?

Choose the same source, event definitions, attribution setting and observation window for both versions. Preserve raw counts and tracking gaps. An observed click difference does not by itself establish more purchases or more profit.

Sources

  1. Statement on statistical significance and p-values, American Statistical Association (March 7, 2016; accessed October 24, 2026)
  2. Difference of proportion hypothesis test, NIST Dataplot (Accessed October 24, 2026)
  3. How Not To Run an A/B Test, Evan Miller (2010)
  4. URL builders: collect campaign data with custom URLs, Google Analytics Help
  5. Marketing reports, Shopify Help Center
  6. Simulate mobile devices with Device Mode, Chrome for Developers (Accessed October 24, 2026)
  7. Engagement rate and bounce rate, Google Analytics Help

ClickBanger Editorial We build presell pages for e-commerce brands and spend our days reading ads, landers and test results. Questions or a correction? Write to us.

Keep reading

All articles

Testing & growth12 min read

Presell page math: when the extra step pays for itself

A presell funnel pays when its orders leave more contribution than the same traffic sent straight to the store. Multiply click-through by the purchase rate, then check costs. Keep the same traffic basis, price and cost rules. More clicks or a bigger order alone do not prove a win.

From product URL to presell page in minutes

Advertorial, listicle, quiz, comparison and 12 more formats, written from your product page and ready to edit.