A/B Test Sample Size and Duration Guide

A/B testing sounds like the end of guesswork: show half the visitors version A, half version B, and let the numbers decide. On a busy store that works. On a site with a few thousand visitors a month, the numbers often cannot decide anything, and a test stopped at the first exciting result produces a confident answer that is simply noise. Before you start, it pays to know what drives A/B test sample size, how long a test really needs, and what to do when your traffic will not support one.

What drives A/B test sample size

Four inputs decide how many visitors a test needs:

• Baseline conversion rate. The rarer the action, the more visitors are needed to see a change in it. A test measured on "add to cart" needs fewer visitors than one measured on completed purchases.
• Minimum detectable effect. The smallest improvement you care about detecting. This input matters most: halving the minimum detectable effect roughly quadruples the sample you need.
• Significance level. How willing you are to be fooled by chance. A common convention is a 5 percent false positive rate.
• Statistical power. How likely the test is to spot a real effect of the chosen size; 80 percent is the usual convention.

Put your own numbers into a sample size calculator before building anything. If the answer is more visitors than the page receives in two months, the test is not worth running in that form. What counts in the end is conversions per variant, not visitors: a test with thousands of visitors but only a handful of purchases in each version has very little to say.

How long to run an A/B test

Run it until the planned sample is reached, and always for whole weeks. Visitors behave differently on a Monday morning than on a Saturday night, so a test that runs from Tuesday to Friday measures only a slice of your audience. One or two full business cycles is a sensible minimum even on busy sites. Very long tests have problems of their own: people clear cookies and are counted twice, and seasonal changes or new campaigns alter the mix of visitors.

Plan the end date in advance and stick to it. Stopping a test early because the dashboard shows a winner is the most common way to get a false result: if you check every day and stop the first time the result looks significant, the real chance of a false positive is much higher than the level you set. Some testing tools use sequential methods designed for repeated looks; if yours does not, decide the sample first and read the verdict once.

What statistical significance does and does not mean

Statistical significance says that if the two versions truly performed the same, a difference as large as the one observed would rarely appear by chance. It does not tell you how likely version B is to be better, and it says nothing about whether the difference is large enough to matter to the business. Read it together with the size of the effect and its confidence interval.

Check for sample ratio mismatch

If traffic is meant to be split evenly and one version received noticeably more visitors than the other, something is broken: a redirect that fails in some browsers, a bot hitting one URL, tracking that fires in only one variant. This sample ratio mismatch invalidates the result, however good it looks. Many testing tools flag it; otherwise, compare the visitor counts with a simple chi-square test.

A/B testing and SEO

Google's guidance on minimizing the search impact of website testing comes down to a few rules. Do not show Googlebot a different version from the one visitors get, which counts as cloaking. If a variant lives on a separate URL, point its canonical tag to the original page. Use a temporary 302 redirect, not a permanent 301, when a test redirects visitors. And end the test once you have your answer, then update the original page. Followed this way, A/B testing and SEO get along fine.

Tools after Google Optimize

Google Optimize was shut down on September 30, 2023, and many small sites have not run a test since. The Google Optimize alternatives fall into three groups: dedicated testing platforms such as VWO or Optimizely, with visual editors and built-in statistics; feature-flag and experimentation services used by development teams, which switch variants in the code; and testing features built into some e-commerce and landing page platforms. Whatever you choose, check three things: whether the script makes the page flicker or slow down, whether it respects your cookie consent setup, and whether it sends results to GA4, where they can be checked against your orders.

What low-traffic websites can do instead

For low-traffic websites, formal testing is often the slowest route to an answer. Better options:

• Test bigger changes. A new offer, a shorter form or a different page structure produces a larger effect than a button color, and larger effects need far smaller samples.
• Watch real sessions. Heatmaps and recordings show where people hesitate, and a handful of moderated sessions with real customers often reveals the main problems.
• Ask your sales staff. The questions people phone or email about are usually questions the page failed to answer.
• Compare periods carefully. Make one change at a time and compare equal periods, noting campaigns, holidays and anything else that changed.

Whatever the method, start each change with a written test hypothesis: "Because we saw this in the data, we believe that changing this for these visitors will produce this result, and we will measure it with this metric." It prevents random redesigns and makes every result, win or lose, something you learn from. Our conversion rate optimization service works the same way on smaller sites: with too little traffic for a reliable test, we make one well-researched change at a time and compare periods.

Frequently asked questions: a/b test sample size

Can I run several A/B tests at the same time?

On different pages or different steps of the funnel, usually yes. Two tests on the same page or the same audience can interfere with each other, so either make them mutually exclusive or combine them into one multivariate test.

Should mobile and desktop be tested separately?

Look at both segments, but decide which segments you will analyze before the test starts. Slicing the results after the fact until one segment "wins" is another way to find noise.

What if the test shows no difference?

That is a result. It tells you the change, if it helps at all, helps less than your minimum detectable effect. Keep whichever version is simpler to maintain and test something bolder next.

Not sure whether your traffic can support testing? Our conversion rate optimization and A/B testing service starts with funnel data and heatmaps, and recommends formal tests only where the numbers can carry them. It works best on top of accurate GA4 tracking. Send us your site address and rough monthly traffic, and we can look at where to start.