A/B Testing Statistics: Sample Size, Significance, and the Mistakes That Produce False Winners
An A/B test compares two versions of something, splits traffic between them, and uses statistics to decide whether the difference in results is real. The procedure is simple. The statistics are where most tests go wrong. This guide covers how to calculate the sample size before you start, what a p-value and a confidence interval actually tell you, why checking results early inflates false positives, which test fits which metric, and how to analyze exported experiment data in QuantumLayers.
What an A/B Test Measures
An A/B test is a randomized controlled experiment. Visitors, users, or customers are assigned at random to a control (A) or a variant (B), and the same outcome metric is recorded for both groups. Random assignment is what makes the method valuable. Because nothing other than chance decides who sees which version, any systematic difference between the groups can be attributed to the change itself. That is the one setting in everyday analytics where a correlation can be read as causation, a distinction covered in more depth in Correlation vs Causation.
Random assignment also guarantees that the two groups will differ by some amount even when the change does nothing. Conversion rates of 5.0 and 5.3 percent might reflect a real improvement or ordinary variation between two random samples. The statistical part of A/B testing exists to separate those two explanations, and it can only do that reliably when the test is planned before it starts.
Four Numbers to Set Before the Test Starts
Every A/B test rests on four quantities. Three are choices, and the fourth, sample size, follows from them.
Significance level (alpha). The false positive rate you accept: the probability of declaring a winner when the variant has no effect. The usual choice is 0.05. Statistical power. The probability of detecting an effect that really exists, usually set at 80 percent. One minus power is the false negative rate. Minimum detectable effect (MDE). The smallest improvement worth detecting, expressed relative to the baseline. An MDE of 10 percent on a 5 percent conversion rate means the test is designed to catch a move to 5.5 percent. Sample size. The number of users needed in each group to reach the chosen power at the chosen alpha for that MDE.
| Outcome | Variant has no real effect | Variant has a real effect |
|---|---|---|
| Test declares a winner | False positive (Type I error), probability set by alpha | Correct detection, probability set by power |
| Test finds no difference | Correct, probability 1 minus alpha | False negative (Type II error), probability 1 minus power |
How to Calculate A/B Test Sample Size
For a conversion-rate test, the standard sample size formula for comparing two proportions takes the baseline rate, the MDE, alpha, and power. The table below gives the result for a two-sided test at alpha 0.05 and 80 percent power. The figures are users per group, so the total traffic required is double.
| Baseline conversion rate | 5% relative lift | 10% relative lift | 20% relative lift |
|---|---|---|---|
| 2% | 315,206 | 80,682 | 21,109 |
| 5% | 122,124 | 31,234 | 8,158 |
| 10% | 57,763 | 14,751 | 3,841 |
Two patterns stand out. Halving the MDE roughly quadruples the sample, because required sample size scales with the inverse square of the effect. Lower baseline rates also need far more traffic, since rare events carry less information per user. A site with 4,000 weekly visitors and a 2 percent conversion rate would need about ten months to detect a 10 percent lift. That is not a reason to skip testing. It is a reason to test bigger changes, pick a metric higher in the funnel, or accept that small optimizations cannot be verified at that traffic level.
Run time should also cover at least one full weekly cycle, and ideally two, even when the sample size is reached sooner. Weekday and weekend visitors often behave differently, and a test that runs Tuesday to Thursday describes midweek traffic only.
Statistical Significance and the P-Value
The p-value answers one narrow question: if the variant had no effect at all, how often would random assignment alone produce a difference at least as large as the one observed? A p-value of 0.03 means a result this extreme would occur about 3 percent of the time under that assumption. When the p-value falls below alpha, the result is called statistically significant.
The p-value is frequently misread. It is not the probability that the variant is better, and it is not the probability that the result is a fluke. It says nothing about how large the improvement is. A p-value of 0.001 on a million users can correspond to a lift too small to matter, and a p-value of 0.08 on a small test can hide a large effect that the sample was too small to confirm. The American Statistical Association’s statement on p-values makes the same point: significance should never be the only basis for a decision.
Report the Confidence Interval, Not Only the Verdict
A confidence interval gives the range of effect sizes that are consistent with the data. It carries the significance verdict inside it, since a 95 percent interval that excludes zero corresponds to p below 0.05, and it adds the information the p-value leaves out: how large the effect plausibly is.
Consider a test with 20,000 users per group. Control converts 1,000 users (5.00 percent) and the variant converts 1,090 (5.45 percent). The two-proportion z-test gives p = 0.043, so the result is significant. The 95 percent interval for the difference runs from 0.01 to 0.89 percentage points, which is a relative lift of anywhere from 0.3 percent to 17.7 percent.
A report that says “variant wins, +9 percent” conceals all of this. A report that gives the interval shows that Test A’s improvement could be negligible, that Test B’s estimate is solid, and that Test C was underpowered rather than proof that the change does nothing. Effect size and its uncertainty should appear in every A/B test readout.
The Peeking Problem
The most common way A/B tests produce false winners is checking the results repeatedly and stopping as soon as the p-value drops below 0.05. The 5 percent false positive rate holds only for a single analysis at the planned sample size. Each additional look is another chance for random fluctuation to cross the threshold, and those chances accumulate.
To measure the effect, we simulated 40,000 A/A tests, in which both groups receive the identical experience with a 5 percent conversion rate and 20,000 users each. Any declared winner is therefore a false positive. Each simulated test was checked at evenly spaced points and stopped at the first significant result.
A single analysis kept the false positive rate at the intended 5 percent. Checking ten times quadrupled it to 19 percent, and checking fifty times, roughly what happens when someone looks at a dashboard daily over a long test, produced a false winner in about one test out of three. None of those variants did anything.
There are two legitimate ways around this. The first is to fix the sample size in advance and analyze once, looking at interim data only to catch broken tracking or a severe drop. The second is to use a method designed for continuous monitoring, such as group sequential designs with alpha spending or always-valid sequential tests, which raise the bar at each look so the overall error rate stays at 5 percent. What does not work is applying a fixed-sample test repeatedly and stopping on the first good result.
Choosing the Right Test for Your Metric
The correct statistical test depends on the type of metric, not on the tool that happens to be in front of you. Revenue per user is the case that causes the most trouble: most users spend nothing, a few spend a lot, and a single large order can swing the mean of a whole group.
| Metric type | Examples | Standard test | Effect size to report |
|---|---|---|---|
| Binary outcome | Converted, clicked, signed up, churned | Two-proportion z-test or chi-square | Absolute and relative lift, Cramér’s V |
| Continuous, roughly symmetric | Session duration (after capping), pages per visit | Welch’s t-test, or ANOVA for 3+ variants | Mean difference, Cohen’s d |
| Continuous, heavily skewed | Revenue per user, order value | Mann-Whitney U, or bootstrap on the mean | Median difference, rank-based effect |
| Three or more variants | A/B/C/D layouts, pricing tiers | ANOVA or Kruskal-Wallis, then pairwise tests | Eta-squared, largest pairwise d |
Testing several variants at once multiplies the number of comparisons, and with it the chance of a false positive. Four variants compared pairwise produce six tests. The same applies when one variant is evaluated on many metrics or many segments. Corrections such as the Benjamini-Hochberg procedure adjust the p-values to account for the number of tests run.
Five More Mistakes That Invalidate A/B Test Results
Sample ratio mismatch. If traffic was meant to split 50/50, the groups should be close to equal. A chi-square test on the group counts checks this. With 100,000 users, a split of 50,400 to 49,600 gives p = 0.011, which points to a problem in assignment or tracking, such as a redirect that loses some users in one arm or a bot filter that treats the variants differently. A test with a sample ratio mismatch should be investigated before its outcome is read, because the groups are no longer comparable.
Segment fishing after the fact. A test that shows no overall effect will almost always show a “winning” segment if enough are examined: mobile users in Ontario, returning visitors on Safari. With twenty segments, one is expected to reach p below 0.05 by chance. Segment results should be treated as hypotheses for a follow-up test unless the segments were specified before launch and the results are corrected for multiple comparisons.
Novelty and primacy effects. Returning users may click a new design because it is new, or avoid it because it is unfamiliar. Both effects fade. Plotting the daily lift over the course of the test shows whether the effect is stable or decaying toward zero.
The wrong unit of analysis. If users are randomized but the analysis counts sessions or page views, one heavy user contributes dozens of correlated observations. The test treats them as independent, standard errors shrink, and p-values become too small. This is the same pseudoreplication problem described in Joining Datasets Without Breaking Them. Analyze at the level at which assignment happened.
Ignoring guardrail metrics. A checkout change that raises conversion while increasing refunds or support tickets has not improved the business. Decide on a small set of guardrail metrics in advance and check that none of them moved in the wrong direction.
An A/B Testing Checklist
| Stage | Check |
|---|---|
| Before launch | One primary metric, a list of guardrail metrics, alpha, power, MDE, sample size per group, planned run time of at least one full week, and any segments to be analyzed |
| During the test | Monitor only for broken tracking or severe harm; do not stop early on a significant result unless a sequential method was planned |
| Before reading results | Sample ratio mismatch check, unit of analysis matches unit of randomization, daily lift plotted for novelty effects |
| Reading results | Test chosen for the metric type, p-value with confidence interval and effect size, correction applied for multiple variants, metrics, or segments |
| After shipping | Watch the primary and guardrail metrics for a change in level after rollout |
Analyzing A/B Test Data in QuantumLayers
Experimentation platforms handle assignment and the headline readout. The deeper analysis, across segments, secondary metrics, and skewed revenue data, usually happens on an export. QuantumLayers is built for that second step. Load the experiment data at the user level, with a variant column, the outcome metrics, and any attributes you want to break results down by, from a CSV, a Google Sheet, or a direct connection to the warehouse where the events land. Setup for each source is covered in Connecting a Database to QuantumLayers.
The variant column is categorical, so the statistical pipeline routes it to the tests in the table above without any configuration. Against each numeric metric it runs a one-way ANOVA and reports eta-squared and Cohen’s d beside the p-value, so the size of the difference is always visible. When ANOVA finds nothing, which is common on skewed revenue columns, Kruskal-Wallis and Mann-Whitney U run as rank-based fallbacks that a few large orders cannot dominate. A binary outcome stored as a category, such as converted or not converted, is tested against the variant with chi-square and Cramér’s V. The same chi-square logic on the variant counts is a quick way to confirm the split before reading anything else.
The segment fishing problem is handled by default. Every hypothesis test in a run, across all metrics and attribute columns, goes through Benjamini-Hochberg correction before it is reported, so a segment effect only appears if it survives adjustment for the number of comparisons made. Results that would have looked significant on their own but do not survive correction are removed rather than shown with a caveat.
After a winning variant ships, the analysis continues on the production data. A Monitor on the relevant dataset reruns the pipeline on each sync and sends an alert by email, Slack, or webhook when a structural break or other statistically real change appears, which is how a fading novelty effect or a guardrail regression after rollout shows up. The same analysis is available to AI agents through the QuantumLayers MCP server, so an agent asked to summarize an experiment works from tested and corrected results rather than computing its own statistics.
Plan the Test, Then Trust the Result
Almost every A/B testing error comes from a decision made after the data started arriving: stopping when the numbers looked good, adding a segment that happened to win, switching to a metric that moved. The defense is to make those decisions in advance. Calculate the sample size, fix the metrics and segments, analyze once at the planned point, and report the effect with its confidence interval. A test run that way can still return an unexciting answer, and that answer is far more useful than a winner that disappears after launch.
This post is part of the QuantumLayers blog series on making data-driven decisions you can trust. For the tests used throughout, see Beyond the Basics: Advanced Statistical Tests That Separate Signal from Noise. For why significance alone can mislead on large datasets, read Inside the QuantumLayers Statistical Pipeline. Analyze your own experiment data at www.quantumlayers.com.