Did the variant really win, or is it noise? Enter your results to get a plain-language verdict, the p-value, a confidence interval for the lift, and a check that traffic was split the way you intended — or plan how many visitors a test needs before you start.
Free, no sign-up. All maths runs in your browser; nothing is sent anywhere.
What the numbers mean
p-value — if A and B were truly identical, how often would a gap at least this big show up by chance? Below 0.05 is the usual bar for “significant” at 95% confidence.
Confidence interval — the range of lifts that fit your data. If it spans zero, the data can’t rule out “no effect” — even if the point estimate looks great.
Sample ratio mismatch — if you meant to split traffic 50:50 but got 52:48 on a big sample, something in the assignment or tracking is broken, and the result can’t be trusted whatever the p-value says.
Don’t peek and stop. Checking every day and stopping the first time p dips under 0.05 inflates false positives a lot. Decide the sample size up front and evaluate once.
Methods. Conversion rates: two-sided pooled two-proportion z-test (equivalent to a 2×2 chi-square), Wald interval for the difference, delta-method interval for the relative lift. Averages: Welch’s t-test and Cohen’s d; Mann–Whitney U on raw values. Sample size: normal approximation for two proportions. All checked against SciPy.
Know when a metric really changed — without running the numbers by hand.
QuantumLayers connects to your database, analytics or spreadsheets, runs significance tests like these across every segment, and monitors your key metrics so you’re emailed when something statistically shifts.