Is the difference between two groups real, or could it be chance? Compare two rates (sign-ups, clicks, defects) or two averages (order value, time on task) and get the p-value, a confidence interval and a plain-language verdict.
Free, no sign-up. All maths runs in your browser; nothing is sent anywhere.
How statistical significance works
Significant means a difference this large would rarely appear if the two groups were really the same — at 95% confidence, “rarely” is less than 5% of the time (p < 0.05). It says nothing about whether the difference is big enough to matter; check the confidence interval and effect size for that.
Rates or averages? Use the Conversion rates tab when each person either did or didn’t do something. Use Averages for amounts — revenue, minutes, scores.
Skewed data (a few huge orders) makes averages unreliable. Paste the raw values in the Averages tab to also get a Mann–Whitney test, which compares the groups by rank instead.
Many comparisons inflate false positives: test 20 metrics and one will look significant by luck. Decide the main metric up front.
Methods. Conversion rates: two-sided pooled two-proportion z-test (equivalent to a 2×2 chi-square), Wald interval for the difference, delta-method interval for the relative lift. Averages: Welch’s t-test and Cohen’s d; Mann–Whitney U on raw values. Sample size: normal approximation for two proportions. All checked against SciPy.
Know when a metric really changed — without running the numbers by hand.
QuantumLayers connects to your database, analytics or spreadsheets, runs significance tests like these across every segment, and monitors your key metrics so you’re emailed when something statistically shifts.