Create an account to save sources and durable progress.
A Smart Bear · Jason Cohen
articleprincipleSource excerpt
About A Smart BearA/B testing tools often lie about whether something is "statistically significant." Here's an extremely simple, mathematically sound formula to compute it for yourself.
This is a limited feed-provided excerpt, not the full original work.
Apollo layer
Treat an A/B testing tool’s “statistically significant” label as a claim to verify, not a decision to accept automatically. Use a transparent, mathematically sound calculation before acting on test results.
Misleading significance signals can turn random variation into product or growth decisions, wasting time and sending the company toward changes that may not produce real gains.
For your next A/B test, require the team to document the sample sizes, observed outcomes, significance calculation, and decision rule rather than relying only on the tool’s verdict.
Private notes
© Apollo Advisors° ABN 34 346 108 139