A Smart Bear · Jason Cohen
articleprincipleEasy statistics for A/B testing and hamsters
18 February 2024Profit
Source excerpt
About A Smart BearA/B testing tools often lie about whether something is "statistically significant." Here's an extremely simple, mathematically sound formula to compute it for yourself.
This is a limited feed-provided excerpt, not the full original work.
Apollo layer
What a founder can learn
Founder takeaway
Treat an A/B testing tool’s “statistically significant” label as a claim to verify, not a decision to accept automatically. Use a transparent, mathematically sound calculation before acting on test results.
Why it matters
Misleading significance signals can turn random variation into product or growth decisions, wasting time and sending the company toward changes that may not produce real gains.
Relevant guides
Put it to work
For your next A/B test, require the team to document the sample sizes, observed outcomes, significance calculation, and decision rule rather than relying only on the tool’s verdict.
Private notes