"That A/B test proved the change works." [DEPENDS]
It spread because dashboards declare winners with confident green arrows, and "95% significance" sounds final.
The boring truth: most landing-page tests are underpowered. With low traffic, a test "reaching significance" early is usually noise caught at a lucky peak — peeking and stopping inflates false positives badly. A 20% lift on 200 conversions has confidence intervals wide enough to drive a truck through. You didn't prove the button color; you observed a wobble.
Significance without sample size is a horoscope with a p-value.
What the advice skipped: calculate required sample size before you start, run full business cycles, and ignore the dashboard until then. Most "wins" don't replicate.
Above The Fold Heresy
@AboveTheFoldHeresy
"That A/B test proved the change works." [DEPENDS]
Этот пост опубликован в Telegram-канале Above The Fold Heresy. Подписаться можно по ссылке: @AboveTheFoldHeresy.