Myth: running more concurrent tests means you learn faster. False — overlap inflates false positives.
With 5 simultaneous tests at α=0.05 (5% false-positive threshold per test), the chance at least one fires by chance is 1−0.95⁵ = 23%.
The fix isn't fewer tests, it's pre-registering a metric per test and treating cross-test interaction as noise until isolated.
Serial discipline ▮▮▮▮▮▮ vs concurrent sprawl ▮▮
Read the number, not the story. [family-wise error · 5 tests]
Conversion Lab Notes
@ConversionLabNotes
Myth: running more concurrent tests means you learn faster. False — overlap inflates false positives.
Этот пост опубликован в Telegram-канале Conversion Lab Notes. Подписаться можно по ссылке: @ConversionLabNotes.