Conversion Lab Notes
Conversion Lab Notes
@ConversionLabNotes

Stop peeking: checking a test daily and stopping at first significance inflates your false-positive rate from

Stop peeking: checking a test daily and stopping at first significance inflates your false-positive rate from 5% to ~30%.
The fix is procedural, not statistical willpower.
— Option A: fix the sample size in advance, look only once at the end. Simplest, cheapest.
— Option B: use a sequential testing method (mSPRT or always-valid p-values) built for continuous monitoring.
— Never: eyeball the dashboard and call it when the line crosses 95%.
Fixed-horizon p-values assume one look. Twenty looks at 95% means roughly one in three 'wins' is noise.
The test ends when the plan said, not when you got the answer you wanted. Read the number, not the story. [α inflation 5%→30%]
Этот пост опубликован в Telegram-канале Conversion Lab Notes. Подписаться можно по ссылке: @ConversionLabNotes.
tech

Свежие посты в категории «Tech Infrastructure»

Все каналы категории →

start

Готовы запустить рекламу через сеть public.tg?

Новый оффер, продукт, GEO, кейс, событие или партнёрский запуск — соберём маршрут под задачу и отдадим медиаплан.

Telegram для медиаплана: @AFFtop_connect. Быстрый тест: $20 за канал, $1000 за пакет по сети.