Myth: checking a test early doesn't hurt if you wait for significance. False — peeking quadruples your false-positive rate.
Fixed-horizon tests assume one look. Peek daily for 2 weeks and your true α drifts from 5% toward ~20–25%.
If you must monitor live, use sequential methods (mSPRT or always-valid p-values) built to survive continuous looking.
Fixed-horizon ▮▮ vs sequential-safe ▮▮▮▮▮▮
Read the number, not the story. [14 looks · α inflated 4–5×]
Conversion Lab Notes
@ConversionLabNotes
Myth: checking a test early doesn't hurt if you wait for significance. False — peeking quadruples your false-p
This post was published in the Telegram channel Conversion Lab Notes. Subscribe here: @ConversionLabNotes.