Myth: a winning variant helps all visitors. False — pooled wins routinely hide segment reversals (Simpson's paradox).
A +4% aggregate can be +12% on mobile and −6% on desktop. The blend looks like a win; half your traffic lost.
Pre-declare 2–3 segments (device, new vs returning, traffic source). Don't slice 20 post-hoc — that's just peeking by another name.
Read the number, not the story. [Simpson's paradox · 2–3 pre-set segments]
Conversion Lab Notes
@ConversionLabNotes
Myth: a winning variant helps all visitors. False — pooled wins routinely hide segment reversals (Simpson's pa
This post was published in the Telegram channel Conversion Lab Notes (@ConversionLabNotes). All its posts are on the channel page.
Open on Telegram