<b>My winning variant died on day 9. Here's why.</b>
New nav menu. First week: clicks up 22%. I called it. Marked it shipped.
Day 9 it flattened. Day 14 it was <i>below</i> control.
Novelty effect. Regulars clicked the new thing because it was new, not because it was better. Once the shine wore off, real behavior showed up.
The fix: segment by new vs returning visitors. New users showed +1%. Returners drove the fake lift.
— Always run UI tests past two full weeks
— Split results: first-time vs repeat
— If the lift lives only in returners, it's novelty, not value
Go re-check your last UI win past day 14. Report back.
Split Test Street
@SplitTestStreet
<b>My winning variant died on day 9. Here's why.</b>
Этот пост опубликован в Telegram-канале Split Test Street. Подписаться можно по ссылке: @SplitTestStreet.