Myth: more traffic means faster tests
Guy on a call told me 'you've got 200k visits, you'll get answers in days.' Cute.
Traffic isn't the bottleneck. Conversions are. To detect a 5% relative lift on a 3% baseline you need roughly 25k+ conversions per variant. Math doesn't care how many lookers you have.
Day 4 of a pricing test, 140k sessions in, my tool said 'significant.' Conversions per arm? 380. I waited. By day 14 the 'winner' had flipped to a loser.
High traffic + low conversion rate = slow tests. Always.
Go to your last test and count conversions per arm, not sessions. Under a few hundred? You learned nothing yet. Keep it running. Report back.
Split Test Street
@SplitTestStreet
Myth: more traffic means faster tests
Этот пост опубликован в Telegram-канале Split Test Street. Подписаться можно по ссылке: @SplitTestStreet.