Multivariate vs running A/B tests back-to-back
Wanted to test headline (3) x button color (2) x image (2). That's an MVT temptation. 12 combos.
Did the math: my traffic needed 14 weeks to power all 12 cells. Fourteen. I'd be dead.
So I ran sequential A/Bs instead. Headline test week 1 (winner). Button test week 2 on the winning headline. Image week 3. Done in 3 weeks, not 14.
The catch: sequential misses interaction effects. Maybe red button only wins WITH headline B. MVT would've caught that.
My rule: MVT only when you have monster traffic AND suspect interactions. Otherwise sequential, always.
Go count how many weeks your next MVT actually needs. The number will scare you straight. Report back.
Split Test Street
@SplitTestStreet
Multivariate vs running A/B tests back-to-back
Этот пост опубликован в Telegram-канале Split Test Street. Подписаться можно по ссылке: @SplitTestStreet.