Multivariate (MVT) tests combos: 3 headlines × 3 buttons × 2 images = 18 variants. Sounds powerful. It's a traffic-eating monster.
18 arms means you need ~18x the sample. On normal traffic you'll get a result around the heat death of the universe.
What actually works:
— Run A/B. One change.
— Win, ship it, then A/B the next element.
— Sequential beats simultaneous when traffic is mortal.
MVT is for sites doing millions of sessions. You are not that site yet.
Go kill any test with more than 2-3 variants. One question at a time. Report back.
Split Test Street
@SplitTestStreet
"A/B test or multivariate?" For beginners: A/B. Always.
This post was published in the Telegram channel Split Test Street (@SplitTestStreet). All its posts are on the channel page.
Open on Telegram