<b>Two winning tests. Ran them together. Both lost.</b>
Rookie move I still made last month. Test A (new hero) won +7%. Test B (new pricing toggle) won +5%. Obviously stack them, right?
Stacked, the page did <i>worse</i> than control.
Interaction effect. The new hero promised "simple," the new pricing toggle screamed "complicated." Apart they worked. Together they fought.
Lifts don't add up. They collide.
— Don't assume winners stack
— Re-test the combo as its own variant
— Or run a multivariate test if traffic allows
Go check if your last two "wins" are even live on the same page together. Mine were canceling out. Report back.
Split Test Street
@SplitTestStreet
<b>Two winning tests. Ran them together. Both lost.</b>
Этот пост опубликован в Telegram-канале Split Test Street. Подписаться можно по ссылке: @SplitTestStreet.