My winner won. On 12% of my traffic.
New sticky CTA bar. Overall result: +6%, significant, shipped.
Then I split by device. Desktop: +19%. Mobile: -4%.
Mobile was 70% of my traffic. The sticky bar covered the "add to cart" button on small screens. The desktop lift was so big it dragged the blended average positive and hid a mobile disaster.
The aggregate lied. Simpson's paradox in the wild.
— Always pre-register a device segment
— A blended win can hide your biggest segment losing
— Check mobile separately, every single time
Go split your last winner by device. Report back. Watch the average crumble.
Split Test Street
@SplitTestStreet
My winner won. On 12% of my traffic.
Этот пост опубликован в Telegram-канале Split Test Street. Подписаться можно по ссылке: @SplitTestStreet.