Running mobile and desktop as separate tests
Blended a test once where B won overall. Split it after: B crushed on desktop, LOST on mobile. The average lied. Now I check device before trusting any result. Playbook:
— Before launch: pull device split. If mobile is 40%+ of traffic, plan to segment.
— Run the test site-wide but DON'T read the blended number as gospel.
— At analysis, split results: mobile B-vs-A and desktop B-vs-A separately.
— If they disagree, ship per-device. Winner on desktop, original on mobile.
— If your tool can't target by device, that's your next tool upgrade.
The blended winner is the most dangerous number in testing. It hides a loser inside a winner.
Go re-open your last 'winning' test. Split it by device. Bet it disagrees.
Split Test Street
@SplitTestStreet
Running mobile and desktop as separate tests
Этот пост опубликован в Telegram-канале Split Test Street. Подписаться можно по ссылке: @SplitTestStreet.