Myth: one test result covers all your traffic

Myth: one test result covers all your traffic

Ran a hero test. Variant won +7% overall. Shipped sitewide. Done, right?

Wrong. I segmented after the fact. Desktop: +19%. Mobile: -6%. The blended +7% hid a mobile loss because desktop was the bigger chunk.

I'd just shipped something that hurt half my traffic.

Aggregate numbers lie by averaging. A winner overall can be a loser for a segment that actually matters to your revenue.

Now I split desktop vs mobile before I call anything — they behave like two different websites.

Go re-open your last winner and split it by device. If the verdict flips on one, ship per-device, not sitewide. Report back.