<b>I shipped a winner. The split was lying to me.</b>

<b>I shipped a winner. The split was lying to me.</b>

Day 4 of a checkout test. Variant B up 18%. I almost popped the champagne.

Then I checked the traffic split. Supposed to be 50/50. Was 54/46.

That's a sample ratio mismatch. Means the randomizer or a redirect is leaking users. Your data is contaminated before the first conversion fires.

The 18% wasn't real. It was a broken bucketing assignment eating mobile Safari.

— Pull your raw assignment counts, not conversions
— Run a chi-square on the split
— If p &lt; 0.01, kill the test, don't trust it

Go check the actual user counts in your last "winner." Bet it wasn't 50/50. Report back.