<b>I almost shipped a fake winner</b>

<b>I almost shipped a fake winner</b>

This week I ran a hero test. Variant B up 14%. I was reaching for the ship button.

Then I checked the split. 52/48. Supposed to be 50/50.

That gap is sample ratio mismatch. Means something upstream broke the randomization — a redirect, a cache, a bot bucket. The 14% was probably an artifact, not a win.

Killed the test. Found a cached page serving B to returning users only. Of course B looked better. It was talking to warm traffic.

Day 1 lesson I relearn every quarter: before you read the lift, read the counts.

— Pull your variant traffic numbers right now
— If the split is off by more than ~1%, your result is garbage

Go check the SRM on your live test. Report back.