"My split test found a winner"

"My split test found a winner"

Myth: "Lander A beat Lander B at 40 conversions to 28, so A wins — kill B."

Dating operators call winners on counts that wouldn't survive a freshman stats class. At a few dozen conversions, a 40-vs-28 gap is comfortably inside noise — run the same test again tomorrow and B might "win." You then pour budget behind a coin flip you mistook for an insight, and when it regresses to the mean you blame fatigue.

The vertical makes this worse: dating conversions are bursty and whale-skewed, so one user's deposit cluster can swing a small sample hard. The variance you're measuring is often a single anomalous session, not a creative difference.

It's also why people "can't replicate their winner" — there was never a real effect to replicate. They optimized into randomness and built a theory on top of it.

Either run to a sample that clears significance for your conversion rate, or be honest that you're making a judgment call, not reading a result.

Reality: A 40-to-28 split at low volume is noise wearing a result's clothes, and scaling it just buys you a slower way to learn that.