My biggest "winner" was just luck regressing.

My biggest "winner" was just luck regressing.

Day 3, variant up 40%. Insane number. I knew it was too good but I wanted it.

Day 10: +6%. Day 18: +2% and not significant.

That day-3 monster was regression to the mean. Small samples produce wild swings. Early extreme leads almost always shrink toward reality as data piles in.

If your lift is shrinking every day, that's not your variant fading. That's the noise washing out.

— Ignore early extreme leads, they're noise
— Trust the number after you hit your pre-set sample size
— Big early lift + steady decline = regression, not magic

Go look at your test's lift trend by day. Shrinking toward zero? Report back.