Myth: an early lift proves the variant is better

Myth: an early lift proves the variant is better

Day 2, my redesigned variant was up +18%. I was already writing the win-post in my head.

Then it bled. Day 5: +9%. Day 10: +2%. Day 14: flat.

That's novelty effect. Returning users notice the new thing, poke it, click more — because it's NEW, not because it's better. The bump fades as novelty wears off.

This is why early reads on tests with lots of returning visitors are basically lies. You're measuring curiosity, not preference.

Rule: ignore the first few days on any test with a strong returning-user base. Let novelty burn off.

Got a test up big early? Don't call it. Wait for the curve to flatten. Report back.