<b>My winner was fake. Blame returning visitors.</b>

<b>My winner was fake. Blame returning visitors.</b>

Day 6: new nav design, +14% engagement. Looked legit.

Then I segmented new vs returning. New visitors: flat. Returning visitors: +40%.

That's the novelty effect. Your regulars notice the change, poke at it because it's new and shiny, and inflate your numbers. It fades in a week or two.

The real test of any UI change is how NEW visitors react — they have no 'ooh what's different' bias.

Now for any design test I segment by new vs returning, and I weight new-visitor behavior way heavier for permanent decisions.

— Segment your current test: new vs returning
— If the win is all in returning users, wait it out

Go split your test results by visitor type. Report back.