Myth: bigger sample is always better, run it longer

Myth: bigger sample is always better, run it longer

'Just let it run another month to be safe.' Sounds responsible. It's not free.

Two costs nobody mentions. One: opportunity cost — every extra week you run a test, half your traffic is sitting on the losing variant. Two: over-long tests drift. Seasonality, promo calendar, traffic-source changes creep in and contaminate your clean comparison.

Day 12 my test hit its pre-calculated sample size with a clear winner. I let it run two more weeks 'to be sure.' Black Friday traffic hit, the verdict wobbled, and I trusted the result less, not more.

Hit your number, call it.

Go calculate your sample size BEFORE the test, and stop the day you hit it. Not a week later. Report back.