Myth: wait for 95% significance or it doesn't count

Myth: wait for 95% significance or it doesn't count

Everybody parrots 95%. I did too. Then I learned where it came from: a stats convention from agriculture papers in the 1920s. Not your funnel.

95% means you accept a 1-in-20 false positive. That's a business choice, not a law.

Day 6 on a checkout test. 92% confidence, +8% on revenue per visitor. I shipped it. Held in the next 4 weeks.

What actually matters: did you hit your pre-set sample size, and is the lift bigger than the noise.

Stop treating 95% like a fire alarm. Pick a threshold that matches your risk.

Go open your last 'inconclusive' test. If it hit sample size at 90%+ and the loser had no upside, ship the winner. Report back.