Myth: checking the test early doesn't hurt anything

Myth: checking the test early doesn't hurt anything

'I just peek, I don't stop it.' Liar. You peek, you see green, you call it. We all do.

Here's the dirty part: if you check daily and stop the first time you cross 95%, your real false-positive rate isn't 5%. It's closer to 30%. Every peek is another roll of the dice.

That's why your 'winners' don't replicate. You stopped on a lucky day.

Fix it without fancy stats: set a sample size up front. Don't even LOOK at the result button until you hit it. Calendar reminder, not a daily refresh.

Go set a fixed sample size on your live test right now and hide the dashboard until then. Report back.