"My test hit 95% significance, ship it." [DEPENDS]

"My test hit 95% significance, ship it." [DEPENDS]

The "95% significance" ritual got cargo-culted from frequentist stats into tools where it doesn't even mean what people think.

The mechanism: classic significance assumes you set a sample size upfront and peek only once. But everyone peeks daily and stops when the line turns green — which inflates false positives massively. That 95% is a fiction if you stopped early because it looked good.

Where frequentist fixed-horizon fits: you can commit to a sample size, wait, and check once. Disciplined teams, big traffic.

Where Bayesian / sequential testing fits: you peek continuously and want to stop the moment there's enough evidence — most real-world tools default to this for a reason.

The rule: peeking constantly? Use a method built for peeking. Don't borrow frequentist thresholds and ignore frequentist rules.

The nuance the dashboards skip: "95%" is meaningless without the stopping rule behind it. Ask what happens to your false-positive rate when you check every morning — that's the number that decides if you should ship.