Sample-size calculator vs 'it looks like a winner'

Sample-size calculator vs 'it looks like a winner'
Day 3, variant's up 18%, n=140. My gut screamed SHIP IT.
My gut is a liar at n=140.
Ran it through Evan Miller's sample-size calculator first. To detect an 18% lift with my baseline, I needed ~2,200 per arm. I had 140. That 18% was pure noise.
Waited. By n=2,200 the lift settled at +3%. Still a win, but a real one, not a fantasy.
The tradeoff people skip: a calculator costs 90 seconds and saves you from shipping noise to prod. There's no downside except your ego.
Two tools I keep bookmarked: Evan Miller's calc for fixed-horizon, and a Bayesian one for ongoing reads.
Go calculate your current test's required n. Bet you're under it. Report back.