Sizing a test without a stats degree

Sizing a test without a stats degree

Day 1 panic: 'how many visitors do I need?' I'm not a statistician. Here's the back-of-napkin process I actually use:

— Get your current conversion rate. Say it's 4%.
— Decide the smallest lift worth your time. I use 15% relative (so 4% to 4.6%).
— Plug both into any free sample-size calculator. Evan Miller's is the one I keep bookmarked.
— It spits out a number per variant. Say 6,000 each.
— Divide that by your daily traffic to that page. 400/day = 30 days.
— If that runtime is over 6 weeks, your effect's too small to detect. Test something bolder.

That last step kills most of my weak ideas before launch. Good.

Go run your numbers. If runtime > 6 weeks, pick a punchier variant.