<b>I ran a test that mathematically couldn't win.</b>

<b>I ran a test that mathematically couldn't win.</b>

Landing page, 600 visitors/week, 2% conversion. I tested a new headline.

Day 11, no significance. Day 20, still nothing. I kept waiting like an idiot.

Then I did the math I should've done day zero. To detect a 10% relative lift on a 2% base, I needed ~26,000 visitors per arm. At 300/week/arm that's <i>87 weeks</i>.

The test was dead on arrival. I was waiting for a result the traffic could never produce.

— Calc your minimum detectable effect BEFORE launch
— Low traffic? Test big swings or test further up the funnel

Go run a sample-size calc on your live test. If it's &gt;6 weeks, kill it. Report back.