I killed 9 winners to find 1 — because I judged on 40 clicks

I killed 9 winners to find 1 — because I judged on 40 clicks

The setup. Testing 10 creative/angle combos on push traffic, $20 each. Impatient as ever. Illustrative numbers, statistically embarrassing.

The move. Killed any combo without a conversion by ~40 clicks to "save budget." Felt disciplined. Felt fast.

The numbers. The offer's true CR was around 1.5% — roughly 1 conversion per 65 clicks. At a 40-click kill threshold, a perfectly average winner had a ~55% chance of showing zero conversions purely by variance. I was coin-flipping my winners into the trash. Of 10 combos I killed 9; the survivor scaled to 1.6 ROI. Statistically, 3-4 of the dead ones were probably just as good — I never gave them the sample to prove it.

The lesson. Kill thresholds below your offer's conversion window aren't discipline, they're noise-trading. If it takes ~65 clicks to expect one conversion, killing at 40 means you're judging on a sample that literally cannot show the signal yet.

What I'd do differently. Set the kill threshold at 2-3x the expected click-to-conversion interval, funded upfront. For a 1.5% CR offer that's ~130-200 clicks before a no-conversion verdict means anything. Testing costs more than I want — but killing winners on variance costs far more, I just couldn't see it on the invoice.