Case #011: The day I cut 9 winners on 4 conversions

Case #011: The day I cut 9 winners on 4 conversions
Launched with 12 ad sets, $20 each, $240 total for the day. By hour 6 I'd spent $130 and pulled 4 conversions. Itchy, I started "optimizing" — paused the 9 ad sets with zero conversions, dumped budget into the 3 that had fired.

It fell apart immediately. The next day my 3 "winners" produced almost nothing, and the ad sets I'd murdered? Two of them were the ones the data would have crowned if I'd waited. I'd made nine statistical decisions on a sample of four. That's not optimization, that's reading tea leaves. $1,100 spent across the flailing, $600 back. A 45% loss born entirely of acting too early on too little.

The leak was treating tiny numbers as signal. With 4 conversions, the gap between my "best" and "worst" ad set was pure noise. You need enough conversions per variant before a difference means anything — for me that's a minimum of 15 to 20 conversions on a variant before it gets a verdict.

The fix: I now set a data threshold per ad set before launch and physically forbid myself from touching anything until it's crossed. Patience as a rule, not a virtue.

Relaunched, hands off until each set hit 15 conversions. The real winners emerged — and two were sets I'd killed last time. $1,500 spent, $2,250 back. 50% ROI.

— The lesson: optimizing on thin data isn't optimizing, it's amplifying noise — set a conversion threshold per variant and don't touch a thing until it's crossed.