When is a split test actually done? My stopping rules
I used to call winners at 30 clicks and waste money on noise. Here are the rules I use now to stop a test honestly.
The setup
Two landing pages, A vs B, $14 payout offer.
The move
Three stopping conditions, whichever hits first:
— Conversions, not clicks: wait until the leading variant has at least 25-30 conversions, not 30 clicks. Conversions are the unit that pays you.
— Spend floor: each variant must spend at least 3x payout before any verdict. Below that it's a coin flip.
— Gap test: if after the floor the CPA difference is under 15%, call it a tie and pick the cheaper-to-run one. Tiny gaps don't survive scale.
The numbers
LP-A looked like a winner at 8 conversions (CPA $9). By 28 conversions it had reverted to $12.50, dead even with LP-B. I'd have scaled the wrong page.
The lesson
Early leaders regress. The first 10 conversions are mostly luck.
What I'd do differently
Write the stopping rule before launch so I'm not tempted to call it early when I'm impatient. Figures illustrative.
Arb Files
@ArbFiles