Running a creative A/B test that survives iGaming variance, in 5 steps

Running a creative A/B test that survives iGaming variance, in 5 steps

iGaming conversion is noisy — high-variance players wreck small tests. Structure for signal.

▸ Step 1 — Decide the metric upfront: FTD rate for top-of-funnel creative, not NGR (too lagged and variance-heavy for a creative test).
— Step 2 — Size the test. At a 2% FTD rate, detecting a lift to 2.4% needs roughly ~6,000 clicks per variant for confidence. Under-powered tests 'find' winners that are noise.
— Step 3 — Split traffic simultaneously, not sequentially — a Tuesday-vs-Saturday split confounds the creative with the day.
— Step 4 — Hold GEO and device constant across variants; a creative 'win' driven by a GEO skew isn't a win.
— Step 5 — Don't peek-and-stop. Set the sample size and read it once at the end, or you'll call early noise a winner.

Benchmark of the day: across ~20 creative tests, ~35% of 'winners' called before reaching sample size reversed when run to full power.