The A/B testing mistake that makes you ship losing variants

The A/B testing mistake that makes you ship losing variants

Deep-dive on the most expensive review-site optimization error: stopping a test the moment it hits 95% 'significance'. You peek daily, see variant B winning, call it, roll it out — and conversions don't move or drop. You committed the peeking fallacy: checking repeatedly inflates false positives massively. On a review site with maybe 2,000 sessions/week to a money page, you almost never have the sample size you think.

The fix:
— Fix sample size and duration BEFORE starting (use any sample-size calculator: baseline rate, min detectable effect, power 0.8). For affiliate CTRs at ~3-5% baseline, expect to need weeks, not days
— Run a minimum of two full business cycles (2 weeks) regardless of significance, to absorb weekday/weekend behavior
— Use tools that support sequential testing (some VWO/AB Tasty modes) if you must peek

Gotcha nobody mentions: affiliate conversion (the actual sale) happens on the merchant's site days later — your tool only sees the click, so you're optimizing outbound CTR, which can rise while real revenue falls. Tie the test to network-side EPC, not just clicks.

Verdict on DIY A/B on low-traffic review pages: 3/10 — usually underpowered theater.

Bottom line: pre-commit sample size, run two weeks minimum, and judge on revenue per click, not CTR.