"We're too small to A/B test, so we just ship our best guess." [MOSTLY MYTH]
Low-traffic teams swing between two bad poles: pretend a 200-visit test is significant, or skip testing and call HiPPO opinion "data." Both spread because nobody taught the middle option.
The mechanism: under ~1,000 conversions per variant per month, a classic A/B can't separate a real 10% lift from noise — it'll run for months or never resolve. Shipping on a hunch, though, throws away the learning entirely.
The tool that actually fits small traffic: bigger swings tested as before/after with eyes open. Don't test button colors — test a fundamentally different offer, headline, or page structure where the effect is large enough to see through the noise. Pair it with five user interviews to explain the move.
The rule: low traffic → big bets, big expected effects, qualitative to confirm. Not micro-tests, not blind guessing.
The nuance: "can't A/B test" doesn't mean "can't learn." It means small effects are invisible to you — so stop optimizing buttons and change something that matters.
Above The Fold Heresy
@AboveTheFoldHeresy
"We're too small to A/B test, so we just ship our best guess." [MOSTLY MYTH]
Этот пост опубликован в Telegram-канале Above The Fold Heresy. Подписаться можно по ссылке: @AboveTheFoldHeresy.