Conversion Lab Notes
Conversion Lab Notes
@ConversionLabNotes

Run a sample-ratio mismatch check before reading any result: ~6% of A/B tests fail it silently.

SRM (your 50/50 split isn't actually 50/50) invalidates the whole test, not just one metric.
The 4-step audit:
— Count actual users per arm. Expected 50/50, observed maybe 50.8/49.2.
— Run a chi-square goodness-of-fit on those raw counts.
— If p < 0.001, you have SRM. Do not trust the uplift.
— Trace the cause: redirect latency, bot filtering applied post-split, or a broken variant erroring out before assignment.
A 'winning' variant is often just one that crashed for slow users. Read the number, not the story. [SRM p<0.001 = halt]
This post was published in the Telegram channel Conversion Lab Notes. Subscribe here: @ConversionLabNotes.
tech

Latest posts in “Tools and infrastructure”

All channels in this category →

start

Готовы запустить рекламу через сеть public.tg?

Новый оффер, продукт, GEO, кейс, событие или партнёрский запуск — соберём маршрут под задачу и отдадим медиаплан.

Telegram для медиаплана: @AFFtop_connect. Быстрый тест: $20 за канал, $1000 за пакет по сети.