Geo holdout tests measure true incremental lift; standard A/B can't when treatment leaks across users.
Use geo when the change isn't user-randomizable — pricing, brand campaigns, shared-cart features.
Cost: geo tests have tiny sample (you have ~50 regions, not 50k users), so MDE is large and you need big effects to detect anything.
— A/B ▮▮▮▮▮▮ precision, leaks on spillover
— Geo ▮▮ precision, clean on spillover
Reserve geo for non-splittable changes; default to user-level A/B otherwise.
Read the number, not the story. [design choice · spillover-driven]
Conversion Lab Notes
@ConversionLabNotes
Geo holdout tests measure true incremental lift; standard A/B can't when treatment leaks across users.
Этот пост опубликован в Telegram-канале Conversion Lab Notes. Подписаться можно по ссылке: @ConversionLabNotes.