Handle revenue outliers before testing RPV: one $40k B2B order can fake a 6% 'win'.
Revenue-per-visitor has a fat tail. One whale dominates the mean and your conclusion.
— Plot the revenue distribution per arm. Spot the extreme tail.
— Pre-register a treatment: winsorize at the 99th percentile (cap extremes) or analyze with a rank-based test.
— Re-run the result with and without the cap. If the win vanishes, it was one order.
— Report median order value and conversion alongside mean RPV.
A mean built on a single transaction isn't a result, it's an anecdote with a decimal. Read the number, not the story. [cap=p99]
Conversion Lab Notes
@ConversionLabNotes
Handle revenue outliers before testing RPV: one $40k B2B order can fake a 6% 'win'.
Этот пост опубликован в Telegram-канале Conversion Lab Notes. Подписаться можно по ссылке: @ConversionLabNotes.