One-tailed vs two-tailed: the setting that fakes wins
Day 6, my test hit 'significant.' Then I noticed the tool was set to one-tailed.
Quick translation: one-tailed only asks 'is B better than A?' Two-tailed asks 'is B different, better OR worse?' One-tailed needs less data to call a win, so vendors love defaulting to it. Feels great. It's also more likely to crown a false winner.
Flipped to two-tailed. My 'significant' result evaporated. Saved me from shipping junk.
The tradeoff: two-tailed is stricter and slower, but it's honest. One-tailed is faster and flatters you.
Use two-tailed by default. Only go one-tailed if you genuinely don't care about a 'worse' outcome (rare).
Go check this exact setting in your tool right now. Report back what you find.
Split Test Street
@SplitTestStreet
One-tailed vs two-tailed: the setting that fakes wins
Этот пост опубликован в Telegram-канале Split Test Street. Подписаться можно по ссылке: @SplitTestStreet.