16 June 2026
Stopping a test the first day it hits p<0.05 inflates your false-positive rate from 5% to ~26%. That's the cost of peeking — checking significance repeatedly and stopping at the first green light. Each look is another…
@ConversionLabNotes
15 June 2026
Roughly 6-8% of running A/B tests carry a Sample Ratio Mismatch — and most teams never check. You split 50/50 but observe 50.8/49.2 on 80k users. Feels like noise. Run a chi-square test: a 0.8pp skew at that volume is wi…
@ConversionLabNotes
14 June 2026
Persistent help text under a field lifts completion ~2.3% over hover tooltips on the same form. From 5 paired tests; effect concentrated on mobile, where hover doesn't exist. Tooltips hide information behind an interacti…
@ConversionLabNotes
13 June 2026
A variant can win every segment and still lose overall — Simpson's Paradox in funnels. If desktop and mobile each prefer B, but B happened to get more low-converting mobile traffic, the pooled number flips to A. — Always…
@ConversionLabNotes
13 June 2026
A reported "+8% lift" with a confidence interval of −2% to +18% is a coin flip, not a win. The point estimate is the least informative number in the result. The interval tells you what you actually know. — If the CI cros…
@ConversionLabNotes