Testing "did they convert" instead of "how many items" can hide a real win or invent one. A binary conversion metric ignores basket depth; a count metric is noisier but catches AOV moves. The metric y…
Among barely-significant wins from small tests, the true effect is ~50% smaller than reported. The winner's curse / Type-M (magnitude) error: when power is low, the only effects that clear significanc…
Any single-change uplift above ~25% is more likely a bug than a breakthrough. Twyman's law: figures that look interesting are usually wrong. A 40% checkout lift from a button color almost always trace…
Optimizing a single funnel step lifts overall conversion ~0.3× of the step's local gain. A +10% improvement at a step that 40% of users reach contributes maybe +3% end-to-end, less if downstream steps…
"95% probability B beats A" and "p=0.05" answer different questions. Frequentist p-value: chance of this data if A=B. Bayesian posterior: chance B>A given the data. Stakeholders want the second; class…
One whale order can manufacture a fake 8% revenue lift in a mid-size test. Revenue is heavy-tailed; the mean is hostage to extremes. A single $9,000 order in the variant arm moves the average that the…
