A 4-factor multivariate test needs ~16x the sample of a single A/B to detect the same main effect.

A 4-factor multivariate test needs ~16x the sample of a single A/B to detect the same main effect.
Combinatorics: 2^4 = 16 cells, each needing power. MDE held at 5%.
MVT (multivariate testing) earns its sample cost only when you suspect interactions between elements. If elements are independent, run sequential A/Bs — cheaper and faster.
— A/B ▮▮ sample, fast
— MVT ▮▮▮▮▮▮▮▮ sample, slow but reveals interactions
Reserve MVT for high-traffic pages where interaction effects are plausible. Otherwise it starves on data.
Read the number, not the story. [combinatorial · MDE=5%]