Case #044: The test that needed more traffic than I had

Case #044: The test that needed more traffic than I had

Optimizing a lead-gen lander, $2,000 in traffic. I wanted to test headline, hero image, and button color. Two roads: clean sequential A/B tests, or one multivariate test running all combinations at once.

Multivariate: 3 variables, 2 versions each = 8 combinations. I split my traffic eight ways. After $2,000 and 4,100 clicks, every cell had ~80 conversions — nowhere near statistical significance. The "winner" flipped three times as data trickled in. I'd spread my traffic so thin that noise looked like signal. I made a decision on a coin-flip and didn't know it.

A/B sequential, the rerun: I tested headline alone first. $700, clear winner at 95% confidence in five days. Then locked it and tested the image. Then the button. Three clean rounds, $2,100 total, each decision actually significant.

The tradeoff is speed versus certainty per dollar of traffic. Multivariate tests interaction effects and tests everything at once — gorgeous if you have 50,000 conversions to spend. A/B is slower calendar-wise but each result is trustworthy at low volume.

Final arc: multivariate gave me a fast answer that was statistically garbage. Sequential A/B gave me three slow answers I could actually bank — and the rebuilt lander converted 31% better than the multivariate "winner."

The lesson: multivariate testing is a luxury that demands traffic most of us don't have — below five figures of conversions, every cell you add is just a thinner slice of guessing.