The A/B test that won and lost money

The A/B test that won and lost money

A marketplace declared a winner: a new product layout beat control with 96% confidence on click-through. They rolled it to 100%. Revenue slid for a month before anyone connected the dots.

The clue was the metric the test measured. Significance was real — for clicks. But the winning layout pushed cheaper items higher, so people clicked more and spent less. Downstream revenue per visitor was never in the test.

Statistical significance answers "is this difference real," not "is this difference good for the business."

We re-ran it with revenue per session as the primary metric. The original "winner" lost by 6%. The actual winner lifted revenue per session from $3.10 to $3.48.

—
Lesson: significance validates the metric you chose, not the metric that pays the bills. Pick the business outcome as your primary before you trust the green checkmark.