The winning button that won nothing

The winning button that won nothing

A marketplace changed its CTA button from blue to green, watched signups rise 12% the next week, and declared victory. The before-after comparison had a passenger nobody noticed.

A pure before-after measures the change plus everything else that moved that week: seasonality, a press hit, payday. The clue: a competitor had paused ads that exact week, lifting the whole category.

The change: they re-ran it properly as a split test, half the traffic on each button, same moment in time, so external noise hit both groups equally.

The green button's true lift came in at 1.4%, inside the margin of error, not 12%.

Before-after answers 'did the number move,' which is rarely the question. A controlled split answers 'did my change move it.' When you can randomize, do, because the calendar is never a fair control group.