The redesign that "increased revenue" — except it didn't

The redesign that "increased revenue" — except it didn't

A team launched a homepage redesign in late November and saw revenue jump 30% the next two weeks. They credited the redesign and planned to roll the pattern everywhere. We asked to see the same window a year earlier.

The clue was seasonality. Black Friday and the holiday run had lifted the entire industry 30%+ in that exact window the prior year too. The redesign had launched into a tailwind and taken the credit.

When we isolated the effect with a year-over-year comparison on a stable segment, the redesign's real lift was about 3% — and one section had actually hurt mobile conversion.

We fixed that section and verified properly. Mobile conversion recovered from 2.4% to 2.9%, the honest win.

—
Lesson: a number that moves after your change didn't necessarily move because of it. Annotate launches and compare against the same period last year before you claim a victory.