My test won because of a holiday, not my variant.

My test won because of a holiday, not my variant.

Ran a promo banner test across two weeks. Variant crushed it, +24%. Bottled it. Shipped permanently.

Next month: gone. Worse than before.

Turns out week two of my test overlapped a payday + a niche shopping holiday. The variant happened to be live during a buying surge. I credited my banner. It was the calendar.

Week-over-week confounds are silent killers when arms aren't time-balanced.

— Run both variants simultaneously, never sequentially
— Watch for paydays, holidays, your own email blasts
— A lift that vanishes next cycle was probably seasonal

Go check if anything external spiked during your last "win." Report back.