I keep a group that NEVER sees my winners. On purpose.

I keep a group that NEVER sees my winners. On purpose.

Got tired of shipping "wins" that didn't show up in the monthly revenue chart. Lifts on paper, flatline in the bank.

So now 5% of traffic is a permanent holdout. They get the original site, frozen. Everyone else gets every winner I ship.

After a quarter, I compare the 95% (all my "wins") against the frozen 5%. That gap is the REAL cumulative impact. No peeking, no novelty, no seasonality games.

First time I ran it, half my "wins" had evaporated. Humbling.

— Keep a long-run holdout to validate stacked wins
— It catches decay individual tests miss
— It's the only number your CFO actually believes

Go carve out a 5% frozen holdout this week. Report back in 90 days.