Holdout group vs pre/post comparison

Holdout group vs pre/post comparison
Launched a sitewide change I couldn't cleanly A/B (a new nav). Two ways to measure.
Pre/post: compare conversion before launch vs after. Easy, but garbage when seasonality, a sale, or a traffic-source shift muddies it. I've fooled myself this way twice.
Holdout: keep 5-10% of users on the OLD experience permanently for a few weeks. Compare against the 90% who got the new thing. Same time window, same season, same traffic. Clean read.
The tradeoff: a holdout 'wastes' a little traffic on the old version. That's the price of a real causal answer.
For anything you can't slot into a normal test, a holdout beats pre/post every single time.
Got a big launch coming? Don't go all-in. Hold back 5%. Report back.