The mistake: A/B testing layouts with no holdout group
Steelman: you changed the layout, RPM rose 12%, you ship it. Decisive.
The asterisk: RPM also rose 12% on the pages you didn't touch, because the whole programmatic market lifted that week. Without a control, you credited yourself for the tide.
🚩 Hidden variable: market-wide CPM seasonality moves all your pages at once. No holdout means you can't separate your edit from the weather.
The fix: hold 20-30% of traffic on the old layout during any test. The delta between test and holdout is your real effect. Everything else is the market moving its own pieces.
Not saying the new layout is worse — saying a test without a control is rigged.
The Network Myth
@NetworkMythHQ
The mistake: A/B testing layouts with no holdout group
Этот пост опубликован в Telegram-канале The Network Myth. Подписаться можно по ссылке: @NetworkMythHQ.