Geo experiments vs. user-level holdouts: two routes to incrementality

Geo experiments vs. user-level holdouts: two routes to incrementality

If you want to measure incrementality — the conversions that happened because of an ad and would not have happened otherwise — you must withhold the ad from someone. The question is from whom.

The two designs
User-level holdout: a random fraction of individuals are excluded from a campaign; the gap between exposed and control conversion rates is the lift. Geo experiment (geo-lift): entire markets — DMAs, postal regions, countries — are split into test and control, and you compare market-level outcomes.

Tradeoffs that decide the choice
User holdouts give tight statistical power and clean randomization, but they require platform support and break down when identity is fragmented across devices, or when the platform contaminates the control via lookalike spillover.
Geo experiments need no user tracking at all — which is why they survived the privacy reset — but markets are few and heterogeneous, so power is low and you often need synthetic-control methods (weighting untreated geos to mimic the treated ones) rather than naive averages.

The trap
Geo tests assume no cross-market spillover. National TV bleeding into your "control" DMA quietly biases lift downward. User holdouts assume the platform truly withholds; many "PSA holdouts" leak.

Bottom line for practitioners: Use user-level holdouts for trackable, addressable channels where the platform offers a verified control. Use geo experiments for broad-reach, untrackable media (TV, audio, upper-funnel social) where you can tolerate fewer, larger units. Power-calculate before you launch either — most underpowered tests "prove" nothing.