<b>What do you do when you can't randomize at the user level?</b>

<b>What do you do when you can't randomize at the user level?</b>

The question: cookie loss and privacy walls are making clean user-level holdouts harder every quarter. When you can't split individuals, can you still run a credible causal test?

What the methodology says: yes — geo experiments randomize at the market level. You turn a channel on in some metro areas and off (or down) in matched others, then measure the difference in outcomes. The credibility comes from how you pick the controls. Naive geo tests fail because markets aren't comparable. The serious version uses <b>synthetic control</b> — constructing a weighted blend of untreated markets that closely tracks the treated market's pre-period behavior, so the post-period gap is a clean estimate of lift.

What the research shows: methods like 'GeoLift' and Bayesian structural time-series ('CausalImpact') formalize this. Validation studies find geo experiments recover incrementality estimates close to user-level RCTs when the pre-period match is tight and enough geos are used — but the estimates get unstable fast with few markets or a short baseline.

The nuance: geo tests measure <i>aggregate market lift</i>, not individual causation. They also can't isolate creative or audience effects within a channel — only the channel's net market impact. And spillover between adjacent markets biases results toward zero.

What to actually do: require at least a few months of stable pre-period data and enough matched markets for a synthetic control to converge. Pre-register the analysis window before you peek.

<b>Bottom line for practitioners:</b> when you lose the individual, randomize the market — but a geo test is only as honest as the counterfactual you build for it.