How to design your first geo holdout test (without burning the quarter)

How to design your first geo holdout test (without burning the quarter)

The question: you want to prove a channel is incremental — that it causes sales rather than just correlating with them. Where do you start when you have no clean experiment yet?

The playbook, in order:

— Pick the channel where last-click overstates the most. Branded search and retargeting are the usual suspects; they harvest demand others created.
— Group your markets into matched pairs. Pre-period correlation between candidate geos should run above 0.9 on the outcome metric. Studies on synthetic-control methods (Abadie's work) show matching quality dominates everything downstream.
— Designate test geos (turn spend off) and control geos (leave it on). Power-check first: small markets and noisy conversions can need 6+ weeks to detect a 10% effect.
— Pre-register the analysis window and the metric before you peek. Looking early is how teams talk themselves into a false positive.
— Run a synthetic control or difference-in-differences to estimate the counterfactual.

The nuance: a flat result is signal, not failure. If turning the channel off doesn't move conversions, the last-click credit it earned was borrowed demand. And matched geos drift — a competitor promo in one region quietly breaks your control.

Bottom line for practitioners: start with one channel, one clean pair structure, a pre-registered window, and enough weeks to clear noise. A modest, well-powered holdout beats an elaborate model fit to observational data every time.