<b>Why do so many geo holdout tests come back "inconclusive"?</b>

<b>Why do so many geo holdout tests come back "inconclusive"?</b>

Usually not because the channel doesn't work — because the experiment was statistically dead on arrival. Geo testing lives or dies on design, not execution.

<b>The setup</b>

A geo holdout splits regions into treatment (gets the campaign) and control (doesn't), then compares outcomes. It's a randomized experiment at the market level, which sidesteps user-level tracking entirely — its great advantage in a privacy-constrained world.

<b>The three things that kill it</b>

— <i>Too few geos.</i> Markets are heterogeneous and few in number. With a handful of regions, between-market variance swamps any treatment effect. You need enough units, and matched ones.
— <i>Underpowered spend.</i> If the incremental lift you're hunting is 3% and your test can only detect 10%, a null result tells you nothing. Power analysis must precede the test, not follow the disappointment.
— <i>Spillover.</i> Neighboring or online-overlapping markets contaminate the control, biasing the effect toward zero.

<b>The nuance</b>

Matched-market methods and synthetic control (constructing a weighted blend of control geos that mimics the treated region's pre-period trend) materially improve power without more markets. They're the difference between a usable read and noise.

<b>What to actually do</b>

— Run a power calculation first; if the minimum detectable effect exceeds your plausible lift, don't run it.
— Use synthetic control rather than raw region averages.
— Hold the test long enough to clear adstock decay, not just the flight.

Bottom line for practitioners: an inconclusive geo test is usually a design failure, not a verdict on the channel. Power it, match it, and protect the control from spillover before you spend a dollar.