A checklist for prepping a clean incrementality test in programmatic

A checklist for prepping a clean incrementality test in programmatic

Incrementality (the conversions your ads actually caused, versus those that would have happened anyway) is the only honest performance question. Set the test up correctly or it lies.

1. Build the control group as a true holdout — users eligible to be served who are deliberately withheld via a ghost bid or PSA, not users who simply weren't reached. Unreached users are a biased comparison.

2. Randomize at the user level before the auction, so exposure status is assigned independently of bid outcome. If control is defined post-hoc as 'people we lost the auction on,' you've selected for cheaper, different users.

3. Size the holdout for your baseline conversion rate. Low base rates need large holdouts to detect a real lift above noise.

4. Lock the measurement window to your conversion lag and freeze targeting and budget for the duration.

5. Suppress cross-DSP contamination — if another buy reaches your control group, lift collapses.

6. Pre-register the metric and window before launch so you can't cherry-pick afterward.

Why it matters: Almost every 'lift' number in programmatic is selection bias dressed as causation, because the control group was defined by auction losses rather than random assignment. A pre-randomized, pre-registered ghost-bid holdout is the difference between knowing your ads work and merely hoping they do.