Average lift vs. heterogeneous effects: when one incrementality number hides the answer

Average lift vs. heterogeneous effects: when one incrementality number hides the answer

A lift test returns one number: the campaign drove X% incremental conversions on average. But the average can be useless if the effect is wildly uneven. Two methods take you from "average lift" to "lift for whom."

The limitation of the headline number
An average treatment effect can be a positive aggregate hiding a profitable core and a money-losing tail — or a near-zero average masking strong lift in one segment cancelled by a do-not-disturb effect in another. Acting on the average over-spends on non-responders and under-spends on responders.

Two ways to decompose it
Pre-specified subgroup analysis splits your experiment by segments you chose in advance (new vs. returning, device, geo) and estimates lift within each. It's transparent and honest if segments are pre-registered — slicing after the fact until something is significant is p-hacking, and it manufactures false findings reliably.
Causal forests (and related machine-learning causal estimators) instead discover which features drive heterogeneous response, estimating an individual-level treatment effect without you naming segments upfront. Powerful, but data-hungry and prone to overfitting the noise if the experiment is small.

Bottom line for practitioners: If you have strong hypotheses about who responds and a modestly-sized test, use pre-registered subgroup analysis — commit to the splits before you peek. If you have a large experiment and want the data to reveal unexpected responders, use a causal forest, then validate the discovered segments on a fresh holdout before trusting them. Either way, resist reporting a single average lift for a campaign whose whole value lives in its tails — the average is where heterogeneous effects go to hide.