How wrong can a media mix model be before you calibrate it with an experiment?

How wrong can a media mix model be before you calibrate it with an experiment?

The question for any team rebuilding measurement after tracking loss: media mix modeling reads aggregate spend-versus-sales, but how do you know its channel estimates aren't fitting noise?

The approach. Media mix modeling is regression on aggregate time series — it can't see individuals, only the relationship between total spend per channel and total sales. Its weakness is identification: collinear channels (ones that move together) and confounders (seasonality, pricing, distribution) make coefficients unstable. The fix that leading teams adopted is calibration — using incrementality experiments as priors or ground-truth checks on the model's estimates.

What calibration revealed. In documented cases (including work behind Meta's open-source Robyn and Google's Meridian frameworks), uncalibrated models produced channel ROIs that disagreed with paired geo-experiments by wide margins — sometimes a factor of 2-3x, occasionally the wrong sign. After feeding experiment results in as Bayesian priors, the model's estimates converged toward the experimentally measured truth and out-of-sample prediction error fell materially.

The nuance. A media mix model is a structured correlation. Left alone, it confidently attributes sales to whichever channel best fits the curve, confounders and all. The experiment is what injects causation into a correlational machine.

Bottom line for practitioners: never deploy a media mix model uncalibrated — its coefficients are correlations until an experiment anchors them. Run even one or two geo-holdouts and use them as priors. The documented 2-3x corrections are the gap between a model that fits the past and one that predicts the future.