Is your incrementality win real, or did you just test during a novelty spike?

Is your incrementality win real, or did you just test during a novelty spike?

The question: you run a clean lift test on a new channel or creative, see strong incremental conversions, and scale. Three months later the lift has evaporated. Was the experiment wrong, or did the world change?

What the literature describes: this is the novelty effect (and its cousin, ad fatigue) — well-documented in both UX and advertising research. Early exposure to something new generates a temporary response that decays as the audience habituates. A short incrementality test run at launch captures the peak, not the steady state. The estimate isn't biased by bad design; it's biased by timing — you measured a transient.

The deeper problem: this isn't correlation-vs-causation. The causal effect was real during the test. The error is assuming a point-in-time causal estimate is stationary — that what lifted conversions in week one will lift them identically in month six. Most attribution and incrementality work implicitly assumes stationarity, and audiences are anything but stationary.

The nuance: the inverse also happens — some effects build over time (brand, retargeting saturation), so a too-early test understates them. The direction depends on the channel.

What to actually do: run lift tests over a duration long enough to see the decay curve flatten, not just clear significance. Re-test scaled channels periodically — incrementality is a depreciating asset, not a constant.

Bottom line for practitioners: a single incrementality number is a snapshot of a moving target. Measure the curve, not the peak, before you bet the budget.