How to run a power analysis before you launch any attribution experiment

How to run a power analysis before you launch any attribution experiment

The question: half of incrementality tests come back 'inconclusive' — not because the channel doesn't work, but because the test was never large enough to detect its effect. How do you avoid wasting a quarter on an underpowered test?

The power-analysis steps, before launch:

— Define the minimum detectable effect (MDE) — the smallest lift worth acting on. A test built to catch a 30% effect is blind to a real 8% one.
— Pull your baseline conversion rate and its variance. Rare, noisy conversions need far more sample than common ones.
— Estimate required sample and duration for ~80% power at your MDE. The arithmetic: smaller effects and noisier metrics demand quadratically more data.
— Reality-check against your traffic. If hitting power needs 12 weeks and you have 6, redesign now — switch to a more granular metric, a cleaner geo pairing, or a bigger holdout share.

The nuance: a null result from an underpowered test tells you nothing — you can't distinguish 'no effect' from 'effect too small to see.' Teams routinely misread these nulls as proof a channel is worthless and cut it wrongly. Absence of evidence isn't evidence of absence.

Bottom line for practitioners: compute MDE, baseline variance, and required duration before you start. An experiment you can't power is a coin flip dressed as rigor — design it to detect the effect size that would actually change your decision, or don't run it.