Playbook: Running an Incrementality Test on a Partner Channel

Playbook: Running an Incrementality Test on a Partner Channel

Attribution tells you which partners touched deals; incrementality tells you which deals wouldn't have closed otherwise. The two diverge constantly. A field-experiment approach:

— Step 1. Choose a design. A geo or account-list holdout is cleaner than a time-based on/off, which confounds with seasonality and macro shifts.
— Step 2. Power the test honestly. B2B base rates are low and cycles long, so detecting a 10% lift on a quarterly conversion often needs hundreds of accounts per arm. Compute this before, not after.
— Step 3. Randomize at the account level, not the lead level — multiple leads per account violate independence.
— Step 4. Pre-register the metric and horizon. Per the broader marketing-science literature on lift testing, post-hoc metric shopping is the dominant source of false positives.
— Step 5. Read incremental ROI, not gross ROI: (treatment conversions − control conversions) × deal value ÷ partner cost.

Trade-off: holdouts cost real pipeline you deliberately forgo, and long B2B horizons mean results arrive a quarter or two late. That lag is the price of causal evidence.

A recurring finding across lift studies: channels with the best last-touch attribution often show the weakest incrementality, because they harvest demand others created.

Implications: budget for one rigorous holdout per major channel per year. One causal read beats twelve correlational dashboards.