Playbook: Set Up a Clean Incrementality Test for In-App Spend

Playbook: Set Up a Clean Incrementality Test for In-App Spend

Last-touch lies. Measure true lift in 7 steps:
— Pick the method: geo holdout (simple, coarse) or PSA/ghost-ads holdout (granular, harder)
— Define test vs control with comparable baseline installs — match on size and seasonality, not randomly
— Size the holdout for statistical power; too small and your lift is noise
— Run for a full conversion cycle — minimum 2-4 weeks, never a long weekend
— Hold all other variables: don't change creative, budget, or geos mid-test
— Measure lift on the true KPI (revenue, retained users), not just installs
— Compute incremental cost-per-action, then compare to your last-touch CPA

✓ Reveals how much spend just claims credit for organics
✓ Geo holdout works without special ad-network tooling
✗ Geo tests are coarse and confounded by regional effects
✗ Needs enough volume for significance — small apps struggle

Verdict: Use geo holdouts for a quick directional read; invest in ghost-ads only when your platform supports it and spend justifies it.
Best for: Teams suspicious that their reported in-app ROAS is borrowing organic credit.