How to A/B test two bid strategies without contaminating the result

How to A/B test two bid strategies without contaminating the result

Comparing bid strategies on the same inventory is harder than it looks, because the two arms compete with each other. Here's a clean protocol.

1. Split on a stable hash of the user ID, not on time or geo. Time splits absorb daypart effects; geo splits absorb price differences. A user-hash split keeps both arms facing the same market.

2. Critically, ensure the two arms never bid in the same auction. If both strategies can enter one auction, they inflate each other's clearing price and you measure interference, not strategy.

3. Fix budget per arm proportional to its user share, and pace identically. A faster-pacing arm wins cheaper morning inventory and corrupts the cost comparison.

4. Run long enough to clear the conversion lag of your KPI — a 2-day test on a 7-day conversion window measures noise.

5. Compare on outcome per dollar (cost per action, viewable cost per mille), not win rate. A strategy that wins more can still cost more.

6. Check that win-price distributions didn't drift between arms; if they did, the split leaked.

Why it matters: Most in-platform bid experiments are invalid because the two arms cannibalize each other's auctions and budgets, producing a confident but meaningless winner. A user-hash split with auction isolation is the only design that lets the market treat both strategies fairly — without it, you're comparing two strategies that sabotaged each other.