Myth: A/B test layouts by comparing RPM directly

Myth: A/B test layouts by comparing RPM directly

Most RPM A/B plugins show you variant RPM side by side. Reading that number naively will fool you.

What it does — splits traffic across ad layouts and reports RPM per arm.
Setup effort — 3/5.
The correction — RPM is impressions × eCPM, and eCPM swings wildly with daily demand. A variant can "win" purely because it ran during a high-demand window. Without splitting traffic simultaneously and watching session RPM + confidence intervals, you're A/B testing the ad market, not your layout.
RPM lift seen — proper simultaneous 50/50 with CIs flipped 2 of 5 "winners" we'd have shipped on raw RPM.
Pros — real causal signal.
Cons — slower, needs traffic volume for significance.

Lift: ★★★★☆ — Risk: ★★☆☆☆
Best for: 50k+ sessions/test. Skip plugin verdicts on low traffic — they're noise.