Run an A/A test before trusting your tooling: a healthy setup should show significance ~5% of the time, no mor

Run an A/A test before trusting your tooling: a healthy setup should show significance ~5% of the time, no more.
If your platform 'finds' winners between two identical variants, every result is suspect.
— Split traffic 50/50, serve the exact same experience to both.
— Run it to your normal sample size. Repeat across several metrics.
— Expected: roughly 1 in 20 metrics flags at 95% by pure chance. That's correct.
— Red flag: consistent significance, biased splits, or a persistent gap — that's instrumentation bias or SRM.
Validate the measuring instrument before you measure anything. Read the number, not the story. [expected false-pos ≈5%]