SRM (your 50/50 split isn't actually 50/50) invalidates the whole test, not just one metric.
The 4-step audit:
— Count actual users per arm. Expected 50/50, observed maybe 50.8/49.2.
— Run a chi-square goodness-of-fit on those raw counts.
— If p < 0.001, you have SRM. Do not trust the uplift.
— Trace the cause: redirect latency, bot filtering applied post-split, or a broken variant erroring out before assignment.
A 'winning' variant is often just one that crashed for slow users. Read the number, not the story. [SRM p<0.001 = halt]
Conversion Lab Notes
@ConversionLabNotes
Run a sample-ratio mismatch check before reading any result: ~6% of A/B tests fail it silently.
This post was published in the Telegram channel Conversion Lab Notes. Subscribe here: @ConversionLabNotes.