Case: A test protocol that ended 'gut-feel' calls and lifted CR 19%

Case: A test protocol that ended 'gut-feel' calls and lifted CR 19%

A team declared test winners after 200 clicks and kept picking noise. A disciplined test runbook with a stop rule fixed decision quality.

Run this whenever you launch a landing page split test.

Stage 1 — Design (Owner: analyst)
☐ One variable changed, hypothesis written down
☐ Set sample target: min 100 conversions/variant

Stage 2 — Run (Owner: buyer)
☐ Split traffic 50/50, no mid-test edits
☐ No winner calls before sample target hit

Stage 3 — Decide (Owner: analyst)
☐ Confirm 95% confidence before declaring
☐ Log result and roll winner to all traffic

Trigger: split test launch.
Done-when: sample reached, winner logged with confidence.

Result: across 12 tests, valid wins lifted average CR 19%; false 'winners' dropped to near zero.

Save this. Run it every time.