SOP: Setting up a valid A/B test

SOP: Setting up a valid A/B test

Run this whenever you want to test a landing page change.

Stage 1 — Design
☐ Change exactly one variable between A and B (Owner: buyer)
☐ Write the hypothesis and the metric that decides the winner
☐ Calculate the sample size needed before you start, not after

Stage 2 — Build
☐ Split traffic 50/50 with the tracker, not manual rotation
☐ Confirm both variants fire conversions correctly
☐ Tag each variant in the tracker so reporting is clean

Stage 3 — Run and call it
☐ Do not peek before the sample size is reached
☐ Check the split actually ran 50/50, not 70/30
☐ Declare a winner only if it clears your significance bar

Owner: media buyer. Trigger: lander change to validate. Done-when: the pre-set sample size is reached and one variant clears the significance bar.

Save this. Run it every time.