SOP: Calling an A/B test winner

SOP: Calling an A/B test winner
Run this whenever a split test reaches your pre-set sample size, not before.

Stage 1 — Validity gate
— ☐ Analyst confirms the test hit the sample size set at launch (decided up front, never moved)
— ☐ Analyst confirms only one variable changed between A and B

Stage 2 — Read the result
— ☐ Analyst checks the difference clears your significance threshold (e.g. 95%), not just "looks bigger"
— ☐ Analyst checks the winner on the metric that pays you (CPA or ROI), not CTR vanity

Stage 3 — Decide + record
— ☐ Media buyer ships the winner and archives the loser with the data
— ☐ Analyst logs the learning in the test journal so it isn't re-tested in 3 months

Owner: Analyst (call), Media buyer (ship)
Trigger: Sample size reached
Done-when: Winner is shipped on the money metric and the learning is logged.

Save this. Run it every time.