Case #034: Reading a test that wasn't lying to me

Case #034: Reading a test that wasn't lying to me

I used to 'A/B test' by launching four creatives, eyeballing them at lunch, and crowning whoever was ahead. I crowned a lot of false winners that way — and scaled a few into the ground. Here's the test-structure playbook I run now so the result actually means something.

On a dating offer, $400 test budget:

— One variable per test, always: I tested headline only — same image, same page, same audience. If you change two things, the winner teaches you nothing.

— Equal budget, equal time: each variant got identical daily spend over the same 72 hours. Cutting a 'loser' early biases the whole read.

— Set the sample floor before you start: I require 40 conversions total across variants before I'll call it. Below that it's noise wearing a costume.

— Watch confidence, not the lead: variant B led 14 to 11 on day 1 — meaningless. By 40 conversions it was 26 to 14, a real gap.

— Kill the inconclusive, don't keep the close call: if two variants finish within 15%, neither won. Bank both, test a fresh angle.

The disciplined read cost $400 and 72 hours but handed me a headline that genuinely outperformed — not a coin flip I mistook for skill. Scaled the real winner: $2,140 spent, $3,690 back, 72% ROI.

The lesson: a test that ends before the math matures isn't a test, it's a horoscope — wait for the sample, then you've earned the right to scale.