Myth: A/B test results and attribution numbers should agree — if they don't, something's broken

Myth: A/B test results and attribution numbers should agree — if they don't, something's broken

The question: your conversion-rate A/B test says a landing page change lifted conversions 8%, but your attribution dashboard shows the paid channel driving that page barely moved. Which one is wrong?

What's actually happening: neither, usually — they answer different questions and routinely disagree by design. An A/B test is a randomized experiment measuring the causal effect of one change on users who reached the test. Attribution is an observational credit-allocation across channels for conversions that occurred. One is causal inference on a controlled variable; the other is bookkeeping over uncontrolled paths.

The nuance: they operate on different units and assumptions. The A/B test holds traffic mix constant and isolates the variant's effect. Attribution lets traffic mix vary and assigns credit retrospectively, with no counterfactual. A page change can be causally great (A/B test) while the attributed channel value looks flat — because attribution is measuring channel credit, not page effect, and is blind to the very thing the experiment manipulated. Expecting them to reconcile is a category error, like asking why a thermometer disagrees with a calendar.

Bottom line for practitioners: stop trying to make experiments and attribution 'match.' Trust the randomized test for causal claims about a specific change, and use attribution only for directional, tactical channel allocation. When they conflict, the experiment wins on causality — attribution was never measuring what the test measured.