Testing "did they convert" instead of "how many items" can hide a real win or invent one.

Testing "did they convert" instead of "how many items" can hide a real win or invent one.
A binary conversion metric ignores basket depth; a count metric is noisier but catches AOV moves. The metric you choose pre-decides what "better" means.

— Match the metric to the change: a bundling test needs items-per-order, not yes/no.
— Don't switch metrics after seeing data — that's p-hacking.

Read the number, not the story. [metric chosen pre-launch]