Computing significance on sessions when users have many sessions understates variance — false positives climb.

Computing significance on sessions when users have many sessions understates variance — false positives climb.
The mistake: the t-test assumes independent units, but one heavy user's 12 sessions aren't independent. Standard errors come out too small.
— Result: p-values look better than reality; "significant" isn't.
— Fix: randomize and analyze at the user level, or use clustered/bootstrap standard errors (delta method) on the ratio.
Read the number, not the story. [unit = user, not session]