The on-page test with no control

The on-page test with no control

"We changed the titles and traffic went up — the test worked."

Did it.

This is the most common on-page testing mistake: a single before/after with no control group and no seasonality adjustment. You changed titles in March, traffic rose, you declared victory. So did the competitor who changed nothing — because March is just up.

Here's what actually happens: without a holdout set of comparable pages left unchanged, you can't separate your edit from the algorithm update, the seasonal curve, or the random walk of rankings. You're reading tea leaves and calling it data.

Fix: split comparable pages into a test group and a control group, change only the test group, and compare the delta between groups, not the before/after of the test group alone. CTR tests in particular need this — impressions move on their own constantly. (n=1 page over 14 days proves nothing, no matter how the line looks.)