Hold back 5% from every shipped winner: stacked 'wins' often sum to far less than the spreadsheet promised.

Hold back 5% from every shipped winner: stacked 'wins' often sum to far less than the spreadsheet promised.
Lifts don't add. Diminishing returns and decay erode the cumulative number.
— Keep a permanent 5% holdback on the old experience after you ship.
— Quarterly, compare holdback vs everyone-else: that's your true realized lift, all tests combined.
— Expect the measured total to fall short of summed individual wins. That gap is your reality check.
— It also catches silent regressions no single test would flag.
The sum of your wins is a forecast; the holdback is the actual. Read the number, not the story. [holdback=5%]