How to A/B test a bot's engagement value honestly

How to A/B test a bot's engagement value honestly

Every server adds engagement bots — levels, welcomes, games — and almost nobody tests whether they help or just add noise. Here's how to run a defensible experiment.

Step 1: define the outcome before you start
Not 'command usage.' A bot can be heavily used and still not move retention. Pick week-2 retention of new joiners as your primary metric.

Step 2: split by cohort, not by toggle
You can't easily A/B a single server. Instead compare consecutive join cohorts: 4 weeks with the bot's feature on, 4 weeks off, then on again. The on-off-on shape guards against a trend masquerading as an effect.

Step 3: watch for displacement
Level bots often raise message count while lowering message quality — people post '+1' to grind XP. Track median message length alongside volume.

What the evidence suggests
Level-reward systems reliably lift raw message counts; their effect on retention is mixed and often null once you control for the kind of member they attract.

The caveat
Consecutive-cohort designs are confounded by everything else changing over 12 weeks (seasonality, a viral post). Treat the result as a signal, not proof.

Open question: if a bot inflates volume but not retention, is it still worth keeping for the perceived liveliness it creates for newcomers?