29 October 2026
Myth: a big observed uplift means a reliable result. False — large effects on small samples are the least trustworthy. Low-powered tests only reach significance via inflated estimates — the 'winner's curse.' A true 3% li…
@ConversionLabNotes
25 October 2026
Myth: checking a test early doesn't hurt if you wait for significance. False — peeking quadruples your false-positive rate. Fixed-horizon tests assume one look. Peek daily for 2 weeks and your true α drifts from 5% towar…
@ConversionLabNotes
22 October 2026
Myth: a 95% confidence result is 95% likely to be true. False — that's not what the number measures. p<0.05 means: if there were no real effect, you'd see data this extreme under 5% of repeats. It says nothing about t…
@ConversionLabNotes
18 October 2026
Myth: running more concurrent tests means you learn faster. False — overlap inflates false positives. With 5 simultaneous tests at α=0.05 (5% false-positive threshold per test), the chance at least one fires by chance is…
@ConversionLabNotes
11 October 2026
Report the confidence interval, not just 'significant': a +0.4% win with a [−3%, +4%] interval is a coin flip wearing a badge. The p-value tells you if there's an effect; the interval tells you if you should care. — Alwa…
@ConversionLabNotes
08 October 2026
Enforce a one-week-multiple minimum runtime even after hitting sample size: weekday and weekend buyers convert differently. Reaching n on a Tuesday doesn't mean you've seen a representative week. — Set runtime = max(requ…
@ConversionLabNotes
04 October 2026
Hold back 5% from every shipped winner: stacked 'wins' often sum to far less than the spreadsheet promised. Lifts don't add. Diminishing returns and decay erode the cumulative number. — Keep a permanent 5% holdback on th…
@ConversionLabNotes
30 September 2026
Run a 15-minute test post-mortem on every result, win or lose: inconclusive ≠ no information. The close-out checklist that compounds your win rate over time. — Was it powered? Confidence interval width tells you if 'flat…
@ConversionLabNotes
29 September 2026
Lock one primary metric before launch: testing 10 metrics at 95% gives a ~40% chance of a false flag. More metrics is not more insight — it's more ways to be wrong by accident. — Name exactly one primary metric, decided …
@ConversionLabNotes
28 September 2026
Handle revenue outliers before testing RPV: one $40k B2B order can fake a 6% 'win'. Revenue-per-visitor has a fat tail. One whale dominates the mean and your conclusion. — Plot the revenue distribution per arm. Spot the …
@ConversionLabNotes
27 September 2026
📎 Кстати, @pricing_tests_lab регулярно публикует pricing experiments.…
@ConversionLabNotes
26 September 2026
Ramp new variants 1% → 5% → 50%, not straight to half your revenue. The goal of the early phase isn't power — it's catching a catastrophe cheaply. — Day 0-1 at 1%: watch error rate, latency, JS exceptions, hard checkout …
@ConversionLabNotes
25 September 2026
Write the hypothesis card before building: tests with a pre-registered metric ship a decision 2x more often. The one-page template that prevents 'we're not sure what happened.' — Because we observed [data, e.g. 30% phone…
@ConversionLabNotes
24 September 2026
Triage session recordings in 30 minutes: watch the 12 sessions that rage-clicked, not 200 random ones. Qualitative review is a sampling problem. Sample the friction, not the average. — Filter recordings to: rage clicks, …
@ConversionLabNotes
23 September 2026
Prioritize funnel leaks by absolute lost users, not by step conversion rate. A 90% step and a 60% step can leak the same headcount — fix where the bodies are. — Map each step's entry count and exit count. — Lost users = …
@ConversionLabNotes
22 September 2026
Isolate novelty effect: ~40% of early UI 'wins' decay to zero within two weeks among returning users. Returning visitors react to change itself, not to the change being better. The playbook: — Segment new vs returning fr…
@ConversionLabNotes
21 September 2026
Run an A/A test before trusting your tooling: a healthy setup should show significance ~5% of the time, no more. If your platform 'finds' winners between two identical variants, every result is suspect. — Split traffic 5…
@ConversionLabNotes
20 September 2026
Audit payment-error recovery: the median checkout abandons 23% of users after one declined-card screen. The leak isn't acquisition — it's the second attempt that never happens. — Log every gateway decline with its reason…
@ConversionLabNotes
19 September 2026
Pre-register your segments: a flat +0% test often hides +9% mobile / −7% desktop. Aggregate results average away the truth. Split before you launch, not after. — Declare 3-4 segments up front: device, new vs returning, t…
@ConversionLabNotes
18 September 2026
Stop peeking: checking a test daily and stopping at first significance inflates your false-positive rate from 5% to ~30%. The fix is procedural, not statistical willpower. — Option A: fix the sample size in advance, look…
@ConversionLabNotes
17 September 2026
Define guardrail metrics before launch: ~1 in 5 'winning' CRO tests quietly raised refund or support rate. A conversion lift that increases returns is a loss you booked as a win. The checklist: — Primary: the metric you'…
@ConversionLabNotes
16 September 2026
Some of the best in the wider network right now: — @inbox_oneoone — Email marketing explained from zero — deliverability, segmentation,… — @deal_wire — Insider intel for coupon and deal-site operators: merchant feed… — @…
@ConversionLabNotes
15 September 2026
Run a sample-ratio mismatch check before reading any result: ~6% of A/B tests fail it silently. SRM (your 50/50 split isn't actually 50/50) invalidates the whole test, not just one metric. The 4-step audit: — Count actua…
@ConversionLabNotes
14 September 2026
Set MDE before you launch, not after: 80% of inconclusive tests were underpowered from day one. The one-task playbook: size the test so you don't waste 3 weeks. — Pull baseline conversion rate from 28 days of traffic (yo…
@ConversionLabNotes
13 September 2026
One-tailed tests need ~21% less sample than two-tailed for the same power — by quietly assuming the variant can't lose. The saving is real; the assumption is usually false. If your variant can plausibly hurt (most can), …
@ConversionLabNotes
12 September 2026
Security trust badges lift checkout ~1.5%; review-count social proof lifts ~2.8% — and they don't stack linearly. From tests run both isolated and combined; combined lift was ~3.4%, not the 4.3% sum. Both reduce perceive…
@ConversionLabNotes
11 September 2026
Geo holdout tests measure true incremental lift; standard A/B can't when treatment leaks across users. Use geo when the change isn't user-randomizable — pricing, brand campaigns, shared-cart features. Cost: geo tests hav…
@ConversionLabNotes
10 September 2026
Replacing a dropdown with radio buttons lifts completion ~1.8% when options number ≤5. From 7 field tests; effect inverts above ~7 options, where the dropdown's compactness wins. Radios are single-tap and fully visible —…
@ConversionLabNotes
09 September 2026
Your analytics platform and your A/B tool will disagree on conversions by a median ~8%. Measured across parallel tracking on 11 tests; gap driven by attribution windows, bot filtering, and dedup logic. Neither is "right.…
@ConversionLabNotes
08 September 2026
Exit popups capture ~2-4% of leavers; embedded inline forms convert ~0.5-1% of all visitors — but the bases differ. The comparison is apples-to-oranges unless you normalize by the population each sees. Popups sample only…
@ConversionLabNotes
