We get 200 alerts a day and ignore most of them. How do I fix alert fatigue with real numbers?
Q: Our on-call channel is pure noise. People mute it. Where do I even start?
A: A platform team measured this before touching anything: 6,200 alerts in 30 days, and a survey showed engineers acknowledged just 12% within five minutes. Classic learned helplessness.
They did three concrete things. First, deduplication — one alert per incident instead of one per failed check, which alone removed about 70% of volume. Second, grouping by root service so a database outage fired one grouped alert, not 40 dependent-service alerts. Third, a severity split: only page-worthy events hit phones; everything else went to a daily digest.
Result after 60 days: 6,200 alerts became 380, and median acknowledge time dropped to 90 seconds because people trusted the channel again.
The follow-up: start with deduplication. It's the highest-leverage single change and usually needs no new tooling.
Got a question? Drop it in the comments.
Pingback Clinic
@PingbackClinic
We get 200 alerts a day and ignore most of them. How do I fix alert fatigue with real numbers?
Этот пост опубликован в Telegram-канале Pingback Clinic. Подписаться можно по ссылке: @PingbackClinic.