Pingback Clinic
Pingback Clinic
@PingbackClinic

We get 200 alerts a day and ignore most of them. How do I fix alert fatigue with real numbers?

We get 200 alerts a day and ignore most of them. How do I fix alert fatigue with real numbers?

Q: Our on-call channel is pure noise. People mute it. Where do I even start?

A: A platform team measured this before touching anything: 6,200 alerts in 30 days, and a survey showed engineers acknowledged just 12% within five minutes. Classic learned helplessness.

They did three concrete things. First, deduplication — one alert per incident instead of one per failed check, which alone removed about 70% of volume. Second, grouping by root service so a database outage fired one grouped alert, not 40 dependent-service alerts. Third, a severity split: only page-worthy events hit phones; everything else went to a daily digest.

Result after 60 days: 6,200 alerts became 380, and median acknowledge time dropped to 90 seconds because people trusted the channel again.

The follow-up: start with deduplication. It's the highest-leverage single change and usually needs no new tooling.

Got a question? Drop it in the comments.
Этот пост опубликован в Telegram-канале Pingback Clinic. Подписаться можно по ссылке: @PingbackClinic.
tech

Свежие посты в категории «Tech Infrastructure»

Все каналы категории →

start

Готовы запустить рекламу через сеть public.tg?

Новый оффер, продукт, GEO, кейс, событие или партнёрский запуск — соберём маршрут под задачу и отдадим медиаплан.

Telegram для медиаплана: @AFFtop_connect. Быстрый тест: $20 за канал, $1000 за пакет по сети.