An alert fired at 3am and nobody saw it for two hours. How do I prevent that?
Q: Our on-call person slept through a page and the outage ran for hours. We need a safety net. What do teams actually do?
A: A reader's post-incident review found the gap: a single notification to one phone, no follow-up if unacknowledged. That one missed page turned a fixable 10-minute issue into a 2-hour 15-minute outage.
They built a tiered escalation: alert the primary on-call, then if not acknowledged within 5 minutes, escalate to the secondary, then after 10 more minutes, page the team lead. They also switched the critical path from email to a phone call plus push.
Result over the next six months: median acknowledge time fell to 3 minutes, and no incident ran past 25 minutes from detection. The longest gap was one 12-minute escalation to the secondary — which is exactly the system working.
The follow-up: test the escalation quarterly with a fake alert. Untested escalation paths fail silently right when you need them.
Got a question? Drop it in the comments.
Pingback Clinic
@PingbackClinic
An alert fired at 3am and nobody saw it for two hours. How do I prevent that?
Этот пост опубликован в Telegram-канале Pingback Clinic. Подписаться можно по ссылке: @PingbackClinic.