An alert fired at 3am and nobody saw it for two hours. How do I prevent that?

An alert fired at 3am and nobody saw it for two hours. How do I prevent that?

Q: Our on-call person slept through a page and the outage ran for hours. We need a safety net. What do teams actually do?

A: A reader's post-incident review found the gap: a single notification to one phone, no follow-up if unacknowledged. That one missed page turned a fixable 10-minute issue into a 2-hour 15-minute outage.

They built a tiered escalation: alert the primary on-call, then if not acknowledged within 5 minutes, escalate to the secondary, then after 10 more minutes, page the team lead. They also switched the critical path from email to a phone call plus push.

Result over the next six months: median acknowledge time fell to 3 minutes, and no incident ran past 25 minutes from detection. The longest gap was one 12-minute escalation to the secondary — which is exactly the system working.

The follow-up: test the escalation quarterly with a fake alert. Untested escalation paths fail silently right when you need them.

Got a question? Drop it in the comments.