When the primary CDN went dark for 90 minutes

When the primary CDN went dark for 90 minutes

A fintech dashboard in Toronto rode a single CDN. When that provider had a global incident, the whole product was offline for 90 minutes and support drowned.

We built a multi-CDN failover runbook:

— Put both CDNs behind a DNS provider doing health-checked failover, 30s TTL
— Health check hit a /healthz path through each CDN every 10s from three regions
— Pre-configured the secondary CDN identically: same cache rules, same origin shield, same purge tags
— Ran a monthly game-day forcing failover to confirm the secondary actually served warm
— Documented the manual override in case DNS health-check lagged

The rebuild paid off the next incident — failover happened in 40 seconds, most users never noticed. The mistake people make: configuring the backup CDN but never warming or testing it, so it's cold and misconfigured the day you need it.

The number that mattered: 90-minute outage cut to a 40-second blip.