The alert that cried wolf for a year
It was a Monday when a travel brand's "alert at 200% spike" rule fired for the fortieth time that quarter, and the team had stopped looking. A genuine refund crisis was hiding in the noise of forty false alarms, mostly Monday-morning traffic that always doubled.
They switched from a fixed-threshold alert to an anomaly baseline that learned each day's normal rhythm. False alarms fell from roughly 13 a month to 2. When a real event hit, volume was only 140% above raw average, below the old trigger, but 6x above the day's learned baseline, so it fired correctly.
The cost: the baseline needed eight weeks of history before it was trustworthy.
The takeaway: a fixed threshold treats Tuesday like Saturday and trains your team to ignore alarms. A baseline that knows your rhythm cries wolf far less, and gets believed when it matters.
Signal & Noise
@thesignalnoise
The alert that cried wolf for a year
Этот пост опубликован в Telegram-канале Signal & Noise. Подписаться можно по ссылке: @thesignalnoise.