<b>The directory whose origin was 80% bots</b>

<b>The directory whose origin was 80% bots</b>

A business directory with 2 million listing pages watched origin load climb relentlessly. Investigation showed the traffic wasn't users — it was a scraper crawling deep, rarely-visited listings, each one a cold cache-miss that fell straight through to origin. Eighty percent of origin requests came from a handful of aggressive IPs hitting the long tail nobody else touched.

We added an edge rate-limit on requests-per-IP and a managed-bot ruleset, applied only at the edge so legitimate cache hits stayed untouched. Scrapers now hit a 429 at the point of presence; the origin never sees them.

Origin requests dropped 78% overnight. Origin CPU fell from a constant 70% to 16%, and the database stopped thrashing on cold long-tail lookups. Real-user latency in Phoenix improved 30% because origin had headroom again.

The number that mattered: origin requests down 78% after one edge rule.