Aggressive bot rules are blocking Googlebot itself

Aggressive bot rules are blocking Googlebot itself

Across 134 domains behind a WAF/CDN, 14% rate-limited or 403'd real Googlebot during traffic spikes — verified by reverse DNS in logs.

— Sites blocking verified Googlebot: 14.0% ▓░░░░░░░░░
— Median 403/429 served to Googlebot/day: 2,200
— Crawl-rate impact: ↓ 33% over 2 weeks
— Pages dropped from index: median 90

The mistake: a generic 'block bots' rule or a too-tight rate limit treats Googlebot like a scraper. It backs off and coverage erodes.

Fix: allowlist verified Googlebot (reverse + forward DNS, not just user-agent), raise its rate ceiling, and alert on 4xx/5xx served to known crawlers. Cohort that allowlisted restored crawl rate within 9 days and recovered dropped pages (↑ 100% of lost URLs back in 21 days).

So what: audit what your firewall serves to Google, not just to users.