One bad load-spike day cuts your crawl rate for weeks

One bad load-spike day cuts your crawl rate for weeks

From server logs of 119 domains, days where 5xx exceeded a 2% error rate triggered Google to cut crawl rate by a median 38% — and recovery lagged 16 days.

— Sites with crawl-throttle events/yr: 49.0% ▓▓▓▓▓░░░░░
— Median crawl-rate cut after spike: ↓ 38%
— Recovery lag (median): 16 days
— p90 recovery: 41 days

The mistake: serving 500/503 during deploys, backups, or load spikes. Google reads instability and backs off to protect your server.

Fix: return 503 with a Retry-After header for true maintenance (Google waits politely), and fix the 500s — those it reads as broken. Cohort that cleaned on Retry-After avoided throttling entirely; error-day crawl held flat (Δ 0%).

So what: how you fail tells Google how often to come back.