Playbook: respond to a sudden crawl-rate spike

Playbook: respond to a sudden crawl-rate spike
Median benign crawl-rate sits within ±20% week-over-week. A spike >2.5x ▓▓▓▓▓▓░░░░ warrants triage (baseline from 90 monitored sites).

Runbook:
— Step 1: confirm the bot — reverse-DNS verify it's real Googlebot, not a spoofed scraper (median 31% of 'Googlebot' UA is fake)
— Step 2: bucket the spike by path. Concentrated in one prefix = new URL set discovered (often a leak)
— Step 3: check status mix during spike — rising 5xx means you're being overwhelmed, throttle via crawl settings
— Step 4: if it's facet/param URLs, deploy disallow same-day; spikes self-resolve in median 6 days after
— Step 5: log the event, tie to the deploy/sitemap change that triggered it

So what: a third of crawl 'spikes' are spoofed bots. Verify the UA by reverse DNS before you touch a single setting.